Is there a best practice when computations are done in a cluster?

ysig / GraKeL

A scikit-learn compatible library for graph kernels

https://ysig.github.io/GraKeL/

Other

593 stars 97 forks source link

Is there a best practice when computations are done in a cluster? #68

Closed ragnarlevi closed 2 years ago

ragnarlevi commented 3 years ago

Thank for your the package!

I wanted to try to speed up the fitting process on a cluster and I was wondering if there are any best practices in doing so? For example does GraKel have multiprocessing options?

The job is submitted via SLURM.

ysig commented 3 years ago

Hi @ragnarlevi. Nice question.

So Grakel supports basic parallelism using joblib whenever it's possible. On the other hand for some kernels are written only using numpy vectorized operations, which follow low level parallelism through LAPACK.

We haven't made a scaling study for GraKeL in cluster, so please keep us posted about the experiments you will make and any observation you may have.