search

dask / dask-jobqueue

Deploy Dask on job schedulers like PBS, SLURM, and SGE
https://jobqueue.dask.org
BSD 3-Clause "New" or "Revised" License
235 stars 142 forks source link

OARCluster implementation does not let OAR take into account the memory parameter #594

Closed ychiat35 closed 2 years ago

ychiat35 commented 2 years ago

The current implementation of OARCluster does not let OAR take into account the memory parameter.

For example, if 256 GB are asked for Dask workers as memory, OAR is not aware about this memory request and thus the resources selected by OAR to compute the job might not have enough memory.

Consequently, specifying memory in OARCluster is counter-intuitive. As a Dask-Jobqueue user, we expect this parameter to be taken into account but in practice it is not.

guillaumeeb commented 2 years ago

Never used OAR, but it looks you're right! The only way to specify job memory is to use resource_spec (https://github.com/dask/dask-jobqueue/blob/main/dask_jobqueue/oar.py#L25).

Consequently, specifying memory in OARCluster is counter-intuitive. As a Dask-Jobqueue user, we expect this parameter to be taken into account but in practice it is not.

:+1:

ychiat35 commented 2 years ago

Maybe @lesteve could help reviewing this pull request #595 ? Thanks in advance!

guillaumeeb commented 2 years ago

I don't think @lesteve has time to look at dask-jobqueue for the time being. But I'll have a look to your PR as soon as I can (following days I hope, but may be weeks, sorry).

guillaumeeb commented 2 years ago

Fixed in #595.