mem error

[MaAl13]

Hello everyone,

I am trying to run Dask on our cluster, however the following code produces an error i don't really know how to solve

from dask_jobqueue import SLURMCluster
cluster = SLURMCluster(
            cores=24,
            job_extra_directives=['--mem-per-cpu=500MB'],  # Request 500MB per CPU
            queue="regular",
            account="user",
            memory = "12 GB")

cluster.scale(10)

• tornado.application - ERROR - Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x2b83c3fcf7a0>>, <Task finished name='Task-1267' coro=<SpecCluster._correct_state_internal() done, defined at /cluster/home/user/miniconda/lib/python3.12/site-packages/distributed/deploy/spec.py:346> exception=RuntimeError('Command exited with non-zero exit code.\nExit code: 1\nCommand:\nsbatch /tmp/tmp28aolw5t.sh\nstdout:\n\nstderr:\nsbatch: error: lua: Requesting memory by node is not supported. Use --mem-per-cpu.\nsbatch: error: cli_filter plugin terminated with error\n\n')>) Traceback (most recent call last): File "/cluster/home/user/miniconda/lib/python3.12/site-packages/tornado/ioloop.py", line 738, in _run_callback ret = callback() ^^^^^^^^^^ File "/cluster/home/user/miniconda/lib/python3.12/site-packages/tornado/ioloop.py", line 762, in _discard_future_result future.result() File "/cluster/home/user/miniconda/lib/python3.12/site-packages/distributed/deploy/spec.py", line 390, in _correct_state_internal await asyncio.gather(*worker_futs) File "/cluster/home/user/miniconda/lib/python3.12/asyncio/tasks.py", line 684, in _wrapawaitable return await awaitable ^^^^^^^^^^^^^^^ File "/cluster/home/user/miniconda/lib/python3.12/site-packages/distributed/deploy/spec.py", line 74, in await self.start() File "/cluster/home/user/miniconda/lib/python3.12/site-packages/dask_jobqueue/core.py", line 426, in start out = await self._submit_job(fn) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/cluster/home/user/miniconda/lib/python3.12/site-packages/dask_jobqueue/core.py", line 409, in _submit_job return await self._call(shlex.split(self.submit_command) + [script_filename]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/cluster/home/user/miniconda/lib/python3.12/site-packages/dask_jobqueue/core.py", line 505, in _call raise RuntimeError( RuntimeError: Command exited with non-zero exit code. Exit code: 1 Command: sbatch /tmp/tmp28aolw5t.sh stdout:

stderr: sbatch: error: lua: Requesting memory by node is not supported. Use --mem-per-cpu. sbatch: error: cli_filter plugin terminated with error

[guillaumeeb]

Hi @MaAl13,

Your cluster is configured in a way to refuse submission scripts using --mem= sbatch option. You already added the --mem-per-cpu, you should also skip the other directive, see job_directives_skip kwarg.