search

aiidateam / aiida-hyperqueue

AiiDA plugin for the HyperQueue metascheduler.
http://aiida-hyperqueue.readthedocs.io/
MIT License
5 stars 7 forks source link

👌 IMPROVE: Avoid using `sed` to add `$HQ_CPUS` #2

Open mbercx opened 2 years ago

mbercx commented 2 years ago

To avoid the current bash escaping that adds single quotes around every command and argument, I added a bit of hacky sed logic:

https://github.com/aiidateam/aiida-hyperqueue/blob/3f38f178badf7c1ce7ad28eca105a2bec26af226/aiida_hyperqueue/scheduler.py#L150-L153

If/when it's possible to avoid the single quotes around all items in the srun line, we can e.g. simply add this to the srun command of the computer setup.

However, we should probably not add this option at all when running on multiple nodes. I haven't tested this so far.

giovannipizzi commented 2 years ago

Indeed - in addition, other computers might use mpirun instead of srun. The --cpu-bind=... part should be part of the mpirun command when we setup the computer (once the issue of escaping is fixed). Also, for now (see https://github.com/It4innovations/hyperqueue/issues/243#issuecomment-993559290) we should expect for now that we run only single-node jobs with HQ. When HQ will support it, we can change the plugin accordingly.

giovannipizzi commented 2 years ago

If my suggestion in this issue is accepted, we can also drop the chmod part. Otherwise, we should probably have AiiDA do it by default?