Open steven-basart opened 1 year ago
@steven-basart Would you elaborate more on what you mean by Slurm latency and responsiveness? Also did you mean to say that you moved the logs to the FSS?
I saw this comment Which prompted my creation of the issue.
Comment copied below as well for reference in case the page dies.
The file system for /var/spool/slurmctld/ should be mounted on the fastest possible disks (SSD or NVMe if possible).
I had moved them to the FSS and then moved the slurmd logs back off.
Where did you move the slurmd logs?
Check slurm.conf
on the line called SlurmCtrlLogFile (I believe).
As an aside also look in the directory (the default one). I made symlinks from the default location to the new location on the FSS so it could be transparent change to Loki for instance.
I moved
/var/spool/slurmctld/
to be saved onto NFS but this increases slurm latency and responsiveness. We need a better solution. Backup and delete logs once a month or something.