Open finnoshea opened 1 year ago
I remember very well that this was the behavior since the very beginning of Lightning and we've carried this special treatment in SLURM forward through many refactors, always keeping it for backward compatibility. So I think it is so far expected. We've never heard complaints from SLURM users about it, so I assume it is not a big issue. I wouldn't be opposed to changing it but we should only do it if there is a good demand for it, since it would be a breaking change (IMO).
I haven't used SLURM in some time, but I could imagine that the regular versioning could be problematic if users launch a batch of jobs in Slurm and then the folders could be colliding unless users name their folders manually. The job id for this use case provides a unique name so that version folders don't collide. I agree it may seem arbitrary, so I understand the concern.
I'm pinging a couple of users who have previously posted issues recently on our GitHub about SLURM: @shethdhvani @Queuecumber @ipoletaev @ChristophReich1996 @Wildcarde if you've used the TensorBoardLogger versioning in SLURM and want to share your opinion on this behavior.
I actually haven't noticed this, maybe because I use wandb logger (I didn't read the issue fully so I don't know if the code in question is specific to tb)
That said I would love it if other loggers worked that way; I had to put in my own code to do it because it allows me to resume my previous runs when my job gets preempted and requeued.
My guess is that this is where the behavior came from btw
Yes, wandb saves differently and always creates unique runs even if the user-defined run name is not unique. So versioning is handled differently there.
Right, you're making a good point actually. We have also the auto-resubmit feature, and if SLURM resumes a job, the job id stays the same (right?).
I had to put in my own code to do it because it allows me to resume my previous runs
On that note, I think we should pick up an old feature request for storing the experiment id into the checkpoint, so if resuming from a checkpoint, the logger could resume into the same experiment (https://github.com/Lightning-AI/lightning/issues/5342)
It seems like I have an unusual use pattern for SLURM jobs. That is cool. I have been meaning to re-learn tensorboard anyway.
This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions - the Lightning Team!
Bug description
When I use lightning on my personal computer the logs are named with an ascending version number. When I use lightning on a cluster where access is provisioned via SLURM, the version number is the job id number of the SLURM job:
In the above line v_num=8483572 is the SLURM job number. The logs are also saved in a directory named
version_8483572
.I don't know if this is working as intended.
What version are you seeing the problem on?
v1.9
How to reproduce the bug
Error messages and logs here please
- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
- PyTorch Lightning Version (e.g., 1.5.0):
- Lightning App Version (e.g., 0.5.2):
- PyTorch Version (e.g., 2.0):
- Python version (e.g., 3.9):
- OS (e.g., Linux):
- CUDA/cuDNN version:
- GPU models and configuration:
- How you installed Lightning(
conda
,pip
, source):- Running environment of LightningApp (e.g. local, cloud):