Open stas00 opened 1 month ago
this seems to do the trick:
dist_executor.run()
print(f"*** Find the slurm logs under: {root_dir}/logs/slurm_processing/slurm_logs/ ")
if dist_executor.job_id != -1:
print(f"tail -F {root_dir}/logs/slurm_processing/slurm_logs/{dist_executor.job_id}_0.out")
When using a local executor the running logs appear right away, in the console it was launched from. But when using slurm one has to fish for the log files.
This can be made easier by automatically printing:
first_slurm_job_id
coming from:though we want the first, not the last one here.
even fancier would be to run the
tail
on behalf of the user in the launcher - this way the local and slurm launching experiences will be identical.But even printing the command to copy-n-paste would already be faster than manual fishing for the log file.
if this doesn't resonate as a feature is it possible to make
run()
return some attributes? e.g. the first slurm job id - and then the user can code this feature easily themselves.Thank you!
reading the code I see
launch_slurm_job
returns some job id and it's then set intorun.job_id
but this would only be correct if tasks<1000, correct? otherwise it'll return the last job array and not the first one (since your log says... (last) id=
)?