[New Feature] Slurm training script

IDEA-Research / detrex

detrex is a research platform for DETR-based object detection, segmentation, pose estimation and other visual recognition tasks.

https://detrex.readthedocs.io/en/latest/

Apache License 2.0

1.95k stars 204 forks source link

Closed rayleizhu closed 1 year ago

rayleizhu commented 1 year ago

How can I start training on a slurm cluster? Do I need to take care of ddp initialization routines by myself?

FengLi-ust commented 1 year ago

Yes, you can refer to DINO slurm scripts as an example. We will upload this script in the future.

rayleizhu commented 1 year ago

I’ve implemented slurm launcher using submitit. It supports:

one-line launch command on slurm cluster, just like existing trainer provided by current detrex.
automatic experiment name generation according to hyper-parameter overrides.
automatically requeue and resume from latest checkpoint when a running job is preempted or reaches maximum execution time.

Are you interested in merging my implementation to detrex? If yes, I will create a pull request by the end of this week.

FengLi-ust commented 1 year ago

Of course, that would be great!

rentainhe commented 1 year ago

Thanks for your contribution, we will close this issue~ reopen this at anytime if you need more help~