Closed milely closed 1 year ago
Hi!
In the beginning of training, it's indeed very slow, probably due to the hungarian matching process. As you can imagine, hungarian matching would be slower if there are a lot of random masks. After about a few thousands to 10K steps, the training speed should converge to 0.3~0.4 steps/sec.
You can try to use a smaller number of object queries (https://github.com/tensorflow/models/blob/11cea3e52955bf767ea7edb31786150d2654cfd8/official/projects/unified_detector/configs/gin_files/unified_detector_model.gin#L5) and smaller images to speed up training.
Thanks for your reply, I adjusted object queries from 384 to 256, the training speed has sped up a little, and I will continue to train. Thanks again
Thank you for the open source code. In the process of using the open source dataset to train the model, I used 8 x 3090 for model training, and the batch size was adjusted to 16. But the overall training process is very slow, and the utilization of GPU is also very low. I am not very familiar with the Tensorflow framework training model. Are there any specific parameters that need to be adjusted? Do you have any suggestions? Thank you very much Below is the training log.