Open 9thDimension opened 3 years ago
How much number of epoch are you using ? and try to reduce hidden layer
Hi there, I have the same issue related to the speed. I managed to train my object detection model with TF2 on Google Colab, but it took about 33 seconds per step. This issue hasn't seen before with the same config. Could you help me figure out some things wrong?
I'm seeing this: 2022-07-21 20:25:44,906 [INFO]: Step 2000 per-step time ### 3.160s
This is for object detection mobilenet v2 ssd training on COCO. I am using 2 GPU's and num_workers as 8. 3 secs per step is very slow. Using the RTX 8000 GPU.
1. The entire URL of the file you are using
https://github.com/tensorflow/models/blob/master/research/object_detection/model_main_tf2.py
2. Describe the bug
Training SSD object detection with GPU enabled is the same speed as CPU training.
3. Steps to reproduce
Install according to the documentation https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/install.html
Run the "Training a custom object detector" tutorial https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html
Note that contrary to the installation guide you must replace CUDA 10.1 with the appropriate version, as this reddit user points out https://old.reddit.com/r/tensorflow/comments/limkyu/gpu_bug_using_tensorflow_2x_object_detection_api/gnfl8sl/
4. Expected behavior
Training faster than laptop CPU
5. Additional context
Here is how long iterations took on laptop CPU:
Here is the corresponding output from EC2 P2 instance with the Tesla K80 GPU with the same data and identical configuration:
Nvidia-smi showed full GPU memory usage:
And here is the full output from GPU training, complete with many warnings:
6. System information