search

RangiLyu / nanodet

NanoDet-Plusāš”Super fast and lightweight anchor-free object detection model. šŸ”„Only 980 KB(int8) / 1.8MB (fp16) and run 97FPS on cellphonešŸ”„
Apache License 2.0
5.63k stars 1.03k forks source link

Why the training phase is quite slow with V100 GPU? #530

Closed ForeverPs closed 6 months ago

ForeverPs commented 10 months ago

I have two V100 GPUs, however, the training phase is extremely slow with only 643 images? I wonder the reason.

ForeverPs commented 10 months ago

I find the reason, I suggest the author to modify the code: nanodet/data/dataset/yolo.py, line 97, img = cv2.imread(img_name). This function is extremely slow for images with high resolution, I change this into w, h = imagesize.get(img_name).

ForeverPs commented 10 months ago
image

pip3 install imagesize

ForeverPs commented 10 months ago

However, the code is stuck here:

image
Sanath1998 commented 9 months ago

Hi @ForeverPs, Are u aware of how to save all the checkpoints of each epochs and what epochs does model.best.ckpt saves? Where to check those?

Sanath1998 commented 9 months ago

And also i think in Nanodet training script we can't control. Is that what you have also experienced?