Performance Bottleneck

[Unice-YuFang]

Hi guys,

I'm training YOLOv4 on my own dataset, however, it appears "performance bottleneck on CPU or Disk HDD/SDD" in the log.

I'm wondering how to solve it. Please give me some advice!

[vamsiduranc]

@AlexeyAB Greetings!!

I am facing issue while training (performance bottleneck on CPU or Disk HDD/SSD). Can you please look in the below details and let us know how to resolve the issue?

Please do let me know if you need more information. Thanks in Advance!

Issue:

204726: 0.457573, 0.391292 avg loss, 0.001000 rate, 0.251206 seconds, 13102464 images, 120.645873 hours left
Loaded: 0.542850 seconds - performance bottleneck on CPU or Disk HDD/SSD

Training Details:

Train Images = 620543
Valid Images = 68764

CFG attached here: CFG_File.txt

Server Configuration:

System:    Host: Training-Server Kernel: 5.11.8-051108-generic x86_64 bits: 64 Desktop: Gnome 3.36.7 
           Distro: Ubuntu 20.04.2 LTS (Focal Fossa) 
Machine:   Type: Desktop System: Dell product: Precision 3640 Tower v: N/A serial: <superuser/root required> 
           Mobo: Dell model: 0D4MD1 v: A00 serial: <superuser/root required> UEFI: Dell v: 1.2.3 date: 08/06/2020 
CPU:       Topology: 10-Core model: Intel Core i9-10900 bits: 64 type: MT MCP L2 cache: 20.0 MiB 
           Speed: 800 MHz min/max: 800/5200 MHz Core speeds (MHz): 1: 854 2: 1083 3: 4352 4: 3398 5: 2247 6: 1106 7: 801 
           8: 800 9: 800 10: 800 11: 800 12: 800 13: 800 14: 961 15: 1131 16: 800 17: 800 18: 800 19: 800 20: 800 
Graphics:  Device-1: Intel CometLake-S GT2 [UHD Graphics 630] driver: i915 v: kernel 
           Device-2: NVIDIA GA102 [GeForce RTX 3080] driver: nvidia v: 460.91.03 
           Display: x11 server: X.Org 1.20.9 driver: modesetting,nvidia unloaded: fbdev,nouveau,vesa resolution: 1600x900~60Hz 
           OpenGL: renderer: GeForce RTX 3080/PCIe/SSE2 v: 4.6.0 NVIDIA 460.91.03 
Audio:     Device-1: Intel Comet Lake PCH cAVS driver: snd_hda_intel 
           Device-2: NVIDIA GA102 High Definition Audio driver: snd_hda_intel 
           Sound Server: ALSA v: k5.11.8-051108-generic 
Network:   Device-1: Intel Ethernet I219-LM driver: e1000e 
           IF: eno1 state: up speed: 1000 Mbps duplex: full mac: a4:bb:6d:66:92:eb 
Drives:    Local Storage: total: 931.51 GiB used: 224.01 GiB (24.0%) 
           ID-1: /dev/sda vendor: Seagate model: ST1000DM010-2EP102 size: 931.51 GiB 
Partition: ID-1: / size: 915.40 GiB used: 224.00 GiB (24.5%) fs: ext4 dev: /dev/sda2 
Sensors:   System Temperatures: cpu: 27.8 C mobo: N/A gpu: nvidia temp: 31 C 
           Fan Speeds (RPM): cpu: 638 fan-2: 999 fan-3: 893 gpu: nvidia fan: 30% 
Info:      Processes: 442 Uptime: 40m Memory: 31.07 GiB used: 2.62 GiB (8.4%) Shell: bash inxi: 3.0.38 

[AlexeyAB]

@vamsiduranc

204726: 0.457573, 0.391292 avg loss, 0.001000 rate, 0.251206 seconds, 13102464 images, 120.645873 hours left Loaded: 0.542850 seconds - performance bottleneck on CPU or Disk HDD/SSD

This means that training is not going as fast as possible.

Drives: Local Storage: total: 931.51 GiB used: 224.01 GiB (24.0%) ID-1: /dev/sda vendor: Seagate model: ST1000DM010-2EP102 size: 931.51 GiB

Seagate Barracuda 1 TB ST1000DM010 is HDD. So if you buy SSD instead of HDD, then training can be faster.

[vamsiduranc]

Dear @AlexeyAB Thank you very much for your reply.

204726: 0.457573, 0.391292 avg loss, 0.001000 rate, 0.251206 seconds, 13102464 images, 120.645873 hours left Loaded: 0.542850 seconds - performance bottleneck on CPU or Disk HDD/SSD

Couple of questions please...

• Is the error message is a concern for the training?
• Does it reduce the quality of detection or interference?
• Can we go ahead and continue with the training process even though it is slow?

Appreciate your quick reply on this. Thanks in advance!

[AlexeyAB]

@vamsiduranc You can simply ignore this warning. This will not affect the result or quality.

[vamsiduranc]

@AlexeyAB Thank you very much for your confirmation. Regards!