AlexeyAB / darknet

YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )
http://pjreddie.com/darknet/
Other
21.75k stars 7.96k forks source link

Visualising loss graph while or after training #1564

Open FOX111 opened 6 years ago

FOX111 commented 6 years ago

How can I visualise the loss graph while or after finishing training yolo? I would like to do this in order to see when it makes sense to stop training and also to compare different settings of training. Are there any other built-in tools that might be relevant?

AlexeyAB commented 6 years ago

Look at this folder: https://github.com/AlexeyAB/darknet/tree/master/scripts/log_parser


Another way, if you compiled Darknet with OPENCV=1 and monitor is connected to the computer, then you will see avg loss graph during training, and you can save it to the chart.jpg file if you press and hold S button.

FOX111 commented 6 years ago

@AlexeyAB thanks for your response. How do you save logs when training? I tried standard " > " and " | tee " options but it saves only this to the logs file:

yolov3-tiny_obj Total BFLOPS 5.571

Moreover, the whole training process seems to stop. In contrast, when I run the same command without trying to save to logs file the training goes well and I see output in the console. The command I was using is ./darknet detector train data/voc.data cfg/yolov3-tiny_obj.cfg yolov3-tiny.conv.15 | tee logs.txt