plot_results does not display outputs; weights.pt saved in different location [bug introduced July 9]

[josephofiowa]

🐛 Bug

1. utils.plot_results() creates blank output graphs:

https://github.com/ultralytics/yolov5/commit/603ea0bfdc9b0db3b367fbe7cac5ba6f9a50a49f#diff-e6186f834ed280c4cb710843e5da633b

2. The model weights are no longer saved in the same location (I believe), resulting in them not being found for inference:

These bugs were introduced today. (Perhaps this commit is responsible for # 1).

To Reproduce

This Colab notebook documents the errors clearly: https://colab.research.google.com/drive/12v_XkfhTKg0pZucI0oymdHdNRPItywo5?usp=sharing

Scroll to the expected output graphs, printouts of augmented data, and attempted inference.

(Adapted from YOLOv5 tutorial)

[github-actions[bot]]

Hello @josephofiowa, thank you for your interest in our work! Please visit our Custom Training Tutorial to get started, and see our Jupyter Notebook , Docker Image, and Google Cloud Quickstart Guide for example environments.

If this is a bug report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom model or data training question, please note that Ultralytics does not provide free personal support. As a leader in vision ML and AI, we do offer professional consulting, from simple expert advice up to delivery of fully customized, end-to-end production solutions for our clients, such as:

• Cloud-based AI systems operating on hundreds of HD video streams in realtime.
• Edge AI integrated into custom iOS and Android apps for realtime 30 FPS video inference.
• Custom data training, hyperparameter evolution, and model exportation to any destination.

For more information please visit https://www.ultralytics.com.

[glenn-jocher]

@josephofiowa hi there, yes you are correct! This change is related to a significant rework train logging to bring us more into line with best practices from tensorboard, pytorch lightning etc. Originally all files were logged to the main yolov5/ directory, now we are logging everything to unique yolov5/runs/exp (exp for 'experiment') folders that tensorboard is also logging to that increment with each new training as runs/exp0, runs/exp1 etc.

If you git clone a fresh repo and train, then all training results will beyolov5runs/exp0/, including results.txt and results.png, among other stuff. There is a function that returns last run I believe, I can try to find it so we can update that notebook cell to plot the latest training.

[glenn-jocher]

The main PR for this is here BTW: https://github.com/ultralytics/yolov5/pull/104

[glenn-jocher]

Ok, thinking about this for a second, there are two ways to handle this. The first would be if we added a train.py argparser argument to specify the log_dir, as it's called. You could specify '.' to dump all training results to yolov5/, which would mimic the old behavior, or a fixed dir like runs/tutorial/ (and then update all subsequent code to point to weights/results there).

The other alternative is to move all weights and results from log_dir to . right after training finishes, and then leave all subsequent code alone. What do you think?

I just realized this probably broke our notebook also (!).

[glenn-jocher]

@josephofiowa ok, I've updated the notebooks below now to fix the path issues, and also made a few small improvements. The fix I went with was to simply explain the results saving paths (i.e. runs/exp0, runs/exp1, etc), and to add the runs/exp0/ prefix to the original plotting commands. WIll work if someone trains only once, but its good enough for the tutorial probably. Let me know if you think there are any other problem items.

• Google Colab Notebook with free GPU:
• Kaggle Notebook with free GPU: https://www.kaggle.com/models/ultralytics/yolov5

[glenn-jocher]

@josephofiowa oh btw, the plot_results() cell you pictured above is plotting twice because plot_results() by itself is producing plots in the cell, and then Image() after it is displaying the same plots in the same cell a second time, so in the latest notebooks I removed Image().

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.