ultralytics / yolov5

YOLOv5 πŸš€ in PyTorch > ONNX > CoreML > TFLite
https://docs.ultralytics.com
GNU Affero General Public License v3.0
50.56k stars 16.3k forks source link

Classification accuracy low #3232

Closed XingZeng307 closed 3 years ago

XingZeng307 commented 3 years ago

❔Question

Hi, thanks for the great work and opening up the classifier branch.

Could you please check again? I did training on MNIST too, using yolov5s, but only 41% accuracy in top5 epochs, but if using EfficientNet, the results will be similar to the above one you were shown in issue#2001. (MNIST is at 99% in 5 epochs.)

I also tried a different dataset, there is still a big difference in accuracy between yolov5 classifier and EfficientNet classifier.

I pulled the latest repository. (don't know why it doesn't work). Here is my result.

python classifier.py --data mnist output:

epoch gpu_mem train_loss val_loss accuracy
1/100 0.493G 2 1.86 0.277 : 100%|| 469/469 [00:09<00:00, 49.32it/s]
2/100 0.95G 1.8 1.8 0.273 : 100%|| 469/469 [00:09<00:00, 49.34it/s]
3/100 0.95G 1.72 1.77 0.3 : 100%|| 469/469 [00:09<00:00, 48.96it/s]
4/100 0.95G 1.65 1.67 0.355 : 100%|| 469/469 [00:09<00:00, 49.04it/s]
5/100 0.95G 1.61 1.63 0.417 : 100%|| 469/469 [00:09<00:00, 48.98it/s]
6/100 0.95G 1.58 1.63 0.388 : 100%|| 469/469 [00:09<00:00, 48.68it/s]
7/100 0.95G 1.54 1.57 0.403 : 100%|| 469/469 [00:09<00:00, 48.65it/s]
8/100 0.95G 1.52 1.55 0.417 : 100%|| 469/469 [00:09<00:00, 48.64it/s]
9/100 0.95G 1.49 1.52 0.43 : 100%|| 469/469 [00:09<00:00, 48.50it/s]
10/100 0.95G 1.47 1.52 0.423 : 100%|| 469/469 [00:09<00:00, 48.28it/s]

Thank you!

github-actions[bot] commented 3 years ago

πŸ‘‹ Hello @XingZeng307, thank you for your interest in πŸš€ YOLOv5! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a πŸ› Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://www.ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python 3.8 or later with all requirements.txt dependencies installed, including torch>=1.7. To install run:

$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Status

CI CPU testing

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), testing (test.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

glenn-jocher commented 3 years ago

@XingZeng307 classifier branch was behind by quite a few commits, so I merged master and re-ran the example usage, and also observe poor results now. I'll take a look at this this week, as YOLOv5s should be producing about 0.99 accuracy on MNIST also, so it looks like there's a bug somewhere. Thanks for letting us know!

Screenshot 2021-05-19 at 13 09 20
glenn-jocher commented 3 years ago

@XingZeng307 the output is shown here most recently on 10 March: https://github.com/ultralytics/yolov5/issues/2409#issuecomment-794591997

So it seems like something changed between then and now to cause the accuracy drop.

glenn-jocher commented 3 years ago

@XingZeng307 Ok I see the problem. In the new training only the new Classify() head has it's gradients turned on, so the entire backbone is frozen. You can see in my screenshot from today's classifier training it says 1192362 parameters, 2570 gradients, whereas in the March 2021 results it shows 1194954 parameters, 1194954 gradients.

glenn-jocher commented 3 years ago

@XingZeng307 good news πŸ˜ƒ! Your original issue may now been fixed βœ… in https://github.com/ultralytics/yolov5/commit/ad50c810077c72429433d0294474f53d6d6cd80f. To receive this update you can:

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 πŸš€!

XingZeng307 commented 3 years ago

@glenn-jocher Thank you sooo much. It works now!!

glenn-jocher commented 3 years ago

@XingZeng307 no problem. The classifier branch still needs some work to figure out the best hyperparameters to use for training though. I noticed if I set the epochs too short, i.e. --epochs 5 the losses become unstable, but at --epochs 10 they train fine.

XingZeng307 commented 3 years ago

@glenn-jocher Yeah. As you did in detection, maybe applying hyperparameter evolution helps for hyperparameters optimization? And also adapting the mosaic dataloader for better performance.

github-actions[bot] commented 3 years ago

πŸ‘‹ Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 πŸš€ resources:

Access additional Ultralytics ⚑ resources:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 πŸš€ and Vision AI ⭐!

mzhadigerov commented 2 years ago

Do I need to clone from 'classifier' branch or from the master/main branch?? Since there is no classifier.py in the main branch

mzhadigerov commented 2 years ago

I'm also getting very low accuracy on the custom dataset.

I'm using this command: !python classifier.py --data orientation --epochs 5 --img 640 --batch 16 --model yolov5s --cache

I'm getting around 40% accuracy.