Problems with AMP - yolov11n - Google Colab

Ayazzia01 commented 1 month ago

Search before asking

[X] I have searched the Ultralytics YOLO issues and found no similar bug report.

Ultralytics YOLO Component

Train

Bug


Transferred 926/941 items from pretrained weights
TensorBoard: Start with 'tensorboard --logdir runs/detect/fullImage_arrowShape-v12', view at http://localhost:6006/
AMP: running Automatic Mixed Precision (AMP) checks with YOLO11n...
AMP: checks failed ❌. Anomalies were detected with AMP on your system that may lead to NaN losses or zero-mAP results, so AMP will be disabled during training.
train: Scanning /content/FullImage-ArrowANDShape-2/train/labels.cache... 26 images, 0 backgrounds, 0 corrupt: 100% 26/26 [00:00<?, ?it/s]
/usr/local/lib/python3.10/dist-packages/albumentations/__init__.py:13: UserWarning: A new version of Albumentations is available: 1.4.17 (you have 1.4.15). Upgrade using: pip install -U albumentations. To disable automatic update checks, set the environment variable NO_ALBUMENTATIONS_UPDATE to 1.
  check_for_updates()
albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01, num_output_channels=3, method='weighted_average'), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8))
val: Scanning /content/FullImage-ArrowANDShape-2/valid/labels.cache... 3 images, 0 backgrounds, 0 corrupt: 100% 3/3 [00:00<?, ?it/s]
Plotting labels to runs/detect/fullImage_arrowShape-v12/labels.jpg... 
optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically... 
optimizer: AdamW(lr=0.002, momentum=0.9) with parameter groups 143 weight(decay=0.0), 206 weight(decay=0.0005), 226 bias(decay=0.0)

Environment

Ultralytics 8.3.1 🚀 Python-3.10.12 torch-2.4.1+cu121 CUDA:0 (NVIDIA L4, 22700MiB)
Setup complete ✅ (12 CPUs, 53.0 GB RAM, 36.6/112.6 GB disk)

OS                  Linux-6.1.85+-x86_64-with-glibc2.35
Environment         Colab
Python              3.10.12
Install             pip
RAM                 52.96 GB
CPU                 Intel Xeon 2.20GHz
CUDA                12.1

numpy               ✅ 1.26.4<2.0.0,>=1.23.0
matplotlib          ✅ 3.7.1>=3.3.0
opencv-python       ✅ 4.10.0.84>=4.6.0
pillow              ✅ 10.4.0>=7.1.2
pyyaml              ✅ 6.0.2>=5.3.1
requests            ✅ 2.32.3>=2.23.0
scipy               ✅ 1.13.1>=1.4.1
torch               ✅ 2.4.1+cu121>=1.8.0
torchvision         ✅ 0.19.1+cu121>=0.9.0
tqdm                ✅ 4.66.5>=4.64.0
psutil              ✅ 5.9.5
py-cpuinfo          ✅ 9.0.0
pandas              ✅ 2.1.4>=1.1.4
seaborn             ✅ 0.13.1>=0.11.0
ultralytics-thop    ✅ 2.0.8>=2.0.0
torch               ✅ 2.4.1+cu121!=2.4.0,>=1.8.0; sys_platform == "win32"

Minimal Reproducible Example

!yolo train model=rtdetr-l.pt data=/content/FullImage-ArrowANDShape-2/data.yaml imgsz=1024 epochs=100 patience=20 batch=1 name=fullImage_arrowShape-v1

Additional

No response

Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

UltralyticsAssistant commented 1 month ago

👋 Hello @Ayazzia01, thank you for reaching out to Ultralytics 🚀! This is an automated response to guide you as our engineers prepare to assist you soon.

For issues with Automatic Mixed Precision (AMP) or training on Google Colab, please ensure your setup aligns with our recommended configurations. Check the Docs for detailed setup instructions and the latest examples for both Python and CLI.

Since you've encountered a 🐛 bug, please ensure that your environment meets all requirements. If you haven’t already, try upgrading to the latest ultralytics package:

pip install -U ultralytics

You’re using it in Google Colab, which is great, but you might want to test your setup in another environment as well. These include:

Notebooks:
Cloud Platforms: GCP or AWS are also solid choices.
Docker: Ensure compatibility with Ultralytics Docker Images.

For more real-time support, join our Discord 🎧 or explore discussions in our Community Forum.

If this badge is green, all Ultralytics CI tests are passing, verifying the correct operation of YOLOv8:

Thank you for your patience and understanding while we work on your issue! 😃

Y-T-G commented 1 month ago

Duplicate of https://github.com/ultralytics/ultralytics/issues/16582

Y-T-G commented 1 month ago

A fix will be rolled out soon. Thank you.

glenn-jocher commented 1 month ago

Great news 😃! Your original issue may now be resolved ✅ in PR #16583.

To get this update:

Git – Run git pull from within your ultralytics/ directory or run git clone https://github.com/ultralytics/ultralytics again
Pip – Update with pip install -U ultralytics
Notebooks – Check out the updated notebooks
Docker – Run sudo docker pull ultralytics/ultralytics:latest to update your image

Thank you for spotting this issue and letting us know. Please confirm if this update fixes the issue for you, and don't hesitate to report any other issues you find or feature requests you may have. Happy training with YOLOv8 🚀!

patrio4 commented 1 month ago

My problem was solved by running the following command :) I have a virtual environment in Conda and I ran this code inside it

pip install -U ultralytics

glenn-jocher commented 1 month ago

Glad to hear your issue was resolved by updating Ultralytics! If you have any more questions, feel free to ask.

ultralytics / ultralytics