RuntimeError: CUDA error: an illegal memory access was encountered

alexjaw commented 1 year ago

Search before asking

[X] I have searched the Roboflow Notebooks issues and found no similar bug report.

Notebook name

how-to-auto-train-yolov8-model-with-autodistill.ipynb

Bug

Train target model - YOLOv8: Default epochs=50, got error and tried again with epochs=2 and got same error.

New https://pypi.org/project/ultralytics/8.0.189 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.0.81 🚀 Python-3.10.12 torch-2.0.1+cu118 CUDA:0 (Tesla V100-SXM2-16GB, 16151MiB)
/content
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-22-8c0827b8627e>](https://localhost:8080/#) in <cell line: 6>()
      4 
      5 target_model = YOLOv8("yolov8n.pt")
----> 6 target_model.train(DATA_YAML_PATH, epochs=2)

7 frames
[/usr/local/lib/python3.10/dist-packages/torch/cuda/random.py](https://localhost:8080/#) in cb()
    109         for i in range(device_count()):
    110             default_generator = torch.cuda.default_generators[i]
--> 111             default_generator.manual_seed(seed)
    112 
    113     _lazy_call(cb, seed_all=True)

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Environment

google colab V100 51GB RAM

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

[X] Yes I'd like to help by submitting a PR!

github-actions[bot] commented 1 year ago

👋 Hello @alexjaw, thank you for leaving an issue on Roboflow Notebooks.

🐞 Bug reports

If you are filing a bug report, please be as detailed as possible. This will help us more easily diagnose and resolve the problem you are facing. To learn more about contributing, check out our Contributing Guidelines.

If you require support with custom code that is not part of Roboflow Notebooks, please reach out on the Roboflow Forum or on the GitHub Discussions page associated with this repository.

💬 Get in touch

Do you have more questions about Roboflow that we haven't responded to yet? Feel free to ask them on the Roboflow Discuss forum. Our developer advocates and community team actively respond to questions there.

To ask questions about Notebooks, head over to the GitHub Discussions section of this repository.

alexjaw commented 1 year ago

Have also tested with colab T4 and that works as expected.

roboflow / notebooks