Cloud Training Never Start/End

GuppyMIS commented 3 days ago

Search before asking

[X] I have searched the HUB issues and found no similar bug report.

HUB Component

Training

Bug

Hi,

Yesterday I trained my dataset using Cloud Training using Nvidia L40S, but it seems until now the training not yet finished/started.

My account already being deducted but my model never shown up.

Please advise

Environment

-Ultralytics Pro Account -Ultralytics Cloud Training -Instances: Nvidia L40s

Minimal Reproducible Example

No response

Additional

No response

UltralyticsAssistant commented 3 days ago

👋 Hello @GuppyMIS, thank you for bringing your issue to our attention regarding the Ultralytics HUB 🚀! We're sorry to hear that you're encountering difficulties with the Cloud Training.

To get you started on troubleshooting the issue, please have a look at our HUB Docs which provide comprehensive guidance:

Models: Training and Exporting. Explore how to train YOLOv5 and YOLOv8 models, tackle common issues, and export them for deployment.
Inference API. Gain insight into how to utilize the Inference API for your trained models.

Given that this appears to be a 🐛 Bug Report, we kindly ask you to provide a minimum reproducible example along with any screenshots or logs that illustrate the issue. You can learn more about creating an MRE in our guide.

An Ultralytics engineer will review your issue and assist you shortly. We strive to address all queries as quickly as possible. Thank you for your patience and collaboration in resolving this! 🙌

pderrenger commented 3 days ago

@GuppyMIS hi there! Sorry to hear you're experiencing issues with Cloud Training 🤔

This seems like an unusual situation with your training session. Let me help you troubleshoot this:

First, please verify if you can see the model listed under your Models page in HUB, even if it shows as pending or incomplete.
According to our Cloud Training documentation, the initialization process can sometimes take a while depending on GPU resource availability, but should not be indefinite. However, your situation appears to be an edge case since your account was already charged.
Since this involves billing and potential technical issues with the training infrastructure, I recommend:

a. Taking a screenshot of your billing section showing the deduction b. Opening a support ticket through GitHub Issues with:
- The exact time you started the training
- Your model ID or name
- The screenshots of both the training status and billing
- The dataset size and number of epochs configured
In the meantime, you can try starting a new training session with a smaller number of epochs first to verify if the issue persists.

The Ultralytics team monitors these issues closely and will help resolve any billing discrepancies or technical issues you've encountered.

For future training sessions, you might want to check out our Cloud Training guide which includes best practices and troubleshooting tips for smooth training experiences.

Let me know if you need any clarification or have additional questions! 🙂

ultralytics / hub