search

ultralytics / hub

Ultralytics HUB tutorials and support
https://hub.ultralytics.com
GNU Affero General Public License v3.0
138 stars 14 forks source link

Uploading Dataset Error #894

Open shinbehavior opened 1 month ago

shinbehavior commented 1 month ago

Search before asking

HUB Component

No response

Bug

I have the .zip of my dataset, which is 58 GB (on Hub it became 54). It has a similar structure to the example but without a test set. Full upload completes in 1.5-2 hours and after I get this for 14 hours:

image

I was 100% sure that I had a stable ethernet connection. What error here can be? Is it because of the lack of a test set? Because Loading is endless.

Environment

Minimal Reproducible Example

  1. 58gb size of dataset
  2. Make structure and data.yaml without Test set
  3. Compress
  4. Upload

Additional

image

UltralyticsAssistant commented 1 month ago

👋 Hello @shinbehavior, thank you for raising an issue about the Ultralytics HUB 🚀! It seems like you're having some trouble uploading your dataset. Please check out our HUB Docs to make sure everything is set up correctly:

Since this is a 🐛 Bug Report, could you please provide a minimum reproducible example (MRE) to help us better understand the issue? This would include, if possible, a smaller dataset that still replicates the problem. You can learn more about creating an MRE here.

We appreciate your patience, and an Ultralytics engineer will join the conversation to assist you further soon. Thank you! 🌟

pderrenger commented 1 week ago

@shinbehavior hello!

Thank you for reaching out and providing detailed information about the issue you're experiencing. 😊

It sounds like you've done a great job preparing your dataset, but let's see if we can pinpoint the issue. Here are a few suggestions:

  1. Dataset Structure: While the absence of a test set shouldn't inherently cause an upload issue, it's crucial to ensure that your dataset's structure aligns perfectly with the expected format. Double-check that your data.yaml file is correctly placed in the root directory and that all paths are accurate.

  2. Dataset Size: A 58 GB dataset is quite large, and while Ultralytics HUB should handle it, the upload and processing times can be significant. Ensure that your internet connection remains stable throughout the upload process. You might also want to try uploading during off-peak hours to avoid potential network congestion.

  3. Version Check: Make sure you're using the latest version of Ultralytics HUB. Sometimes, updates include fixes for issues that might be causing your problem.

  4. Validation: Before uploading, validate your dataset locally to catch any potential errors early. You can use the ultralytics.hub.check_dataset function to ensure everything is in order.

  5. Logs and Errors: Check if there are any error messages or logs that might provide more insight into what's going wrong. Sometimes, the issue might be logged in the console or a log file.

If the problem persists after these checks, please let us know, and we can explore further options. Your patience is appreciated, and we're here to help you get your dataset uploaded successfully!

shinbehavior commented 1 week ago
  1. Dataset structure is correct: 
    
train: images/train
val: images/val

names: 0: person


And /labels/train and val.
With the same dataset, trained locally successfully.

2. I tried to upload three times with 100% stable internet connect.

3. I use web, so indeed it was the last version. And you can check date when I open this issue and Hub version which I provide. 

4. I will try it, but you can see, I have correct yaml, and just folders with images and labels, which are for sure in yolo format for detections.

5. I attached the log screenshot which i got in the web console.
sergiuwaxmann commented 1 week ago

@shinbehavior https://docs.ultralytics.com/hub/datasets#upload-dataset Can you please confirm your folder structure is correct? Also, try adding path and test to your .yaml file.

shinbehavior commented 1 week ago

@shinbehavior https://docs.ultralytics.com/hub/datasets#upload-dataset Can you please confirm your folder structure is correct? Also, try adding path and test to your .yaml file.

Yes, folder structure is absolutely the same, like I said earlier. About test set: " While the absence of a test set shouldn't inherently cause an upload issue" - That what @pderrenger said before

sergiuwaxmann commented 1 week ago

@shinbehavior I understand but I suggested trying that way.

shinbehavior commented 1 week ago

@sergiuwaxmann Okay, will try it later

yogendrasinghx commented 1 week ago

@shinbehavior To ensure your dataset is formatted correctly, please watch this YouTube video: Upload Datasets to Ultralytics HUB , which provides a step-by-step guide for creating a valid dataset for Ultralytics HUB. Once you've checked your dataset, try uploading it again following the instructions in the video and in the documentation here.

If the issue persists, please let us know. Thank you for your patience!