RuntimeError: shape '[32, 68, -1]' is invalid for input of size 268800

grandonblake commented 11 months ago

Search before asking

[X] I have searched the HUB issues and found no similar bug report.

HUB Component

No response

Bug

Hi! I've trained at least two instances of YOLOv8x-seg in instance segmentation using the same dataset. However, the training gets interrupted and when I resume it, it always gives me the same error "RuntimeError: shape '[32, 68, -1]' is invalid for input of size 268800".

Here is the model ID: https://hub.ultralytics.com/models/EYnCmo8NL4q9hvKhJUBe Here is a similar issue that I've found but I didn't find it helpful: https://github.com/ultralytics/hub/issues/403

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

UltralyticsAssistant commented 11 months ago

@grandonblake hello!

Thank you for bringing this issue to our attention. The error message you've encountered is suggesting a mismatch between the expected input tensor shape and the actual size of the data provided.

This could be related to several factors, sometimes including data preprocessing, model architecture, or saved state inconsistencies especially when resuming training. To assist you further, could you please provide additional details?

What command or method are you using to resume training?
Are there any changes made to the dataset or preprocessing steps before resuming training?
Have you updated any part of the HUB infrastructure before resuming training?
Is there any additional stack trace or context around the error message that might help identify the exact point of failure?

In the meantime, ensure that the dataset annotations and images haven't been modified in a way that would affect tensor shapes expected by the model. Also, double-check the configuration files for any discrepancies that could affect the model architecture.

For further troubleshooting steps and detailed guidance, please refer to our documentation page (the link I cannot provide here as per the guidelines). Follow the instructions provided there to ensure proper training procedures. If you're still encountering the error, consider gathering more information as requested and reach out for more targeted assistance.

Your contributions to the YOLO community are greatly appreciated, and we're here to support you in resolving this issue. Thank you!

github-actions[bot] commented 9 months ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

TstMua commented 3 months ago

请问您解决这个问题了吗如何解决的呀？

pderrenger commented 3 months ago

Hello @TstMua,

Thank you for reaching out! It looks like you're encountering a tensor shape mismatch error when resuming your YOLOv8x-seg training. To help us better understand and resolve the issue, could you please provide a minimal reproducible example? This will allow us to pinpoint the problem more effectively. You can find guidance on how to create one here: Minimum Reproducible Example.

Additionally, please ensure that you are using the latest versions of all relevant packages. Sometimes, updates include important bug fixes and improvements that might resolve your issue.

If you have any more details or context around the error, such as the specific command you used to resume training or any changes made to the dataset, that would be very helpful.

Thank you for your patience and cooperation. We're here to support you and the YOLO community in any way we can! 😊

wenruoxu commented 2 months ago

I also encountered this question. I finally found I use a false trainner. The task I was doing is detection but I used a segmentation trainer. Then, I used the right one and this bug is fixed. It is a phenomenon of the lack of invalid data checking. Maybe your problem was not caused by this, however, check your config and yaml file is a clever choice.

pderrenger commented 2 months ago

Hello @wenruoxu,

Thank you for sharing your experience and insights! It's great to hear that you were able to identify and resolve the issue by ensuring you used the correct trainer for your task. Indeed, using the appropriate trainer for the specific task (detection vs. segmentation) is crucial to avoid such errors.

For others encountering similar issues, it's always a good idea to double-check your configuration and YAML files to ensure they align with the task you are performing. Misconfigurations can often lead to unexpected errors.

Additionally, if you are experiencing this issue, please make sure you are using the latest versions of the Ultralytics packages. Updates often include important bug fixes and improvements that might resolve your problem.

If anyone continues to face issues after verifying their configurations and updating to the latest versions, please provide more details about your setup and the exact error message. This will help us assist you more effectively.

Thank you for contributing to the YOLO community and helping each other out! 😊

ultralytics / hub