Closed lzziuhh closed 2 years ago
Hi! Can you share a reproducible script?
What's your lightning and wandb versions?
From the stacktrace, it looks it's more of a wandb problem than lightning
yes, it looks like wandb problem, I will check with them
@lzziuhh Hey! a member of W&B team here. Thanks for reporting this. Is this reproducible for you? Is the script crashing after 10 epochs always? If so, this might be a problem with YOLOv5 evolve integration. I'll take a look and get back to you
Thanks for reply. Yes, this is reproducible. it is always crashing after 20 epochs , at the beginning of 3rd round of the evolve.. It does not happen when use evolve without W&B.
thanks & regards Liu
2022ๅนด1ๆ10ๆฅ(ๆ) 19:34 Ayush Chaurasia @.***>:
@lzziuhh https://github.com/lzziuhh Hey! a member of W&B team here. Thanks for reporting this. Is this reproducible for you? Is the script crashing after 10 epochs always? If so, this might be a problem with YOLOv5 evolve integration. I'll take a look and get back to you
โ Reply to this email directly, view it on GitHub https://github.com/PyTorchLightning/pytorch-lightning/issues/11373#issuecomment-1008737807, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG5HF7V25OA3LVNQB4TXTT3UVKYZ5ANCNFSM5LRKN2OA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you were mentioned.Message ID: @.***>
@lzziuhh thanks. I'll look into it this week
๐ Bug
Description
I try the YOLO5 evolve to find the best hyperparameter. the error raised at the third generations. I retried but happen again so I think it should be a bug.
Environment
Colab pro+
Error Log
cc @awaelchli @morganmcg1 @AyushExel @borisdayma @scottire