search

danbider / lightning-pose

Accelerated pose estimation and tracking using semi-supervised convolutional networks.
MIT License
223 stars 32 forks source link

Pose-app crushed at the last moment of training. #155

Open penguinflux opened 1 month ago

penguinflux commented 1 month ago

Hi, every time I run training via Pose-app GUI, it crashes at the last moment before finishing, like the figure attached. The trained model will be kept and could be used to predict new videos. However, I do not know its impact on models trained. Would you give me any suggestions about this? Thank you!

image image

themattinthehatt commented 1 month ago

Hi @penguinflux thanks for raising this issue, looks like an issue with FiftyOne when there are no ground truth labels on a frame. I will look into this soon.

themattinthehatt commented 1 month ago

@penguinflux apologies for the delay - I'm looking at this now and actually I don't get any errors even if every single ground truth label is NaN. Is it possible that you were labeling frames while this model was training? In that case the ground truth labels would have been updated but the final predictions (computed on the dataset created at the beginning of training) would not be.

themattinthehatt commented 1 week ago

@penguinflux checking in on this, did you see my above message?