Train shape detection on Isaac sim data [In Progress]

[MinhxNguyen7]

Summary

Train YOLO shape detection on Isaac data, translating the data with the script in isaac_to_yolo. However, I did not realize that this model is actually only supposed to detect shapes, not letters, and I've been training on both because that's what the script outputs.

TODO

• [ ] Edit dataset formatting code to add option to only output shapes.
• [ ] Maybe generate new data.
• [ ] Train again on new dataset.

Test Plan

TBA

Issues

Closes #15

[EricPedley]

FYI training the model on both letters and shapes is fine, it just requires us to rework our pipeline a bit so that object detection returns box candidates with shape, letter, or both labels and we have classifier models to fill in the blanks. But I guess to keep it simple we can limit the scope of this issue to just training shape detection.

[MinhxNguyen7]

FYI training the model on both letters and shapes is fine, it just requires us to rework our pipeline a bit so that object detection returns box candidates with shape, letter, or both labels and we have classifier models to fill in the blanks. But I guess to keep it simple we can limit the scope of this issue to just training shape detection.

Comparatively, just doing shapes vs. shapes+characters is a similar amount of work; one requires tweaking the pipeline, and the other requires tweaking the data formatting. Considering our performance so far, I think going with just shapes might be a good move for now.

In any case, it's probably a good idea to parameterize what classes to include in the formatted data, so I'll just work on that., then training the model on the newly-formatted data.

[MinhxNguyen7]

@tastysandvich The weights for the new model are up now. I trained a detection model, not segmentation, because we don't need segmentation now. However, the current pipeline assumes segmentation, so it doesn't work. Since we're going forward with detection -> classification, I'll refactor the code to use detection instead.

I think the last time I ran the tests on IRL data, I actually didn't use the new weights, which is why the numbers were so similar. How come we didn't think that there was a mistake when the numbers were exactly the same lol?

[EricPedley]

RIP 🪦

I'm surprised this does worse but oh well :/

I think one glaring problem is that the background in the isaac sim data doesn't vary that much because it's always that same scene.