Generate Complete Isaac Sim Dataset and Document It

[MinhxNguyen7]

Nvidia Omniverse should be able to generate instance segmentation data. Currently, the generation code generates semantic segmentation data.

This may be the case with other data that we need. The data should be precisely enumerated below.

Tasks

• [x] Run Isaac Sim locally
• [x] Enumerate the data that we need.
• [x] Generate data.
• [ ] Verify that the output is well-formed and contains information that we need.
  • [x] Instance semantic segmentation by shape. We can convert to instance segmentation, but we only need to change a param to BasicWriter
  • [x] Letter classification.
  • [ ] Color classification
  • [ ] Camera flight path.
  • [x] Camera intrinsics
  • [ ] "Absolute" object locations
• [ ] Document how the data generation works and how the output is formatted.
• [x] Write an interface for the data

Related issues

Issue about generally what data is missing: #49. See this for more information/problems. Train yolo instance segmentation: #15 Replace test images with new sim data: #41

[MinhxNguyen7]

@EricPedley @kvanuy @scliou

Do you guys mind checking if I have listed all the data that we need? If I haven't, can you add it? Thanks

[EricPedley]

We also want the ground truth 3d position of each target, in the same coordinate system as the camera flight path, and yeah the camera intrinsics (FOV or focal length in pixels). For the color, the semantic segmentation is sufficient because our current approach just tries to segment the letter vs. shape via clustering, then classifies the color based on the colors of the centroids of each of the clusters in rgb space.

[MinhxNguyen7]

Isaac completely fails to launch with the recent hotfix on my machine. It seems like @Dat-Bois is having similar problems as well.

We might want to reassess our posture on Isaac unless we can get it running on the lab PC.

[MinhxNguyen7]

New preliminary dataset has been generated by @Vince-C156 here: https://drive.google.com/drive/folders/18vFwjKqfvJEpGB1ood7JKJ3N_oPJ24_A

From what I can see, we have semantic segmentation, but not instance segmentation. The code currently indicates that we do instance segmentation for shapes, but the data only includes semantic segmentation. I can't imagine that we need to do instance segmentation since we can just assume that two shapes won't overlap. @EricPedley? If that's the case, we're only missing color, from the quick look that I took. I haven't checked whether the data's well-formed. Not sure if I'll have time to check everything, so it'd be good if whoever needs a certain part of the data just looks at the part that they need.

[MinhxNguyen7]

I've written an interface for the synthetic data here with instructions added in the README. It's not ready to be used for training yet, it's only to easily read and validate the data. It should be easily convertible when that comes time.

From looking at a few images, it seems that the existing images and labels are well-formed.

[EricPedley]

From what I can see, we have semantic segmentation, but not instance segmentation. The code currently indicates that we do instance segmentation for shapes, but the data only includes semantic segmentation. I can't imagine that we need to do instance segmentation since we can just assume that two shapes won't overlap. @EricPedley? If that's the case, we're only missing color, from the quick look that I took. I haven't checked whether the data's well-formed. Not sure if I'll have time to check everything, so it'd be good if whoever needs a certain part of the data just looks at the part that they need.

We still want to be doing instance segmentation, but having the semantic segmentation and bounding box labels is sufficient to make instance segmentation labels with a script.

[Vince-C156]

For camera paths we can use Pegasus sim through Ros2 https://github.com/PegasusSimulator/PegasusSimulator

[Vince-C156]

We also want the ground truth 3d position of each target, in the same coordinate system as the camera flight path, and yeah the camera intrinsics (FOV or focal length in pixels). For the color, the semantic segmentation is sufficient because our current approach just tries to segment the letter vs. shape via clustering, then classifies the color based on the colors of the centroids of each of the clusters in rgb space.

The projection matrix is included in the camera parameters and as of now the camera is always pointed straight down at a height of 22m fixed.

Ideally ur right we should get the labels, someone should work on writing a custom writer using omniverse api to support it. Or even implementing this without that by using some of the variables in the code.

[MinhxNguyen7]

| We still want to be doing instance segmentation, but having the semantic segmentation and bounding box labels is sufficient to make instance segmentation labels with a script. We can do conversion, but it would be easier just to make BasicWriter write it. It only needs a param change (instance_segmentation=True). @Vince-C156, can you do that for the next revision of the dataset, whenever that is?

| For the color, the semantic segmentation is sufficient because our current approach just tries to segment the letter vs. shape via clustering, Would we still want color data to test that procedure though?

[MinhxNguyen7]

We'll just use sim data as is (without color labels).