search

johko / computer-vision-course

This repo is the homebase of a community driven course on Computer Vision with Neural Networks. Feel free to join us on the Hugging Face discord: hf.co/join/discord
MIT License
454 stars 142 forks source link

Synthetic Data Creation - Proposed Outline #36

Closed kfahn22 closed 10 months ago

kfahn22 commented 11 months ago

INTRODUCTION (introduction-to-synthetic-data.mdx)

GENERATING SYNTHETIC DATA (generating-synthetic-data.mdx)

PRACTICAL APPLICATIONS & CHALLENGES (practical-applications-and-challenges.mdx)

HANDS-ON DEMOS

RESOURCES (resources.mdx)

lunarflu commented 11 months ago

Love the Practical Applications and Challenges section!

johko commented 11 months ago

Nice outline, I think this is one of the chapters where I will learn a lot. What about 3D engines like Unity or Unreal (don't know if that is actually used), they might be interesting for more complex environment simulations.

Other than that: sounds good too me! :slightly_smiling_face:

lunarflu commented 11 months ago

What about 3D engines like Unity or Unreal

Happy to ask HF members about this if you want, iirc Dylan + Thomas were working on stuff related to Unity (might be more RL focused, but who knows)

kfahn22 commented 11 months ago

We are investigating whether we can use Pytorch3D for the complex environment simulations with the ShapeNetCore dataset (already on HF hub but need permission to use) or the Amazon Berkeley Objects dataset. I would appreciate feedback as to whether anyone is aware of plans to add ABO to the hub (I searched and didn't find it), or whether we could add a small subset (the database is 156GB and contains 147,702 products.

lunarflu commented 11 months ago

plans to add ABO to the hub

Asking about this now 🧐

lunarflu commented 11 months ago

We are investigating whether we can use Pytorch3D for the complex environment simulations with the ShapeNetCore dataset (already on HF hub but need permission to use) or the Amazon Berkeley Objects dataset. I would appreciate feedback as to whether anyone is aware of plans to add ABO to the hub (I searched and didn't find it), or whether we could add a small subset (the database is 156GB and contains 147,702 products.

@kfahn22 Hey Kathy, feel free to upload the ABO dataset (or a subset) yourself to the HF Hub, then we can take care of moving it to the proper org if necessary. Does that work for you?

kfahn22 commented 11 months ago

Yes, I will look in to it.

kfahn22 commented 11 months ago

I have the tar file of the ABO 3d models, but I have only opened a very small fraction of the files as there are 63 product categories w/ 3d models and almost 8K 3d models. At this point, I have 5 models (.glb) for major categories of furniture (sofa, end table, chair, bed, dining table, desk, etc.) I also have a small set of textures from ambientcg. I was thinking of trying to use BlenderProc (library written in python on top of Blender for synthetic data generation), but just saw that we should use PyTorch, so will try to work with PyTorch3d.

I would appreciate feedback on how many ABO models should be added the the hub? I have 5 for a relatively small number of categories at this point. I am not sure how many it would be useful/feasible to add.

Zekrom-7780 commented 11 months ago

Hello, I'm considering working on 2D/3D image and point cloud processing, primarily using the Point Cloud Library (https://github.com/PointCloudLibrary/pcl).

I've noticed that this repository is entirely in C++. Is this programming language acceptable for this project?

Additionally, I believe this work would be an addition (a pretty useful one I believe, but an addition nonetheless) as a new subsection in this chapter. What are your thoughts on this proposal?

johko commented 11 months ago

Hello, I'm considering working on 2D/3D image and point cloud processing, primarily using the Point Cloud Library (https://github.com/PointCloudLibrary/pcl).

I've noticed that this repository is entirely in C++. Is this programming language acceptable for this project?

Additionally, I believe this work would be an addition (a pretty useful one I believe, but an addition nonetheless) as a new subsection in this chapter. What are your thoughts on this proposal?

@Zekrom-7780 Good question. Actually all pointcloud operations I have done were also with PCL and C++, but I'm not sure if it fits this course, as it won't be runable that easily as e.g. a jupyter notebook. I know that the PCL is one of the most extensive libraries for this but maybe we should stick with Python alternatives for the course and just mention PCL. But I have to admit I'm not that deep into the topic anymore. Maybe you can also ask in the #3d channel in discord, people there might now more about current Python packages (apart from PyTorch3D)

johko commented 11 months ago

I have the tar file of the ABO 3d models, but I have only opened a very small fraction of the files as there are 63 product categories w/ 3d models and almost 8K 3d models. At this point, I have 5 models (.glb) for major categories of furniture (sofa, end table, chair, bed, dining table, desk, etc.) I also have a small set of textures from ambientcg. I was thinking of trying to use BlenderProc (library written in python on top of Blender for synthetic data generation), but just saw that we should use PyTorch, so will try to work with PyTorch3d.

I would appreciate feedback on how many ABO models should be added the the hub? I have 5 for a relatively small number of categories at this point. I am not sure how many it would be useful/feasible to add.

@kfahn22 I think it is actually fine to not use PyTorch for Synthetic Data Generation, as it is a bit of a special topic and not directly related to model training (of course if you have a part of training/fine-tuning it would be great to use it). For all processing and data creation tasks you can use any python library you see fit (I know that PyTorch3D can be a bit challenging :sweat_smile: )

kfahn22 commented 11 months ago

Thanks @johko, I will try to use BlenderProc as it is a library that is specifically for synthetic data creation, and has some really nice features. It is a little tricky to run the scripts in Google Colab, though, and I am going to try to figure out a way to make the process a bit easier.

kfahn22 commented 10 months ago

@johko @lunarflu I just wanted to give an update. The tar file with the ABO models is around 156 GB, so I have been very selective about what I have pulled out. The ABO models are in 16 folders (0-9, a-f), but I have yet to figure out the over-arching organizing structure (outside of the fact that the product number ends in the file number or letter). I have tentatively reorganized the 3d models into folders by category (13 at present) and am writing a json file with a field with the original path for reference. BlenderProc already has an example random room generator script, but it uses the IKEA dataset which is no longer publicly available. I am working on editing their script to load ABO instead. Assuming I can figure that out in a timely manner, I will make a pull request to edit the BlenderProc example (currently working on scaling as I loaded one of the ABO beds and it is not scaled properly).

johko commented 10 months ago

Thanks for the update @kfahn22 . Sounds like you're learning a lot right now, which is very cool.

Just make sure that you don't spend too much time on single topics and keep in mind what you want to achieve. Sometimes it easier to take a step (or two) back and go another way too reach a goal, I'm speaking from experience there :wink:

kfahn22 commented 10 months ago

@johko I totally agree and I had already decided that I would not spend too much more time on this!!

lunarflu commented 10 months ago

@johko I totally agree and I had already decided that I would not spend too much more time on this!!

Building off what @johko said, it can be helpful to solidify what you learn into concrete chunks / commits! This week we're aiming to shift a bit of theory into some concrete beginnings of units, so that could be a nice goal 🤗

Zekrom-7780 commented 10 months ago

Point-Clouds Section Outline and Content Feedback

Hey everyone,

I've put together an initial outline and some content for the Point-Clouds section here. I'd love your input and feedback on it.

Specifically, I'm seeking help with the Registration and Alignment section. I'm not entirely sure about what content to include there, so any insights or suggestions would be greatly appreciated.

Feel free to leave comments directly in the document or provide feedback here. Thanks in advance for your help!

Here's the link in case the above one doesnt work xD

@johko @lunarflu @merveenoyan @NielsRogge @MKhalusova Could you PLEASE LOOK INTO IT, and add feedback??

MKhalusova commented 10 months ago

@Zekrom-7780 I'm not familiar with point-clouds, but I've looked at the doc you've shared and the suggested structure looks solid to me.

merveenoyan commented 10 months ago

@Zekrom-7780 I agree with Maria, it looks good to me.

lunarflu commented 10 months ago

LGTM! For the PRs, let's keep them in bite size chunks! 🚀