search

google-research / kubric

A data generation pipeline for creating semi-realistic synthetic multi-object videos with rich annotations such as instance segmentation masks, depth maps, and optical flow.
Apache License 2.0
2.33k stars 230 forks source link

Add Kubric asset from existing Blender object in scene #242

Open gabriel-v opened 2 years ago

gabriel-v commented 2 years ago

There are cases in which it's simpler to create the desired object using Blender operators (append and object.copy()) instead of the provided kb.FileBasedObject.

Sometimes the object we want is in the custom_scene, and this comment accurately explains that objects in the custom_scene are not known to Kubric.

Using the Blender methods above means the resulting object is not known to Kubric, will appear as black in the segmentation map, and will be missing from metadata.

I would like to implement a method that allows registering arbitrary Blender objects from the scene with Kubric. I suspect this means overriding the _add_source similar to this one, to allow for a Blender Object (or ID string) to be added to a Kubric scene.

Does this new method require anything else than just calling the register_object3d_setters function and calling obj.observe for the remainder of the parameters, just like in the kb.FileBasedObject method?

MrXandbadas commented 2 years ago

Hi there, I'm just someone who likes to read the code and try to help xD I am not a developer or contributor to the Kubric Paper or anything that leads to it.

Although, I have been looking into getting custom scenes working in Kubric from a wholly "imported" standpoint. The main issue is ensuring every object has a watertight mesh. (Comment here) Some of the comment is below for context:

To make this work for simulation you would need to create a collision mesh in obj format and a corresponding urdf file that you can pass as simulation_filename. If your mesh is watertight, you can take a look at the GSO preprocessing script about how to do that. If not, then things get complicated, and you will have to do more than that. The ShapeNet preprocessing scripts can serve as a starting point there.

The CSO Preprocessing Script seems to take the object through multiple stages formatting the data to a specific format before eventually making it a AssetSource

It is my best assumption that the best steps forward would be to build a tool that can dynamically handle imports and successfully push them through the pipeline to be passed to the _add_source method just like the KubricAssets.

In my opinion (Please devs correct me if I am wrong) the above mentioned and quoted comment is more of a guide of what type of toolset we need to construct to enable our data to be transformed and accepted by the pipeline.

For instance the thing I've been playing with loosely recently is importing a .blend file object by object into a kubric scene as it seems the next logical step in the pipeline to enable mass adoption of the Pipeline in State of the Art usability. Further confirmed by your post here. Its painful though. I'm not very adapt in programming. in convert.py I added the following at the bottom

def repair_mesh(object, datadir, model_name, stages=None):
  # Re-create the stage functions using dynamic files
  # --- setup logger (→stdout)
  logger = logging.getLogger(__name__)
  logger.setLevel(logging.DEBUG)
  handler = logging.StreamHandler(sys.stdout)
  logger.addHandler(handler)

  # --- execute functors
  object_folder = Path(datadir)/model_name
  stages = [int(stage) for stage in stages]
  if 0 in stages: stage0(object_folder, logger)
  if 1 in stages: stage1(object_folder, logger)
  if 2 in stages: stage2(object_folder, logger)
  if 3 in stages: stage3(object_folder, logger)
  if 4 in stages: properties = stage4(object_folder, logger)
  if 5 in stages: stage5(object_folder, logger)
  if 6 in stages: stage6(object_folder, logger, category_id="test", asset_id="asset_id")

in asset_preprocessing i added a new line at line 96:

shapenet2kubric.convert.repair_mesh(obj, "examples/KuBasic/", str(obj.name), ["0","1","2","3","4","5","6"])

Very basic implementation and its broken for the example i'm working on. But its a start xD

I'm very much against changing a lot of the source code but that's just my preference. I don't like to redo hard work that's been done unless absolutely necessary.

Does this new method require anything else than just calling the register_object3d_setters function and calling obj.observe for the remainder of the parameters, just like in the kb.FileBasedObject method?

Maybe Asset_source is the place to create some kind of forign_asset function 🤷

I'm Exciteddddddd!!!!