search

PJLab-ADG / neuralsim

neuralsim: 3D surface reconstruction and simulation based on 3D neural rendering.
MIT License
582 stars 31 forks source link

running with custom dataset #29

Open xslittlegrass opened 11 months ago

xslittlegrass commented 11 months ago

I'm trying to run StreetSurf with my own dataset (only posed camera data) and have a few questions and I'm wondering whether you can help with that.

I'm doing these steps:

  1. Convert my data to the the data format described in the doc. I populated the camera data in the 'observers' fields ('hw', 'intr', 'c2w' etc).
  2. Run the depth/normal and mask prior extraction following the doc.
  3. Run training with the config withmask_nolidar.230814.yaml. I updated camera_list, the root in the dataset_cfg and scenarios in scene bank_cfg accordingly and others unchanged.

I have two questions:

  1. In the doc, the 'ego_car' is commented as optional, but when I run train.py it complains about missing key scenario['observers']['ego_car']['data']['v2w']. Since I already provided 'c2w' in the camera data I'm wondering why vehicle pose is still required? My cameras are not synced so what vehicle poses should I use?
  2. The ego car hood takes a large part in the front camera image, is there a way to consume the ego car mask? I'm cropping the images for now but some valid pixels are lost in this process.

Also one minor thing: The doc reads scenario['metas']['num_frames'] but the code seems to require scenario['metas']['n_frames'].

hihulan commented 11 months ago

I have met the same problems, the ego_car is necessary, and is the trajectory of the body frame; for problem two, you can add the function def get_ignore_mask() in your own *_dataset.py, which has the definition in the dataio/dataset_io.py, and add the param in your own config yaml , under "rgb_ignore_mask"