search

open-forest-observatory / geograypher

Multiview Semantic Reasoning with Geospatial Data
BSD 3-Clause "New" or "Revised" License
10 stars 4 forks source link

Consider batching rendering operations #12

Closed russelldj closed 2 months ago

russelldj commented 8 months ago

One of the most expensive steps is computing the pixel to face correspondences using pytorch3d. This can be done for a batch of viewpoints at once. Given the capacity of the larger Jetstream instances, this could speed things up substantially by using all available compute.

russelldj commented 8 months ago

I've been working on this and getting indexing errors. It's possible that this was due to already-handled issues with non-uniform image sizes but could also be due to something about how the indices are represented in batched meshes. It requires further attention.

russelldj commented 3 months ago

To summarize our conversation. This issue is slightly more broad than rendering, it's about any time we want to compute pixel-to-face correspondences. A good place to start would actually be in the aggregation (or "splatting") stage. You could use the concept_figures notebook to experiment with this. Specially, you'd want to use the cell called Show aggregated viewpoint and update aggregate_viewpoints_pytorch3d to have a batch_size > 1. If this problem hasn't mysteriously resolved itself, you will most likely get an index error around here.

I've found that debugging in jupyter can be difficult because some error messages are hidden or harder to see and tools like breakpoint()-based interactive debugging (link) don't work. You can export a jupyter notebook to a python script in VSCode using more actions (...) -> export. If you run it as a script, you might have to use a CACAO virtual desktop so the visualizations (that would normally show up inline in the notebook) can be displayed.

If you're having issues using the concept figures notebook, you could instead try the geograypher/entrypoints/aggregate_images.py script and edit the aggregate_viewpoints_pytorch3d method there. This should run properly with the example data without setting any command line arguments. This approach is more streamlined, but since it uses large, realistic data, it might take a little longer to run. But it's not too crazy.