Output prediction resolution & PyTorch Lightning `predict` behavior

[Man4231]

Hi Damien,

I've modified the code a bit to ingest a different dataset. and I have a few questions

• The number of points that are inputs don't match the number of outputs when doing the pre-processing. I've narrowed this down to the grouping that is done in the pre-processing. Is there a way to infer on the entire dataset and not just the mean of the points per voxel?
• Is it because it's just modified from the notebook?
• How do I use the predict function in pytorch lightning, I've seen that the eval.py is just stock standard - is there an implementation for prediction for an entire dataset rather than a subset?

Thanks!

[drprojects]

Hi @Man4231, thanks for your interest in the project and for your detailed feedback. I will address your points in order:

Output prediction resolution

Indeed, the resolution of the level-0 points in the training and inference data is not the raw point cloud resolution. As you noticed, as common practice in the field, the data is voxelized at preprocessing time using GridSampling3D. This allows mitigating point cloud size, density, and desired level of detail.

However, I draw your attention to the fact that measured semantic segmentation metrics (ie mIoU, OA, mAcc) and losses (ie cross-entropy, Kullback-Leibler) are all computed on the full resolution. To this end, we maintain histograms of labels inside voxels and all superpoint levels all through the pipeline.

Still, if you want to produce a full-resolution output, you have two options:

• for each full-resolution point, compute its nearest neighbors among the lower-resolution voxels. You can have a look at the following function for inspiration.
• at preprocessing time, compute and store the mapping between full-resolution points and the voxels they are assigned to. At inference time, load the relevant mapping and index the voxel-wise predictions with it. This would be faster than 1-NN search, but is not supported in the project yet. I intend to work on this in the future but won't have time to do so before mid-August. Please let me know if you would be interested in such feature.

PyTorch Lightning predict behavior

As of now, the pipeline supports train, val, and test modes on a dataset divided accordingly, but we have not implemented the Lightning predict behavior. Unfortunately, I do not have enough coding bandwidth to work on this at the moment. If you are familiar with Lightning and would like to see such feature in the project, we would gladly welcome a pull request ! I could provide guidelines on how the project works if need be. :wink:

Best,

Damien

[drprojects]

I take note of your interest for:

• easy full-resolution inference
• PyTorch Lightning predict behavior

I cannot work on this at the moment, but leaving this issue open in case I find the time next month.

In the meantime, feel free to send us a pull request if you work on those :wink:

[xbais]

Hi Damien, I am interested on working on the full resolution inferences for semantic segmentation on DALES since I will be needing it for my work. I would love to make a PR here if it succeeds :star2: . I am assuming you haven't yet started working on it. But I may need some help understanding parts of code here and there. Is this thread the right place for it?

PS : I may not be as well versed with some concepts / coding schemes, hope you will bear with me for a bit :grimacing: .

[drprojects]

Hi @xbais (I see you everywhere these days :wink: ) ! Thanks for offering your help ! You can try setting this up for the fun of it, but I am working on it, so if you guys can wait just a little bit, I will release it as part of a new version for the repo in the coming days. The new version will also include the implementation of our recent paper on large-scale panoptic segmentation.

Sorry for the wait, I know people have been asking this for a while ! :sweat_smile:

[xbais]

Wow, its cool if you are already on it! I choose to work on it anyways for the fun of it :smile: . Congratulations for your new paper on large scale panoptic segmentation :tada::tada: ...hope to read it soon!

[pyarelalchauhan]

Sorry for the wait, I know people have been asking this for a while ! 😅

Hi @drprojects ! Are you planning to release the code full resolution prediction soon 🔜 ? ⏳

[drprojects]

Hi @drprojects ! Are you planning to release the code full resolution prediction soon 🔜 ? ⏳

Yes, very soon. Like today, normally. Just setting up a new feature related to #72 and will release everything :wink:

Thanks for your patience !

[drprojects]

Hi all, I finally released the v2 of this project, which should address this issue :partying_face:

Closing this issue for now, feel free to re-open if need be.