search

googlecolab / colabtools

Python libraries for Google Colaboratory
Apache License 2.0
2.2k stars 722 forks source link

Adding support for Open3D visualization widgets #1981

Open germanros1987 opened 3 years ago

germanros1987 commented 3 years ago

Dear developers and maintainers.

The Open3D team has been working on extending the Open3D to support 3D visualization in jupyter and jupyter lab. We have created a set of widgets that support the display of point clouds, mesh, etc., with support for advanced lighting, PBR materials, etc.

We were surprised when checked that we couldn't make this work in a Colab environment. Then we discovered that arbitrary widgets are not permitted/supported in colab.

I think it would be very beneficial to start a conversation to see if there is a simple way to bring our widgets into colab. Would you be interested in this?

blois commented 3 years ago

Colab does not currently support third-party widgets, an early explanation of some of the issues is https://github.com/nteract/nes/tree/master/portable-widgets.

Colab does support displaying HTML & JS and Comms for bidirectional kernel communication. The typical model that we use is displaying HTML & JS with the default contents of the visualization which is available to later notebook viewers then establishing a comm channel to provide interactivity when connected to a kernel.

I'd be curious around thoughts on this. There may be some opportunities to reuse more of the widget code in Colab with a tradeoff of much larger binary size for the visualization though I haven't explored this approach deeply.

germanros1987 commented 3 years ago

@blois thanks for your response. We have been following the widget situation quite closely, but we still have a few questions. Not being able to use our brand-new 3D widgets is a bit of a limitation and we are trying to find alternative ways to make this work.

Is it feasible to leverage a default "canvas" widget and just fill it in with the outcome of our kernels (via comms)?

Any guidance is appreciated. Thanks!

germanros1987 commented 2 years ago

I just want to revive this topic since we keep hearing from users complaining about Open3D not running in Colab: https://github.com/isl-org/Open3D/issues/1026

Are there any updates in the philosophy of Colab to run 3rd party widgets?

blois commented 2 years ago

Can you provide a simple example of Open3D rendering with a Jupyter Widget which you would like to have working in Colab?

Colab has been flushing out support for third-party widgets- see https://github.com/googlecolab/colabtools/issues/498.