3D UI Elements

[gonzalocasas]

Description

I would love to be able to display 3D objects / meshes in a UI Element. It could be a "Viewport" UI element, or perhaps a more specific one that only shows a single object / mesh.

Suggested solution

Without knowing specific details about how marimo UI elements need to be implemented, I would assume leveraging three.js would be a good starting point for an idea like this.

Alternative

No response

Additional context

No response

[akshayka]

Thanks for the request! Would you want the data in the plots to be selectable (and selections sent to Python)? Or is just visualizing the 3D objects enough?

[gonzalocasas]

I think just visualizing the objects would be already more than super useful for a lot of use cases. Eventually being able to pan, move and rotate the viewport would be great.

[gonzalocasas]

Maybe I should add more context: I'm one of the maintainers github.com/compas-dev/compas, an open source geometric lib, usually used integrated into a cad but it would be amazing if people could create apps (marimo apps) that do geometry manipulation and processing and the output can be visualized nicely and stand alone without needed a CAD (the library is already cad agnostic so if marimo would support these, then adding an integration world be trivial.

[mscolnick]

compass looks cool! what is the output for compass (or other libraries like this)? does it output HTML that also opens a viewer that renders the HTML? or does CAD have its own markup language?

I could definitely see marimo-compass plugins (and other 3D renderers) being a 3rd party plugin, once we release our plugin api.

[gonzalocasas]

what is the output for compass (or other libraries like this)?

it can output mesh files in different formats (obj/stl/gltf/etc), but we could also write a plugin that directly generates the necessary javascript/html that you'd need to integrate into marimo. I can definitely contribute to that.

[mscolnick]

We can render most javascript/html using mo.Html - does the generation of obj/stl/gltf/etc happen in Python? Could we test it out just using mo.Html, and then judging by the complexity, we can push this into a plugin mo.ext.gltf().

This would be a great example of a 3rd-party plugin since may be a bit too bespoke for "core". But since we don't have an API for it, we can maybe scope it under ext or experimental and move later.

Happy to help along the process of developing the widget.

[gonzalocasas]

does the generation of obj/stl/gltf/etc happen in Python?

yep, it does.

This would be a great example of a 3rd-party plugin since may be a bit too bespoke for "core". But since we don't have an API for it, we can maybe scope it under ext or experimental and move later.

Happy to help along the process of developing the widget.

cool! yes, please point me in the right direction and I'll give it a go

[mscolnick]

@gonzalocasas are you comfortable hopping in the discord and we can chat more there: https://discord.gg/rAhpfKwuaN

[albertoferna]

Hello,

First of all, congratulations on this nice notebook system. I just wanted to point out another use case. We often work with 3d data that represent fields (both vector and scalar fields). We often use pyvista within jupyter notebooks. It would be great to be able to use it with marimo to quickly create apps. Pyvista uses trame and vtk.js under the hood. I assume it should be posible with some effort to do the same with three.js since it used to be an option with pyvista. A scene from pyvista can be exported to html; thus, I've tried to get it to work within a cell using mo.Html. So far though I have not been successful.

[mscolnick]

Given the integration with anywidget, you can now achieve 3D renderings quite easily with popular frontend 3D rendering libraries.

For example, with with three.js: https://marimo.app/l/jg88wv

I am going to close this, but feel free to re-open if there are specific things marimo can help with when building 3D UI elements with anywidget