Display the idyntree-visualizer in a jupyter cell

[GiulioRomualdi]

Given the recent achievement in blf (https://github.com/dic-iit/bipedal-locomotion-framework/pull/366), I would like to use the idyntree-visualizer to write some simple examples.

I would like to implement the examples using the python bindings of blf (currently under development https://github.com/dic-iit/bipedal-locomotion-framework/pull/303)

In the meantime, I tried to run the visualizer in jupyter. I notice that the visualizer is opened in a different window.

https://user-images.githubusercontent.com/16744101/125768008-9fcd2438-6021-4238-b1f1-10960209d4f6.mp4

On the other hand, I also played with meshcat and crocodyl. Running this example, I noticed that meshcat can be embedded directly in a jupyter notebook.

https://user-images.githubusercontent.com/16744101/125773247-4df8c896-8ec2-491c-9623-05c1278e4845.mp4

Since I don't have any experience with jupyter do you know how complex could be to add the possibility to embed the idyntree-visualizer inside a jupyter cell?

[traversaro]

Super fast assessment, I did not checked in detail the feasibility. Irrlicht is quite an old rendering engine that is written in C++ and uses OpenGL for the rendering backend, so whatever process is callng the iDynTree API when you call init is calling the right operating system API to create a window and populate it with OpenGL calls.

What is happening instead when you render Meshcat in a jupyter cell, is that the browser process loads a Javascript-based based rendering engine (in particular, https://threejs.org/) that is then communicating over WebAssembly and ZMQ with the Python interpreter process that is calling the Meshcat Python bindings.

So, something needs to change in the whole idyntree visualizer stack to permit the Python process to communicate with the browser. Irrlicht, differently from more modern rendering engine such as Magnum ( https://magnum.graphics/showcase/ ) or Ogre ( http://wasm.continuation-labs.com/ogredemo/ ) does not support running inside a browser, so among the thing that needs to change I think there is irrlicht iself (unless you want to add support for running insid a browser to irrlicht, that is however non trivial I guess, even as Irrlicht upstream is not really active, as the latest release was 4 years ago).

So, probably the easiest way to proceed if you want to keep the existing idyntree visualizer API in a jupyter cell is to add a meshcat-based backend to the iDynTree Visualizer API, otherwise the alternative if you are ok in using a different API is directly to use the crocodyl.MehsCat display or some other similar already existing API.

[S-Dafarra]

The only way I noticed to avoid the creation of a new window in irrlicht, is to provide a WindowId when initializing it. See https://irrlicht.sourceforge.io/docu/structirr_1_1_s_irrlicht_creation_parameters.html#af287810d910a23f8f7db98cef87b6eae. This should be set in the SIrrlichtCreationParameters when calling the createDeviceEx method in https://github.com/robotology/idyntree/blob/9285f4a5c10a05d80b9a70f790db876817865009/src/visualization/src/Visualizer.cpp#L242-L256.

Basically, instead of creating a new window, it renders in an already created one. I don't know how this would work with a Jupiter cell.

[traversaro]

I think it may help if irrlicht supported a remote webgl visualization as a backend, but without that I am afraid it is not directly helpful.

[GiulioRomualdi]

I think it may help if irrlicht supported a remote webgl visualization as a backend, but without that I am afraid it is not directly helpful.

So something like this one: https://www.ambiera.com/copperlicht/

https://www.reddit.com/r/gamedev/comments/2lno5o/copperlicht_webgl_3d_engine_now_free_and_open/

[GiulioRomualdi]

Given #900 I think we can close this issue