Meshcat in COLAB

[jcarpent]

Dear @duburcqa,

Following the discussion https://github.com/RobotLocomotion/drakae/issues/12645, did you manage to get a colab session with MeshCat? If so, could you provide some pointers? We (with @nim65s) would be interested to watch and take inspiration from the solution.

[duburcqa]

Hi @jcarpent,

Yes I was able to make it "work", in a sense that it is functional. I'm sending/receiving messages to the javascript viewer in colab directly through the web/zmq sockets of the ipython kernel. For now the solution is hacky because I'm using a very low level and undocumented communication protocol that can be changed from one version of ipython to another (and it does unfortunately). So far I'm sending so many messages that the kernel blocks them at some point and the whole colab session freezes. On a local machine it can be avoided by setting the iopub maximum rate, but it is not possible on colab as far as I know. Also, this solution does not work for some Jupiter interfaces, as the one embedded we VSCode at some point, but I think it works now since they reworked their implementation to be closer to a usual local jupyter session.

[traversaro]

@jcarpent for the Python part we have something that took inspiration from drake and pinocchio in https://github.com/robotology/idyntree/pull/905, thanks to @GiulioRomualdi . See https://github.com/GiulioRomualdi/idyntree-meshcat-notebook for an example.

[duburcqa]

So the main points to improve are:

• the portability (being compatible with any, or at least most ipython versions)
• deal with the iopub rate issue on colab

And eventually implement a proper ipython widget, with an documented communication protocol and so on, instead of many wires everywhere ready to break at any time.

[duburcqa]

What I want to emphasize, is that contrary to any existing solution, I'm really using the existing ipython kernel sockets for communicating with the web UI. As a result, there is nothing special to setup, no firewall issue, no ssh issue, no port issue or VPN issue whatsoever. If you can access a jupyter session, it is guaranteed to work out the box whatever the environment.

But the cost to pay is to comply with the (undocumented) limitations and restrictions that applies to those sockets.

[jcarpent]

What I want to emphasize, is that contrary to any existing solution, I'm really using the existing ipython kernel sockets for communicating with the web UI. As a result, there is nothing special to setup, no firewall issue, no ssh issue, no port issue or VPN issue whatsoever. If you can access a jupyter session, it is guaranteed to work out the box whatever the environment.

But the cost to pay is to comply with the (undocumented) limitations and restrictions that applies to those sockets.

@duburcqa Could you provide quick guidance to reproduce it?

[duburcqa]

@duburcqa Could you provide quick guidance to reproduce it?

Unfortunately not really. I have heavily customized meshcat to fit my needs and know it is hard to compare it with the original one regarding the low-level implementation. All the stuffs I implemented regarding meshcat can be found here. The implementation of IPython bridge for meshcat client is implemented in this file and how to communicated with it through ipython kernel can be found here.

But clearly I agree with @RussTedrake when he says: "meshcat that work only through kernel comms, but Colab / Deepnote / etc all offer their own implementations (if at all) of the kernel comms, so writing one clean meshcat implementation that works for all of them is likely impossible.".

That's why I'm no longer putting myself a lot of effort into it, because it has some limitations I can hardly see how to overcome. So for now I gave up with being able to have a nice and portable simulation web viewer for jupyter, vscode and others. For now I'm rather considering using the webgl port of Panda3d, but I'm not convinced it could really solve the limitation I have with meshcat.

[duburcqa]

To wrap up, the only way to get a portable solution working with ipython environments (self-hosted or not) is to communicate through ipython kernel. But doing so it complicated because it heavily relies on the javascript interface with the kernel provided by the every platform (jupyter provides the most advanced interface, colab is quite nice with enough features for meshcat to work properly, but not kaggle nor VSCode at some point), and the name of the methods and how to use them is specific to any of them. So it makes it very hard to get something very portable and highly compatible for every platform. The other issue is that the iopub is limited, so it would requires to rework the way messages are send to gather them whenever it is possible instead of send them separately systematically.

So it is quite a lot a work at the end, and what I implemented is just a POC, only working reliably in self-hosted jupyter environment and custom iopub rate limit.

Hope this helps.

[duburcqa]

If you are still interested to work in this direction, I could eventually make some time to help you on this since it is definitely a feature I would love to see.

[jcarpent]

Thanks @duburcqa for your feedback. It seems @nim65s has found a flexible solution to be deployed soon.

[duburcqa]

No problem. Which kind of solution? it sounds really interesting!