Unbundle gepetto-viewer dependency - alternative visualisers

[wxmerkt]

gepetto-gui is fairly brittle (requires nameserver restarts and manual deletion of files). Ideally, we'd like to have a viewer that works out of the box. Pinocchio is currently bundled so I think it would be good if we could swap in/place other viewers as desired (e.g., as options provided to the RobotWrapper).

While rviz may be a good alternative for people working with ROS, it's not the best for prototyping and general visualisation as generally done using Pinocchio.

A fantastic alternative (with the ability to view in Jupyter notebooks!) is Meshcat - and I think it's worth thinking about implementing it.

cc: @nmansard (per our discussion on Friday)

[jmirabel]

You are not the only one having troubles with the omniorb name server. I dig a bit and discovered that it is not too hard to get rid of the name service. I will try to implement it soon.

If you are familiar with Meshcat, it shouldn't be hard to hack RobotWrapper class to use it.

[jcarpent]

@wxmerkt We have been discussing this topic for a while (see #180) but without taking a final decision. We were mostly thinking on using Panda3d which is available on all the common OS and with Python bindings. For instance, we must be able to load dae files, and Panda provides such a parser. @wxmerkt What is your opinion?

[wxmerkt]

Thank you for the pointer to the previous discussion. We generally have bad experiences with VTK (version conflicts with Qt that are non-trivial to resolve, especially when used on machines with more dependencies, e.g., ROS, PCL, third-party libraries using VTK). Less important, the PreMultiply, PostMultiply operation order for transforms is not as straightforward as we'd hope for.

Panda3D looks promising, but requires binary installation. Is it apt-get-able? The advantage of MeshCat is that it's a thin ZeroMQ-wrapper with minimal dependencies. You can render in any web browser either on the same machine or remote or in Jupyter notebooks. Afaik it's commonly used in combination with RigidBodyDynamics.jl and has also been used to stream pointclouds which are heavy. Last I checked it didn't have Collada support but with WebGL that is five to ten lines to add (and I've previously done it for other projects where we used WebGL rendering).

[jcarpent]

Browser rendering seems a very nice feature. I would vote for it. I will have a try with MeshCat then.

[wxmerkt]

Thank you, Justin! I am happy to be your beta tester :-)

[aelkhour]

Ping @matthieuvigne who is also getting desperate with gepetto-gui :-)

[matthieuvigne]

Thanks, that sounds promising! I was indeed a bit disappointed by the performances of the latter releases of gepetto-gui, and I was thinking of trying to code something more simple and straightforward, getting rid of the corba server and using simple IP protocol. MeshCat seems to be doing just that, with ZMQ sockets. I'll give it a try and see how it goes.

[nim65s]

As I have been saying internally for a few years, I would be reaaaally happy if we could use ZMQ instead of Corba :)

[matthieuvigne]

I gave a quick trial to Meshcat, developping in python a basic example of what a "RobotWrapper" could look like. Here are the files for testing it on your own: it contains tow simple examples that display the Katana robotic arm in random configuration, using Meshcat or gepetto-gui.

katana.zip

I invite you to give it a try and share your feedback.

My thoughts so far :

• as a huge plus, Meshcat is extremely user-friendly and easy to integrate. There is no more dealing with corba servers, the integration in a browser is quite smooth (I didn't test with Jupyter notebooks, but that's a nice feature to have), and the resulting code is extremely short.

• The only truely missing feature I see to replace gepetto-gui (at least the way I use it) is the support of other CAD formats beside obj - namely STL and DAE. But I guess these can be added without too much difficulty.

• On the down side, I was looking for performance, but Meshcat seems quite slow - slower than the last version of gepetto-gui. The example I posted (5000 random configurations) runs on my PC in approximately 11s for gepetto-gui, against 19s for Meshcat. I was expecting more performance from a TCP socket. I believe the reason for this is that the meshcat client is written entirely in python, and takes a lot of time to convert the python objects before actually sending them. Indeed, after some benchmarking, I realized that all the strings sent were converted to Unicode, which is time consuming and useless in my case. After removing that, the benchmark dropped to around 12s for Meshcat, i.e. close to gepetto-gui, so it remains a good alternative I would say. Still, 60% of the time is spent in meshcat/visualizer.py/send. I don't know how much we can push the python code - but I don't see why sending 5000 x 9 lines over TCP would take 10s. Possibly things would be much faster if they were done entirely in C. In other words, keeping the meshcat server but rewriting part of the client in C++. It's something I consider doing, if only to test it out.

[matthieuvigne]

Concerning performances, here is something else to give toughts to. I'm only encountering performance issues when displaying large number of meshes (typically around 40) in gepetto-gui. So I tried to see how meshcat would behave when displaying more meshes - to do that, I simply displayed several copies of the katana arms. This shows that, as I add more meshes, the execution time of the meshcat example grows quite linearly (I tested for 1 to 6 robots and have a slope of 1.15). On the contrary, gepetto-gui scales quite badly: here I have a slope closer to 2.3 (though the results are less stable, for some reason the execution time of the same example script varies quite a lot with gepetto-gui). This means that, as soon as I start displaying two katana arms, meshcat becomes faster than gepetto-gui - a typical use case for me (in number of meshes) would be to display 4 katana robots, in which case meshcat is 40% faster than gepetto-gui. So even in terms of performances, meshcat might be enough for my needs.

[jcarpent]

@matthieuvigne Thanks for your quick feedback on MeshCat. The performances seem to be convincing.

@matthieuvigne Can you make a PR for the RobotWrapper in order to integrate the support of MeshCat? The ideal would be to keep the compatibility with the previous gepetto-gui, but to be also able to use MeshCat with a flag when initializing the display.

[jmirabel]

to do that, I simply displayed several copies of the katana arms.

This is also implemented in gepetto-gui but not used by Pinocchio. It could be done automatically though.

[jcarpent]

to do that, I simply displayed several copies of the katana arms.

This is also implemented in gepetto-gui but not used by Pinocchio. It could be done automatically though.

@jmirabel Can you add this feature too in Pinocchio?

[cmastalli]

@matthieuvigne very nice. It just missing the DAE and STL support

[wxmerkt]

Very nice and thank you so much for the benchmark @matthieuvigne. Adding DAE and STL support is easy - three.js has loaders for it and it's just another switch statement in the MeshCat code (I used to have that in a local copy but can't find it right now). Happy to add the DAE/STL support once the main PR is in :) Also agree with a C++ client interface...

[jmirabel]

https://github.com/Gepetto/gepetto-viewer/pull/77 enables automatic caching of the meshes.

[jcarpent]

@jmirabel What do you mean by automatic caching. It will load all for once the meshes, avoiding duplication?

[jmirabel]

It will not load twice the same file. It also implements https://github.com/Gepetto/gepetto-viewer-corba/issues/103

[jmirabel]

@matthieuvigne I bet in your case, the majority of the 60% of communication time is spent in creating the communications, rather than sending data because it only sends small amount very often.

Last time I had troubles with connection being slow with gepetto-gui, I solved it by using applyConfigurations rather than applyConfiguration, which sends several configuration at the same time. The case was slightly different because I was using two different computers. You can try updating the RobotWrapper to use applyConfigurations once, instead of N times applyConfiguration. Have it in mind too when you try to implement a C++ client for MeshCat.

[matthieuvigne]

Thanks for the feedback.

I'll start by integrating Meshcat support in RobotWrapper - @wxmerkt I'll leave you the STL/OBJ support. I'll then see as I use it - and as the changes you mentioned @jmirabel are merged in gepetto-gui - if performance is still an issue for my use case. If needed I'd give a try to a C++ implementation to see how much faster it can get, and depending on the results develop a full solution - but don't expect that too soon :) Of course, if other people share the concern of performance, this would motivate me to try to do it faster - otherwise it will remain a background pet project.

[jmirabel]

Modifying RobotWrapper to use applyConfigurations, use comprehension list and make the conversion in C++ (se3ToXYZQUATtuple), as show above, I divide the execution time by 2. It could be divided further by caching the list of objects and indices to send to the viewer.

Anyway, I think there is something wrong with the test. It does not make sense to send at more than 50Hz new object poses to a 3D renderer. No 3d renderer have been designed for that and you hit mutex (except if you want to generate a video, but that's a different matter and not the purpose of gepetto-gui. I don't know for MeshCat).

        if self.display_visuals:
            self.updateGeometryPlacements(visual=True)
            gui.applyConfigurations (
                    [ self.getViewerNodeName(visual,pin.GeometryType.VISUAL) for visual in self.visual_model.geometryObjects ],
                    [ pin.se3ToXYZQUATtuple(self.visual_data.oMg[self.visual_model.getGeometryId(visual.name)]) for visual in self.visual_model.geometryObjects ]
                    )

[jmirabel]

See branch conversions on my fork.

[traversaro]

• Still, 60% of the time is spent in meshcat/visualizer.py/send. I don't know how much we can push the python code - but I don't see why sending 5000 x 9 lines over TCP would take 10s. Possibly things would be much faster if they were done entirely in C. In other words, keeping the meshcat server but rewriting part of the client in C++. It's something I consider doing, if only to test it out.

Back in time I was investigating as well the possibility of having a C++ library for MeshCat for integration in iDynTree. I never progressed due to the lack of time, but I had some relevant links in https://github.com/traversaro/not-working-meshcat-cpp/issues/1 , perhaps they could be interesting for your work.

One of the things I was not sure about was if the C++ library should implement the ZMQ protocol used for the ZMQ <--> WebSockets logic contained in the meshcat-python, or directly open a custom webserver that communicates with the browser via WebSockets, similarly to what MeshCat.jl does since https://github.com/rdeits/MeshCat.jl/pull/28 .

[matthieuvigne]

Anyway, I think there is something wrong with the test. It does not make sense to send at more than 50Hz new object poses to a 3D renderer. No 3d renderer have been designed for that and you hit mutex (except if you want to generate a video, but that's a different matter and not the purpose of gepetto-gui. I don't know for MeshCat).

I agree this isn't the intended use, this test was just a quick way of measuring how fast the communication was. Even if for a typical application you would want to communicate at 50Hz, it doesn't change the fact that I feel the communication for both gepetto-gui and MeshCat takes a lot of client time, I think due to the fact that communication and rendering are synchronized. Typcially, for my example with 5 katana arms, it takes on my machine between 20 and 60ms of client time to do robot.display(q) - this means that a rendering at 50Hz is unfeasible, and even if I want to update the screen at 10Hz only, it would still take close to 50% of the client's time...

So I think it's great that you are improving gepetto-gui performance, I'll give that a try.

But to go even further, I was thinking that a paradigm shift on the viewer/server design could enable much faster performance. Indeed, if we separate viewer rendering and communication, we should have much less impact on the client. To be precise, consider a viewer application that has one thread devoted to refreshing the display at 50Hz with whatever content is in the buffer, and a communication layer that fills the buffer based on client request. Thus, when you receive a package, you just store it in the buffer without taking any of the client's time - in fact you don't even need to have the server send an acknowledge to the client. So for the client application, robot.display takes exactly the time required to send a few hundreds of characters across a messaging interface (TCP or UDP) - which should be orders of magnitude below the 10ms threshold. Of course, doing that means that you accept the fact that the client has no control over the actual rendering, and, for instance, that a package send by the client might not be actually displayed on screen if it is overwritten by a new package - but it's something I can live with.

I don't know if there are mesh viewers that have been designed with this framework - possibly it's a bad idea that wouldn't work, but I don't really see why. That's why I was considering implementing it myself to see how it goes - but that would take some time. Just having a C++ library for talking to a browser like MeshCat could already boost the performance over the current meshcat-python interface. Thanks @traversaro for the link, I'll be sure to have a look and tell you if I start working on this.

Anyway, I consider all this potential future work. For now, I think it's already reasonable to finishintegrating meshcat-python in pinocchio, and test the new gepetto-gui - then if I'm still unsatisfied by the two solutions I'll see where to go next.

[jmirabel]

I think due to the fact that communication and rendering are synchronized.

in robot.display(q), the only synchronized call is refresh. It would not be hard to make it asynchronous. It is not always desirable but it can be parametrized.

Typcially, for my example with 5 katana arms, it takes on my machine between 20 and 60ms of client time to do robot.display(q)

Apply the client-side changes I mentioned and you already divide it by 2. I believe with little work you can even improve it. I can give you some hints.

exactly the time required to send a few hundreds of characters across a messaging interface (TCP or UDP) - which should be orders of magnitude below the 10ms threshold.

Are you sure of this statement ? TCP involves synchronous inter-process communication and, at some point, you give the hand to the kernel. I believe there is a lower limit per call that's not much below 1ms. If you want to be faster, you have to send more data per call. For UPD, the limit may be lower.

Note that with CORBA, locally and on UNIX OSes, you can bypass TCP/UDP and use a more direct communication using files. I never tried it and I don't know how fast it can be.

test the new gepetto-gui

don't expect much if no work is done on the client-side. The most visible change is that rendering a still scene is almost instantaneous.

[matthieuvigne]

Are you sure of this statement ? TCP involves synchronous inter-process communication and, at some point, you give the hand to the kernel. I believe there is a lower limit per call that's not much below 1ms. If you want to be faster, you have to send more data per call. For UPD, the limit may be lower.

I'm no communication expert so it's more of an intuition than anything. Clearly there is a major overhead for sending individual messages, so I was thinking more along the line of sending one large message, ideally using UDP, to some rendering server - and I think we agree that shouldn't take long. I know there are inter-communication protocols faster than TCP/UDP but I also never gave them a try. I'm not necessarily looking for infinite performance - typically being able to do a robot.display(q) call in ~1ms would already greatly satisfy me.

I'll have a look at client-side changes for gepetto then.

[olivier-stasse]

On the same computer good middleware are using either IPC or multithread communication. The use of TCP, UDP is normally left to inter-computer communication. Benchmarking CORBA on real-time OS in the context of a 5ms control loop, the cost of a CORBA/omniorb call was negligeable compare to the other computation.

Two middlewares used in the past transformed on meta-middleware: ROS1->ROS2 and a LAAS solution called Genom. I wish to avoid to handle several different middleware and meta middleware in the same sotfware framework.

There was plenty of similar discussion wrt to performances during the design of ROS 2. They are very deep discussion on real-time embedded software on the thread of ROS2 - rosdiscourse.

rviz for ros2 has been ported by Bosch and presented at ROSCon 2018. The target is windows, mac and linux... why not considering rviz 2 ?

[olivier-stasse]

On the other hand, I really like the idea of having a viewer using web-gl. wviz is another project related to this approach

[jmirabel]

On the other hand, I really like the idea of having a viewer using web-gl.

I like that too.

[olivier-stasse]

See #662 and #663, at least we should avoid to have too many dependency in pinocchio. For me a GUI inside a library initially targeted to do computation for control is not a necessary dependency. I would be fine for a package/another repository making the link between meshcat and pinocchio and everything necessary (including ZeroMQ)

[aelkhour]

Side question, how hard is to have python3 support for either solution?

• gepetto-gui seems to require a lot of install from source on Ubuntu 18.04, down to omniorb python3 bindings (I couldn't find python3-omniorb)
• I managed to run pip3 install meshcat but haven't tried it yet.

[jmirabel]

gepetto-gui seems to require a lot of install from source on Ubuntu 18.04, down to omniorb python3 bindings (I couldn't find python3-omniorb)

It isn't hard but requires compilation. OmniORB has Python3 support but it isn't packaged yet so you must compile OmniORB and OmniORBpy. In https://github.com/Gepetto/gepetto-viewer-corba/issues/85, it was reported to be straightforward.

[jcarpent]

Also solved by #662