search

ros-visualization / rviz

ROS 3D Robot Visualizer
BSD 3-Clause "New" or "Revised" License
822 stars 462 forks source link

Refactor for ROS 2 and M-Turtle #1125

Closed wjwwood closed 2 years ago

wjwwood commented 7 years ago

I'm going to be using this issue as a place to track progress on a large refactor of rviz which will hopefully land in the next version of ROS, ROS Melodic or ROS Melodic Morenia, as well as be used in ROS 2.

At a high level the goal is to break rviz up into a few parts, separating as much as possible the non-ROS specific parts from the ROS specific parts, so that as much code as possible can be reused in ROS 1 and ROS 2.

This will be sort of an experiment to see how realistic it is to approach ROS 2 porting this way. And so, if anything with the approach is not working out then I may adjust the plan to try and compensate for that.

Status

TODO: I will fill this out more as work materializes.

The Plan

TODO: I'll continue to refine and clean this up as I go, but this is just a dump of my personal notes at the moment.

I'll try to use rviz in reference to ROS 1 and rviz2 in reference to ROS 2.

Requirements

These requirements are things that would be best to have in ROS 2 to consider rviz2 functional:

Later Requirements

Goals

Ideas

Design Decisions

GUI Library

Use Qt 5.

It's the obvious choice for portable windowing in a desktop environment. It has a large user base, and good documentation. rviz from ROS 1 uses it so it will maximize reuse of code.

Alternative - Web

I looked again at robot web tools and recent advancements in emscripten (c++->llvm->asm->javascript), webgl, and other stuff. I still think starting with C++ and Qt is more viable. Qt is still less risky and likely to be much more performant for now.

Alternative - Electron

Electron is the node.js + webkit app sdk that atom.io is built on, as well as Visual Studio - Code. My experience with it is that it can be performance bound, especially if you don't know what you're doing. It also brings a whole new set of dependencies and tools/languages/libraries for use to learn. On the other hand, it would potentially open the door for other developers with those skill sets to get involved.

Ultimately it sounds risky, potentially slow, and brings in a lot of new dependencies.

3D Engine

Use Ogre, version 1.10.

Ogre is what rviz uses, so it will make porting the easiest. Ogre is already portable (mac, linux, windows, ios, android, etc...), well licensed (MIT), and familiar. It is also what Gazebo uses, so we can drawn their developers, experience, and/or code as well.

There are two main Ogre versions right now, 1.x and 2.x, see:

http://www.ogre3d.org/about/what-version-to-choose

2.1 seems cool, but it doesn't yet support macOS (will soon with new Metal API) and it doesn't support older machines and opengles 2 or 3. In general it is a lot less portable, but it would be cool to see how hard it would be to make rviz2 able to use it optionally to try it out. 1.10 supports the platforms and features we need. rviz also currently supports it so it will make porting easier.

Alternative - OpenGL

Using opengl or opengles or webgl or ... whatever has been proposed in the past, since what rviz does it relatively simple most of the time. It might also make things a little more performant in a few cases, but Ogre provides us a lot, especially when dealing with meshes and textures and such.

Alternative - Unity, Unreal, or some other popular game engine

These are all either free for some cases or semi-open, but most are not under an open license for all cases. The license is a big part of why Ogre rather than one of these. Also these engines tend to be a lot more batteries included on project layout and management which makes it more difficult to integrate with Qt and CMake and all the other typical things we use.

Alternative - VTK

This is Kitware's visualization toolkit library, which is used by lots of scientific applications to render data. It is also used for visualization within PCL. This is a serious contender, as it has efficient and well developed mechanisms to render data. I'm not familiar with how well it can handle meshes and things like that, which is a place where rviz's needs are different from PCL, for example.

This is a serious alternative, because it is purpose built for data visualization (as opposed to Ogre which is really a game engine), it's portable, and it has a decent license (BSD-like http://www.vtk.org/licensing/).

In the past it has been a dependency pain point, but it has become more stable in recent years.

I could still be swayed to use this rather than Ogre, but given that code sharing with rviz is a priority, and given that Ogre is more likely to be portable to other systems (iOS and Android for example), I have a slight preference for Ogre still.

Alternative - ITK

ITK is Kitware's data visualization tool based on VTK, which is similar to rviz in a lot of ways. I talked with the Kitware people a while back and talked about using ITK directly instead of rviz being a custom app. My conclusion was that it might work, but some work would need to be done to ITK to meet our use cases, so we'd be subject to the flexibility of ITK. For one thing, ITK doesn't have a notion of frames, so we'd still have to do all the data transforms before giving them to ITK. Another detractor for me was that ITK is a swiss army knife and it would be a lot of work (or not possible) to whittle down the interface so that it is as simple and lean to use as rviz is now.

This is quite a different direction to take, but a decent contender nonetheless.

Mesh Asset Library

Use Assimp.

This is what rviz uses now.

Seems to work.

Alternative - Custom parsing code

Gazebo does custom parsing of collada, for example.

Doesn't seem sustainable, assimp has worked so far for rviz.

Plan

As described above, the plan is to break rviz into parts some of which will be reused in ROS 2 and ROS 1, while others will be duplicated in ROS 1 and ROS 2 because they have communication specific stuff in it.

Components

These are the distinct components I can foresee right now (will likely change as it becomes clearer):

wjwwood commented 7 years ago

@Karsten1987 fyi

simonschmeisser commented 7 years ago

There is another option when it comes to the rendering engine: Qt3D (https://doc.qt.io/qt-5/qt3d-index.html) or (http://blog.qt.io/blog/2016/06/16/introducing-qt-3d/). It has a scene graph and everything included, no external dependencies and supports many platforms already. And it can be scripted using Qml/JavaScript which might allow for super simple plugins.

We have another use-case that you might want to consider: embedding in Qml/QtQuick. Currently our GUI consists of a QtQuick based main application and a visualization pane in traditional QWidget because this was the easiest way to embed rviz. Being able to pass in a OpenGl context directly would be awesome and allow us to design more flexible UIs.

gbiggs commented 7 years ago

If Ogre 2 will support all target platforms in the near future, then I think it is worth giving serious consideration to switching to it. The work will be backward-portable to ROS 1 rviz, since that only needs to support Linux, and if you restrict the "new" rviz to recent ROS releases you can limit it to machines that have the necessary OpenGL support. (I'm not sure what "it doesn't support older machines" means, though.)

Using VTK would also be good, if it gives us shortcuts to useful visualisations. I know that VTK is highly regarded, but I don't know enough about it to comment more than that.

Also what @simonschmeisser mentioned sounds very appealing. The Qt libraries are widely supported and having an rviz widget that can be quickly added to a GUI would make it much easier to create custom robot control GUIs.

simonschmeisser commented 7 years ago

@gbiggs just to clarify: You can already very easily add a rviz::Display to you QWidget based application. This is super simple and works just fine. What we would love to have as an alternative is to have the same for Qml/QtQuick (or for a generic OpenGl context, ie for people prefering Gtk or similar)

wjwwood commented 7 years ago

@simonschmeisser

There is another option when it comes to the rendering engine: Qt3D (https://doc.qt.io/qt-5/qt3d-index.html) or (http://blog.qt.io/blog/2016/06/16/introducing-qt-3d/).

That's a good one too, thanks for pointing it out. I was aware of it, but I think in the back of my mind I considered it as just Qt + OpenGL + some 2D stuff, meaning you had to do everything with the OpenGL API manually. But looking at it now, it seems much more featureful than I remember. Specifically I was thinking that you would need to write your own code to render meshes using triangles, but it seems Qt3D has support for various mesh formats, it even uses ASSIMP internally:

https://doc.qt.io/qt-5/qt3d-attribution-assimp.html

I'll have a closer look at it, but realistically I think I'll need to stick with what's already implemented in rviz (Qt + Ogre 1.10) because of the timeline I have to make something work for ROS 2. However, in order to keep the possibility of trying out Ogre 2.x and perhaps other things in the future (like Qt3D), I'm going to try and add some decoupling interfaces in rviz to keep that as simple as possible to try out in the future. I understand that Qt3D might be best used in a more integrated fashion, but at least this keeps the door open to trying it out.

Being able to pass in a OpenGl context directly would be awesome and allow us to design more flexible UIs.

Maybe you can help me understand this better, why do you need to pass in an OpenGL context? From what to where are you passing the context? Are you creating the OpenGL in QtQuick and passing it into rviz (eventually into Ogre)?

Sorry if this is an obvious question, I haven't spent much time with QML or QtQuick.

@gbiggs

If Ogre 2 will support all target platforms in the near future, then I think it is worth giving serious consideration to switching to it. The work will be backward-portable to ROS 1 rviz, since that only needs to support Linux, and if you restrict the "new" rviz to recent ROS releases you can limit it to machines that have the necessary OpenGL support. (I'm not sure what "it doesn't support older machines" means, though.)

I had two reservations with Ogre 2.x, first that it doesn't support macOS at the moment, which is a deal breaker for me at least 😄, and second that it does not have packages for any version available in either apt-get or Homebrew (they haven't tried to do any packaging yet). So it doesn't really meet the platform requirements for ROS 2 or ROS 1 at the moment, and it adds some additional burden on us to package and release it. That being said, I'm keeping an eye on it and hopefully in the future there will be resources (time/money) to look into different visualization options for rviz in the future (whether it be Ogre 2.x, VTK , Qt3D, or something else).

The bit about "it doesn't support older machines" means older video cards which don't support the newer OpenGL standard that they need. That's actually why macOS isn't currently supported, because the OpenGL version of macOS is pretty old (no matter what video card you use) and so they are adding support by using Apple's new "Metal API". Anyways, good enough to say many older cards would not be supported.

Using VTK would also be good, if it gives us shortcuts to useful visualisations. I know that VTK is highly regarded, but I don't know enough about it to comment more than that.

Yes, I think it might be a good option. But again, I don't think I have the time right now to really consider changing the rendering engine.

gbiggs commented 7 years ago

OK, it sounds like Ogre v2 is perhaps a little too new to rely on. But I agree with your approach to keeping it an option in the future.

bponsler commented 7 years ago

@wjwwood This looks like a great plan and ideas that should definitely improve the design of rviz

I have some general comments/observations/thoughts:

  1. A major pain point in the migration is the use of messages across most aspects of the code base. A lot of code needs to be modified to switch to the ROS 2 messages. You mentioned this above as it would make a generic (rviz and rviz2) plugin difficult. Being able to isolate plugins from specific message naming/include path changes would be useful for making rviz easier to port in the future. I don't have any great ideas at the moment on how best to accomplish this.

  2. I'm not sure what you're plan is for handling the robot model. I updated the robot_state_publisher to publish the model (see: PR 9) but would be curious to hear your thoughts on the best way to get the model into rviz.

  3. The mapviz package has a lot of functionality that could be very useful to incorporate into rviz itself. Notably the bird's eye view and ability to view satellite images. I'm not suggesting you do that work now, but it might inform the early design process to facilitate future support of those features. I imagine the current design won't specifically restrict these features from being implemented.

wjwwood commented 7 years ago

For anyone following along here, I moved our work to a fork of rviz on the ros2 organization, primarily so that I could have a separate issue tracker that I could integrate into our ROS 2 waffle board (https://waffle.io/ros2/ros2) without bringing in all the existing ROS 1 rviz tickets. I still intend for the end result to live in this repository long term.

https://github.com/ros2/rviz

Thanks @bponsler for your feedback, sorry for not getting back to you quickly.

Being able to isolate plugins from specific message naming/include path changes would be useful for making rviz easier to port in the future. I don't have any great ideas at the moment on how best to accomplish this.

I have some ideas, but I'll have to see if any of them work out or are feasible. It's unclear if I'll have any time between now and ROSCon to fully figure that out. I hope to do some work on it though.

I'm not sure what you're plan is for handling the robot model. I updated the robot_state_publisher to publish the model (see: PR 9) but would be curious to hear your thoughts on the best way to get the model into rviz.

For a long time I've thought that the robot description should not be in a parameter, but instead should be in a topic. I think that still make sense to me, but one of the main reasons behind it (that parameters are not recorded in a bag file) may not necessarily be true in ROS 2 once we implement bag files there.

If we do port the robot description display before ROSCon it will probably be using your patch to robot_state_publisher (thanks for that by the way).

The mapviz package has a lot of functionality that could be very useful to incorporate into rviz itself. Notably the bird's eye view and ability to view satellite images. [...] I imagine the current design won't specifically restrict these features from being implemented.

I agree that those features would be great, but I would come to the same conclusion that the current design doesn't prevent those features. But if you (or anyone else) knows of how the existing rviz plugin api prevents these features from being implemented I'd like to know about it while I'm iterating on those interfaces.

vincent-hui commented 6 years ago

If qt3d is used in rviz2, we can use Qt 3D Studio. It will be released in December possibly. http://blog.qt.io/blog/2017/02/20/introducing-qt-3d-studio/

wjwwood commented 6 years ago

@vincent-hui there's no plan to do that at the moment. Also, most everything in rviz is done programmatically, so I don't know how useful qt 3d studio would be, it seems to be made for animations and 3D user interfaces neither or which rviz uses by default.

beetleskin commented 6 years ago

Thanks for starting this!

However, you start rather technical. Shouldn't we rather gather some high-level requirements from the users ans find out, what is really needed, wanted and useful in terms of UI/UX/API? Thousands of developers have several years experience working with RViz and bending it to their needs, by working with custom DisplayTypes, custom Panels, Interactive Markers etc. Hence, there are a lot of learnings (and I suppose a lot of loathing too ;)) out there.

On the other hand, people are starting to take Visualization into their own hands as RViz is just not feasible for a lot of common tasks. For instance the trend to web-apps and easy HUDs with robotwebtools or the new RoboWare Viewer.

I believe that if we don't want to letting diverge the visualization into several tools. I.e. if we still want an RViz as our base visualization tool everyone is happily contributing to, one needs to face why people are diverging from it and have a look into the requirements of the future. A sole rewrite of the very outdated RViz and even more outdated Ogre as it is now is not going to fly.

That said, I don't think backwards-compatibility is going to fly either. I suggest to start from scratch with a modern and suitable engine as backbone. Otherwise, overcoming the current design-flaws, overcoming the pain of writing RViz code and meeting the challenges of todays requirements will be narrowed by the possibilities the current framework offers, and end-up in workarounds of workarounds of .. You get the point :)

wjwwood commented 6 years ago

Shouldn't we rather gather some high-level requirements from the users ans find out, what is really needed, wanted and useful in terms of UI/UX/API?

This work is not meant to fundamentally change the UX of rviz, only the underlying technical implementation to allow to be reused in ROS 1 and ROS 2 with as much code sharing as possible. See this from the OP:

At a high level the goal is to break rviz up into a few parts, separating as much as possible the non-ROS specific parts from the ROS specific parts, so that as much code as possible can be reused in ROS 1 and ROS 2.

For the other:

Thousands of developers have several years experience working with RViz and bending it to their needs, by working with custom DisplayTypes, custom Panels, Interactive Markers etc. Hence, there are a lot of learnings (and I suppose a lot of loathing too ;)) out there.

That, in my opinion, should be an on going community activity to improve it. This specific issue and work/project is more targeted than that.

On the other hand, people are starting to take Visualization into their own hands as RViz is just not feasible for a lot of common tasks.

My experience is that visualization with rviz is one of the main reasons people use ROS, so of all the robot development problems, I think there is actually evidence to conclude that people don't want to do it themselves, and instead use rviz. That's not to say everyone does that, but you'll never get 100% coverage on use cases.

For instance the trend to web-apps and easy HUDs with robotwebtools or the new RoboWare Viewer.

We've used web based viewers, and while they have made impressive progress, we find that getting data into and out of the web interface becomes a bottleneck. robot web tools, for instance, does some tricks to limit redundant tf information for performance reasons, and the transmission of point clouds is done differently to minimize data sent and processing on the javascript side.

I'm not familiar with the RoboWare Viewer.

I believe that if we don't want to letting diverge the visualization into several tools. I.e. if we still want an RViz as our base visualization tool everyone is happily contributing to, one needs to face why people are diverging from it and have a look into the requirements of the future.

Again the goal of this work is to take rviz, more of less as-is, and make it more reusable, not to tackle this issue you've described.

I don't feel there is a silver bullet to address what you're talking about, but I can't point to any one place where Ogre or rviz being C++ (rather than web based) holds you back in terms of visualization capabilities (unless your requirement is to use a web browser). There are some fuzzy, hand-wavy arguments that rendering could be more efficient if we didn't use Ogre, but it's hard to quantify that and back it up with facts. One of the things I'm trying to do is factor out the direct use of Ogre as much as possible, so that in the future we could move away from it if we need to, but that's a secondary goal (and a lot of work btw).

One by product of this work will be that rviz is hopefully easier to integrate into existing Qt applications and that it runs on Windows and macOS natively (one of the reasons people pick up robot web tools in place of rviz). That being said, I think rviz will continue to be part of the solution space for visualization, not a panacea.

A sole rewrite of the very outdated RViz and even more outdated Ogre as it is now is not going to fly.

This sounds like an opinion. rviz meets the needs of many users and is one of the most requested features missing in ROS 2. I think that an incremental improvement of rviz's current state will work just fine.

That said, I don't think backwards-compatibility is going to fly either. I suggest to start from scratch with a modern and suitable engine as backbone. Otherwise, overcoming the current design-flaws, overcoming the pain of writing RViz code and meeting the challenges of todays requirements will be narrowed by the possibilities the current framework offers, and end-up in workarounds of workarounds of .. You get the point :)

I would love to tackle that engineering problem, but I'm afraid that's simply not logistically possible.

I believe my plan is the best compromise, which is to maintain backwards compatibility with existing plugins (or with small changes required to update) and deprecate the functions that access Ogre directly. Even deprecating all the functions that use or provide access to Ogre directly is only a long term goal. Several of the plugins would likely just need to be rewritten for a different engine.

Also, I will concede that Ogre (especially 1.x) is old and not leveraging the latest API's, but I think it's also important to recognize that rviz doesn't actually require most of the new features in modern opengl API's. It has relatively simple lighting, and no need for physics or most particle effects or things like water simulation. The only benefit that could be gained to rviz is if there is some intrinsic performance boost to shaders or data pipelines. That's worth investigating, but I can hardly think it justifies a rewrite of rviz from scratch. Having just waded through the entire code base, I can tell you that's a non-trivial task.

machinekoder commented 5 years ago

@simonschmeisser I create an issue in the ROS2 fork for the QtQuick support enhancement: https://github.com/ros2/rviz/issues/268 It would be great if you could give some input.