Semantic, waypoint, and costmap filter GUI editing tool(s)

[SteveMacenski]

As we're merging in speed restricted zone capabilities into Nav2 (in addition to the keepout zones recently merged), it's a good time to think about tools for creating and editing those maps. To date, we have folks go into gimp or similar to edit them by hand, which is error prone (from experience) if you do something silly like rotate or move the origin by complete accident.

It might be a good idea to build some basic tools for working with these maps. Since they're offline tools not run on the robots, we can be a little more relaxed about them (qt python!) and they don't even need to involve ROS (structures are much more simple).

A very simple prototype (that would work for our needs) would involve:

• takes in a file path to a map
• displays that map and allows you to rotate and move the origin by simple "drag and drop" features
• has some drop down for the editing mode: keepout, speed restricted (%), speed restricted (m/s)
• Allows the user to "paint" over the base map with this editing mode
• save to file (probably also removes the input map layer just so that occupancy values aren't added, but matches size and origin)

[SteveMacenski]

Also perhaps a good time to consider other GUI tools that might be helpful. https://github.com/ros-planning/navigation2/issues/1721 is for a general setup assistant like Moveit. What other things could a simple GUI tool be very helpful for?

[SteveMacenski]

GUI tool for marking waypoints (GPS or cartesian) in a scene and storing to file for waypoint following. Also possibly a waypoint follower GUI where you give waypoints and it has start/stop/etc buttons to waypoint follow analog to the waypoint follower tutorial demo server - marked wps in GUI or from file.

[TomasHricina]

I am interested in working on the first project, that was mentioned. Steve, will you create repository where I can upgrade the progress, or should I create it ?

Thank you, Tomas

[SteveMacenski]

@TomasHricina for now, I think the best place to put this would be to fork this project and add a new directory under /tools/ for the GUI project you'd like to start with (costmap filters, waypoint, sementic, etc). Eventually we could cover all of them, but lets just start with 1 for now to get our bearings and we can always bite off more later. It's always better to get one thing done than 4 things 25% done.

My suggestion would be to start playing with pyQT and making a basic widget and building up from there. Start with a button, then a field to open a dialog to select a file, then rendering a map. Once you have those basic building blocks, then assembling them is perfect: a dialog to select a file, button to hit "open" and render that map. I think that's really the most core capability to start off with that all of the features will need.

I'd make the code behind the QT buttons for opening a file / loading a map into a member a separate class from the QT GUI so that it can be portable across other widgets we want to make described above. Also that way if we want to resign the GUI to be more slick, we're not fumbling around with functional code!

Does that seem like a good starting point to you?

After I think it would look something like (for costmap filters starting):

• Create a "paint brush" library to allow a user to "draw" onto the map surface some cost to mark for keepout/speed zones
• Input fields for that to support: brush diameter, value to write, and probably whatever costmap filter metadata is interesting
• Save to file and generate the pgm / yaml files

And that would be 1 full application where we'd have an end-to-end setup! I think that's also the hardest one with the paint brushes, so marking waypoints and rotating maps would be a breeze!

[TomasHricina]

First of all, thank you for your time and suggestions.

I think it is good starting point and I will stick to the roadmap, that you have outlined,. When I have something meaningful, I will make a fork and drop link here.

Thank you & Have a nice day, Tomas

[SteveMacenski]

Awesome, thanks for getting started on this, it will be a great quality of life improvement for everyone so they don't have to use gimp/inkscape for annotating maps (which is both complex and error prone).

[SteveMacenski]

any update? :smile:

[TomasHricina]

any update? smile

https://github.com/TomasHricina/navigation2/tree/main/tools/MapViz

Hello Steve, sorry for the wait. Current features: Load Yaml button (with checks and error messages), Resize slider, Rotate slider Next: I would like to implement Zoom and after that, I have to calculate the coordinates transforms (after resizing, rotating and zooming) and then I would like to start with the Paint (which will have different brushes, colors etc - PyQT already has some of the tools) I have spent 0 time on making it look good, that will come at last. Everything is modular, in classes, just like you suggested, so it can be replaced and reworked easily. Current name is MapViz :-D, thats where my naming creativity ends. There is gif preview, to save you time. If you do decide to run it yourself, you might want to install cv2-headless, because there is some issue with running full opencv with PyQT5.

Edit: Please let me know your suggestions, feedback

Have a nice day, Tomas

[SteveMacenski]

Hello Steve, sorry for the wait.

No waiting, no worries, takes time to do things! Just wanted to touch base. This is really great progress!

I'm not sure I think the resize slider is needed, but that's a fun one :laughing: But who knows, why not keep it. Now if you could do a drag on the map to move the origin (and/or update the yaml with it) that would be awesome. Scale / rotate / translate sounds like the basic 2D operation set!

I would like to implement Zoom

Isn't zoom, resize?

I think the next step after those different 2D operations you mentioned above would be to figure out how to then save that information to file before we get into the painting. If you can load, rotate, and save, you just made the MVP required for the first full feature!

I have spent 0 time on making it look good, that will come at last.

Agreed, though a couple of things I'd like you to add (even if poorly looking) to make sure we don't forget

• On the file path, make sure to have something that shows that file path of the currently displayed file
• "Save" lets do "save as" so that they have to specify a file path / filename to not accidentally overwrite the original by default. Having it start in the same path as the original file is logical though.
• Make sure to abstract out the file path stuff, we'll be using that alot for yaml files and other stuff.
• Maybe for the rotate / scale sliders having a text field to specify a specific rotation angle as well / display the current value for rotation angle / scale? May also be good for the origin moving too.

Current name is MapViz

That's a pretty good name.

[TomasHricina]

This is really great progress!

Thanks for the motivating words :-)

I'm not sure I think the resize slider is needed

You are right, the resize slider will go away and instead it will be mouse edge dragging resize.

Isn't zoom, resize?

The original thought was that resizing is changing window size and zooming is for getting closer view of the map. Right now, you cannot resize your window beyond your screen size, therefore you cannot zoom with it. I can see, why the slider made it confusing :smile: Either way, I will remove the "resize slider" and add Mouse Wheel zoom and Mouse Drag translation. I will also add CAD style rotation with SHIFT + Mouse drag with addition to the rotation slider

Agreed, though a couple of things I'd like you to add (even if poorly looking) to make sure we don't forget

Those sounds like reasonable and doable suggestions, I will get on it :smile_cat:

Have a nice day, Tomas

[SteveMacenski]

Ooooh, also crop, I didn't think about crop, that could be useful for maps that are sometimes oversized

The original thought was that resizing is changing window size and zooming is for getting closer view of the map.

Got it, zoom to see detail on the map vs actually changing anything about the map. I'd do that with the mouse wheel thing rather than sliders though! Zoom with that and click + drag for moving the map is intuitive. Though we need to separate zoom/move map for visualization and rescale / move origin for messing with the underlying map.

[TomasHricina]

Hello, I am incrementally working on MapViz and I decided to share some progress. Version: 1.0.2

Old features: General structure Loading YAML Verifying YAML Rotating map (1 - clockwise, 2 - counterclockwise)

New features: Zoom with mouse wheel, where cursor is located Zoom with box selection (Shift+Mouse drag) Crop with box selection (Alt+Mouse drag) Translation (pan) with Ctrl+Mouse Translation with W,A,S,D, diagonal Q,E,Z,X

All operations can be made in any order.. you can zoom, rotate, crop in any sequence, it will behave like you expect it.

Next features: -Undo, Redo buttons - this is currently in progress. We cannot afford to store image at every little change. So instead I will create stack of "routines" so we can go to any point in time without storing image at each change (must-have for painting), it will work like we are used, when you Undo and Make routine, you will override older routines, so there is no branching. -Entry box for angle, so you dont have to rotate by one. -Entry box for rotation speed, panning speed -Buttons for all the operations (zoom selection, crop) -Save As button -Saving map & YAML -Paint library - when I use PyQT paint one, I cannot seem to find, where exactly the paint is - the pen or brush does weird rotations when moved, so I "cannot" find the coordinates algebraically nor seems to be there API for that. This is open question, if I cant do it in PyQT, I will implement my own pen/brush and box drawing. -more stuff in the paint library

I included preview-gif to save you time. https://github.com/TomasHricina/navigation2/tree/main/tools/MapViz Have a nice day. Tomas

[SteveMacenski]

Ooooh very cool! This is really taking shape :smile:

-Entry box for rotation speed, panning speed

Maybe you could explain this more, wouldn't it be proportional to your mouse movements?

-Undo, Redo buttons

Oooooh, that's a nice feature, didn't think about that one. I was just thinking for waypoints having an "add" and "delete" button and for the paint have an "eraser" mode to undo mistakes. But having a literal "undo" button would be nice (though notably harder)

This is really neat. You're chugging along to a great MVP! Very excited for the paint and save features!

[TomasHricina]

Maybe you could explain this more, wouldn't it be proportional to your mouse movements?

Sure, currently you can move with keyboard or with mouse dragging. The speed of panning is, as you said, proportional to the mouse movement. However, with keyboard, there is no one-speed that fits all, so I included keyboard stroke for changing that speed. But, while writing this comment, I had better idea, the panning speed will change proportionally to the zoom. Pretty sure, this is small detail, because the mouse drag works great.

I am on vacation right now, so next update might few days later, but I am already feeling guilty for not working on it :smile:

Also, I have bad news. There is ROS package called MapViz already. It is interesting package and it is actively maintained, so I think we should be using different name. https://github.com/swri-robotics/mapviz

From top of my head suggestions, NavViz or Nav2Viz :rofl:

[SteveMacenski]

We'd end up calling it nav2_mapviz anyhow (part of the metastack), so it would be ok :smile:

[SteveMacenski]

Great progress! Roadmap (for my sanity and memory) https://github.com/TomasHricina/navigation2/blob/main/tools/MapViz/preview_1_0_3.gif

• Paint tool to enable painting on top of a map for keepout/speed zones
• Paint tool for enabling to paint on the map itself (so we can remove artifacts or directly modify the underlying map)
• Waypoint marking and saving in yaml tool for waypoint following / route definition / etc, temporarily until we have the semantics files running

[TomasHricina]

Thanks, I have uploaded all the code. https://github.com/TomasHricina/navigation2/tree/main/tools/MapViz

New features: -Undo, Redo (implemented without need for saving every pixmap - great for painting) -History box - easy to go back/forward in time to particular edit -Angle box - set any angle or increment/decrement by one, with buttons or keyboard -YAML input validation and notification - entry box goes red, if particular field is not valid (and cannot be saved) -Default buttons for YAML input -Save as - saves both YAML and image (Todo: make sure user knows, that image will be overwritten)

Next features: As Steve said, it will be mainly Paint functionality

Paint tool to enable painting on top of a map for keepout/speed zones Paint tool for enabling to paint on the map itself (so we can remove artifacts or directly modify the underlying map) Waypoint marking and saving in yaml tool for waypoint following / route definition / etc, temporarily until we have the semantics files running

• Command line capability - "mapviz $path_to_yaml" will launch Nav2-MapViz or give error message

[SteveMacenski]

Super excited!!

Does all of this resize? Ex. can I full screen this window or drag dimensions (within some boundaries)

[TomasHricina]

Does all of this resize? Ex. can I full screen this window or drag dimensions (within some boundaries)

I think it scales nicely. It starts about one quarter of the screen and you can drag it or maximize it. In the GIF it shows the minimal size.

https://github.com/TomasHricina/navigation2/blob/main/tools/MapViz/scaling.gif

Do you think it should be able to go smaller ? I can definitely look into it.

I am planning getting rid of the Top menu buttons and put smaller versions into the Left menu or use Drop down menu on the top.

I think the Top menu should have the editing functions - like painting and cropping

Any suggestions appreciated! :wink:

Edit: The code is IMO modular enough, so we can easily change where stuff is located anytime.

[SteveMacenski]

Perfect, no complaints on that! What happens if the map is much larger (like a warehouse size)? Does it zoom out more by default to keep that initial size?

[TomasHricina]

If you load larger map than previous, it will correctly resize the canvas, but it will be zoomed at the location, where the smaller map used to be. This is nice TODO, I will make sure, it zooms into full view, every time new map is loaded.

How big can these maps usually get ? I have only tested with ~4000x4000.

[SteveMacenski]

if 4000x4000 worked, I think we're OK.

[felrock]

Hello! I was wondering what's the status on this? Seems like an awesome feature that would be super useful.

[SteveMacenski]

Its been under work, but students and classes can get in the way :wink:

@TomasHricina can you give us an update? You were like 90% there last we spoke!

[TomasHricina]

Hello @SteveMacenski @felrock - Thank you for showing interest. The project is indeed mostly ready, it just needs final touch plus convert it to ROS package. I have been promising to finish up, but I was overwhelmed in new job, I apologize - I had to triage my time / fuel in tank. Things settled, so I believe October is the month.

[SteveMacenski]

Apologies, jobs, not classes. But similar levels of real-life blockages :wink:

Awesome!

[SteveMacenski]

@TomasHricina any word?