[bug] ROS launch can not load launch file that includes gazebo (program path 'PATH' is missing or invalid)

[rickstaa]

System info

Version: 1.58.0-insider Commit: 3e0c442a8ee9fdc037d627d70fe291f72af04cfd Date: 2021-06-16T05:18:57.982Z Electron: 12.0.11 Chrome: 89.0.4389.128 Node.js: 14.16.0 V8: 8.9.255.25-electron.0 OS: Linux x64 5.8.0-55-generic ROS VERSION: ROS 1 Noetic VSCODE-ROS VERSION:

what is the bug

This week I was not able to debug a ROS launch script anymore if it contained a launch script that loads the gzserver. Every time I try to launch such a file, it gives me the following error messages:

Further in the /gazebo_gui console, it gives the following error message:

The program '/opt/ros/noetic/lib/gazebo_ros/gzserver' has exited with code 42 (0x0000002a).

Repro steps

1. Download the following example repository.
2. Start the ROS: Gazebo launch (works) launch file that is found in the .vscode/launch.json file of the example_repository.
3. See the errors that appear.

expected behaviour

Some weeks ago, the launch file would still launch.

Vscode developer tools log

Extra info

• Bug is also present in the normal vscode version.
• Setting export DISPLAY=:0.0 doesn't solve the issue.

[ooeygui]

This error looks like it is coming from gdb, not vscode - does gazebo run on your system? Are you actually debugging gazebo or a plugin? If not, I'd recommend creating a separate launch file for just the code you are debugging and launch everything else separately.

[rickstaa]

@ooeygui Thanks for your response. I'm currently using your suggestion as a workaround. I, however, was wondering why this issue came up in the last week. The gazebo simulator is working. A launch file that loads the gazebo is working when I run it in the terminal; however, when I try to debug such a gazebo launch file using the vscode-ros debugger I get the error.

I think your right that the gdb debugger might cause it. Do you maybe have some troubleshooting steps to test this?

[ooeygui]

@rickstaa The VSCode ROS extension processes the launch file to identify the launched files and command lines - then calls out to the vscode-cpptools. the vscode-cpp tools extension independently updates. It looks like there was a large update at the end of May; perhaps this coincides with when you observed this issue?

I have to set up a reproduction of this to see where the failure is. I'm not sure this can be diagnosed outside.

[rickstaa]

@ooeygui Ah, I see; I haven't used this feature since the beginning of May so that the upstream update could have caused the error I'm experiencing. I looked through the vscode-cpp tools issues, but I couldn't find a related issue. I added a gif of the behaviour I experienced next to the example (reproduction) GitHub. The strange thing is that it complains that the gzserver is not executable while it has the following permissions-rwxr-xr-x`.

For now, I will manually load the gazebo simulator or attach it to a running session. Let me know if I can do something to debug this issue.

[ooeygui]

@rickstaa Thank you for the gif - that is really helpful! I did a VSCode ROS release in May which included several fixes - took an action item to set up a repro and debug this.

[rickstaa]

@ooeygui Thanks a lot for looking into it. I created the following repository to reproduce the issue. Let me know if you need any other info or help.

[rickstaa]

For other people that also experience this issue. You can still debug your scripts using the vscode-ros Attach feature. You can use the code below to wait for the debugger.

Python

# DEBUG: Wait code
import time

dummy = 0
while dummy == 0:
    print("Waiting for debugger")
    time.sleep(1)

C++

  int i = 0;

  while (i == 0) {
    usleep(100000);  // sleep for 0.1 seconds
  }

[ooeygui]

gzserver and gzclient are shell scripts. Our code detects python, but not shell scripts, so it treats it as a binary program.

I'm going to add some code to detect this case and just launch it.

[rickstaa]

@ooeygui Ah I see thanks again a lot for looking into this problem!

[rickstaa]

@ooeygui Did you manage to fix this issue? I noticed that the behaviour is still present in v0.6.9.

[ooeygui]

@rickstaa I'm sorry, no, I haven't worked on this yet. It's one of the top items I want to fix in 0.7.0.

[rickstaa]

@ooeygui No problem. I was just curious. Thanks a lot for all the work you do for this extension.

[ooeygui]

@rickstaa I appreciate that comment :-) Thank you!

[KoganOleg]

@ooeygui It looks that I got the same problem when calling the bash script from launch file. Do you have any estimation when the fix will be ready and released?

[ooeygui]

I have implemented an experimental feature in 0.7.0 which allows you to filter out gazebo. However, it does a "fire and forget" for ROS nodes... which isn't ideal as you can't shutdown the entire graph easily. It's going to take some rework to detect running nodes and not relaunch them.

[rickstaa]

@ooeygui Great. I will test it out when I am working on a ROS project!

[rickstaa]

@ooeygui I verified your solution and it works! Thanks a lot for implementing this. About the "fire and forget", I noticed that this also occurred before v0.7.0 when you try to attach to a ROS node that is launched by a subprocess Popen call. What I currently do is keep a htop window open to close any dangling processes after the debugger has stopped.

I think for now we can close this issue and create a new issue for the "fire and forget" problem when it is stumbled upon. For me, closing processes through htop is not too bad. I am very happy with the extension being able to debug roslaunch files that contain gazebo again.

[rickstaa]

@KoganOleg can you verify whether the solution also works for you?

[KoganOleg]

@rickstaa Looks that it doesn't fix my case, still issuing the same problem. VS code version 1.63.2. Would you like me to provide more details?

[ooeygui]

@KoganOleg Can you verify that you are using the 0.7.0 version of the VSCode ROS extension? If so, can you post the contents of your launch.json file?

[KoganOleg]

@ooeygui It was the first thing to check the version of ROS (0.7.0). Attaching my launch file, error message and ROS version screen shots. Another detail - I'm working inside the docker container. conti_ars540_default_with_script_launch.txt

[ooeygui]

@KoganOleg It looks like your ros node is just a shell script, which isn't a supported target for the vscode ros extension. I would recommend using python instead of a shell script if you are able to.

If you are not able to switch to a python script, you can filter this out in 0.7.0, with something like this:

{
  "name": "ROS: Launch my file",
  "request": "launch",
  "target": "full path to launch file",
  "launch": ["ars540_route_config"],
  "type": "ros"
}

[KoganOleg]

@ooeygui Thanks a lot for your clarification. Indeed, I'll try to replace the bash with python script. Also now understand the second part of your explanation, tested this option and it works for me as well - script is executed without being attached.

[ooeygui]

I'm working on improving support for scripts in the launch path.

[ooeygui]

After looking through this, I decided to add a new feature - https://github.com/ms-iot/vscode-ros/issues/647. This will expose UI for graph control. The ROS Launch debugger can introspect the graph control and launch and attach based on that.

the ROS launch code doesn't handle sh or bat files, so I'm fixing that as a code filter. the code will just execute it - but not track that execution.

[KoganOleg]

@ooeygui Thank you! Should be really nice feature!

[rickstaa]

@ooeygui Thanks amazing implementation. Very helpfull! I think we can close this issue for now?

[ooeygui]

@KoganOleg Thanks!

@rickstaa For this specific bug, I'm working on a fix to improve detecting of bat/sh/bash/cmd files so that it doesn't just fail out right. I hope to have that committed in a few days. Once that bug fix is committed, I'll close this issue. I have a couple of blocking things like this going into the 0.8.0 release.

[rickstaa]

Cool, I will leave the closing of the issue to you! Thanks again for your work on the 0.8.0 release.

[xzzzw]

@KoganOleg Hello, I am a student from Tsinghua University in China. Recently, I also purchased an ARS540 radar from Continental Germany, but the manufacturer did not provide a driver. Now it looks like you have the ros driver for ARS540, do you suggest sharing it with me to learn? I would appreciate your reply，thanks a lot!

[PENGYESYSU]

@rickstaa I encountered exactly the same promblem. I am debugging ros launch within docker using vscode ros extension. The ros launch file include another launch file for simulation using gazebo. Can you explain more about your solution above using "sleep" trick? I don't understand where to insert the code below.

  int i = 0;

  while (i == 0) {
    usleep(100000);  // sleep for 0.1 seconds
  }

[ooeygui]

@PENGYESYSU I'll reopen this issue. My original fix was to just not attach the debuggers to gazebo.

[PENGYESYSU]

@PENGYESYSU I'll reopen this issue. My original fix was to just not attach the debuggers to gazebo.

Can you please explain the "fire and forget" trick step by step? I have no experience with some "fire and forget" and I really don't understand how to do that making ros launch debug to work.

[ooeygui]

@PENGYESYSU The way the launch system works in VSCode is that that the launch files are processed to output what would have been launched by the system. In ROS 1, roslaunch is used to 'dump' the executables and command lines, then the VSCode extension executes each under the appropriate debugger. for ROS 2, an alternative launch system was implemented which most emulates ros2 launch, which is used to launch the individual nodes.

Because those nodes are under a debugger, VSCode 'tracks' those nodes so they don't get lost when you shut down a graph. This is important later.

Attaching a debugger to gazebo and rviz is mostly undesirable - so the launch.json file has 'filters' for them. When you launch the debugger, these are detected and simply executed - without launching under a debugger. This means VSCode loses track of those filtered out - they are fired as in launched, and forgotten, meaning you need to manually close them.

I've had a desired to revamp the launch system so that we can monitor all running nodes, and allow selectively debugging them, but it is a big work item.

[PENGYESYSU]

Appreciate for the vivid explaination. As you reply to KoganOleg a node could be ignored by debugger by using "launch": ["ars540_route_config"] in the launch.json. How to do that to ignore gazebo and rviz? I added some code like "launch": ["gzserver"] but it seems not working.

[ooeygui]

@PENGYESYSU I reactivated this bug, to verify that this codepath is working. The 'filter' is on the executable name, not the node.

[PENGYESYSU]

I add "launch": ["gzserver", "gzclient", "rviz", "some_other_simulation_related_executable_name"], it works fine from ros:launch debug.