j-rivero commented 3 years ago

The Conda project has available a package for Gazebo11 inside the conda-forge organization. The Goal of the ticket is to evaluate if this installation is stable enough to go into our documentation.

Help wanted: We are interested in knowing if our users on Windows can install and use it without problems. If you want to help, please follow Conda instructions to install anaconda, miniconda or other variant of it, try to install Gazebo from conda-forge and try to run a basic simulation on it. Please comment in the issue with details about Windows version, conda installer, problems/not-problems found, graphic card drivers installed and other information you find useful. You are also welcome to comment and help with other people's problem.

Windows specific: check https://github.com/osrf/gazebo/issues/2901 for the list of configurations, problems or issues on Windows.

j-rivero commented 3 years ago

My testing of the superbuild:

System: Win10 Home 64bits
Graphics: Geforce MX150 + Intel (configured in nvidia control panel to run gzserver/gzclient on nvidia), driver version 398.36, DirectX runtime 12.
Conda version: Miniconda installation, gazebo 11.3.0 package used from conda-forge
Other: Disabled antivirus.

Runtime testing:

conda install gazebo
...
C:\ProgramData\Miniconda3\Library\share\gazebo\setup.bat
gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.

seems like something is blocking the network connection or the network connection is miss-configured somehow. The GUI did not appear. Did not have time to debug this.

traversaro commented 3 years ago

Which shell are you using? Powershell and Git Bash are currently know to not be working (see https://github.com/conda-forge/gazebo-feedstock/issues/42). Note that it should not be necessary to call C:\ProgramData\Miniconda3\Library\share\gazebo\setup.bat manually, but it should be sufficient just to re-open a new command prompt and just activate conda again there.

Note that it seems that you installed Gazebo in the base environment, it is generally recommended to avoid to install heavy packages in the base environment, but rather to create a new environment and install the dependencies that you want to use there.

cc @wolfv @Tobias-Fischer

j-rivero commented 3 years ago

Which shell are you using?

plain cmd.exe.

Note that it seems that you installed Gazebo in the base environment, it is generally recommended to avoid to install heavy packages in the base environment, but rather to create a new environment and install the dependencies that you want to use there.

Thanks for the tip. Seems to me like it should not change the behavior of the problem, is this correct?

wolfv commented 3 years ago

@j-rivero it would be interesting to know which version of gazebo got installed, too.

wolfv commented 3 years ago

PS: this is exciting! I am happy to help in any way I can!

j-rivero commented 3 years ago

Thanks @wolfv ! From my previous comment:

* **Conda version**: Miniconda installation, gazebo 11.3.0 package used from conda-forge

danzimmerman commented 3 years ago

Windows Insider Build 20279.fe_release.201209-1332
miniconda3 
conda 4.8.4
NVIDIA GTX 1050 Ti / Driver version 460.20
DirectX 12

Installed with

conda install gazebo -c conda-forge

into a virtual environment (Python 3.8 if that matters)

Set gzserver.exe and gzclient.exe to use the NVIDIA card in NVIDIA control panel, with default settings for the rest.

Set the windows firewall:

Ran

miniconda3\envs\<envname>\Library\share\gazebo\setup.bat

Expected Behavior

The command

gazebo --verbose

should show some messages and then pop up some GUI windows (I guess? I am new to Gazebo.)

Actual Behavior

Running from an Anaconda prompt for the virtual environment, after running setup.bat:

(<envname>) C:\Code>gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140

<about three seconds delay>

Silently dies, no messages, no windows...

Same behavior with Antivirus disabled.

Same behavior from regular cmd prompt (after activating appropriate conda environment)

Please let me know if this is useful or too n00b-y to be helpful. I don't have a clear entry point into this, just found it trying to find Gazebo binary builds for Windows and thought I'd give it a try.

traversaro commented 3 years ago

Please let me know if this is useful or too n00b-y to be helpful. I don't have a clear entry point into this, just found it trying to find Gazebo binary builds for Windows and thought I'd give it a try.

Thanks @danzimmerman for the useful feedback! I personally think that any feedback is useful, until gazebo runs easily on Windows. Two additional questions:

Can you copy&paste the output of the conda list and conda info commands? Furthermore, to understand if there is anything that is interfering with your environment, it would be useful to know the output of set command in the command prompt terminal with the gazebo environment enabled.
I read that both you and @j-rivero manually ran the setup.bat script, however this should not be necessary as the setup.bat script is already run by the activation process of the environment.
What happens if you try to run gzserver.exe --verbose in one terminal and gzclient --verbose in another?
What happens if you don't specify configure gzserver.exe and gzclient.exe

traversaro commented 3 years ago

Note that it seems that you installed Gazebo in the base environment, it is generally recommended to avoid to install heavy packages in the base environment, but rather to create a new environment and install the dependencies that you want to use there.

Thanks for the tip. Seems to me like it should not change the behavior of the problem, is this correct?

Sorry, I did not notice this message. In theory not, but if there is some bug in environment variables handling somewhere that affects the behavior and cause the crash.

danzimmerman commented 3 years ago

Output of conda info:

     active environment : npe
    active env location : C:\Users\<username>\miniconda3\envs\npe
            shell level : 1
       user config file : C:\Users\<username>\.condarc
 populated config files : 
          conda version : 4.8.4
    conda-build version : not installed
         python version : 3.8.3.final.0
       virtual packages : __cuda=11.2
       base environment : C:\Users\<username>\miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\<username>\miniconda3\pkgs
                          C:\Users\<username>\.conda\pkgs
                          C:\Users\<username>\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\<username>\miniconda3\envs
                          C:\Users\<username>\.conda\envs
                          C:\Users\<username>\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.8.4 requests/2.24.0 CPython/3.8.3 Windows/10 Windows/10.0.20279
          administrator : False
             netrc file : None
           offline mode : False

Output of conda list (edited this to attach & remove the wall of text)

original-comment-npe-conda-list-output.txt

I read that both you and @j-rivero manually ran the setup.bat script, however this should not be necessary as the setup.bat script is already run by the activation process of the environment.

Understood. I've been setting up ROS all day so it's kind of a knee-jerk reaction.

What happens if you try to run gzserver.exe --verbose in one terminal and gzclient --verbose in another?

Server:

(npe) C:\Code>gzserver --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140

<about three seconds delay>

Dies silently...

Client:

(npe) C:\Code>gzclient --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.

<re-starts gzserver in other window>

[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140

<dies when gzserver dies>

What happens if you don't specify configure gzserver.exe and gzclient.exe

You mean in the NVIDIA control panel? Same. Firewall rules were actually added when I first ran gazebo, okayed them in GUI window.

Are there dependencies to gazebo conda-forge installation that conda wouldn't handle? I might be missing something?

Also happy to create a cleaner environment to try this. I also don't know exactly how I'm going to interact with Gazebo. I've set up ROS melodic on Windows according to http://wiki.ros.org/Installation/Windows, using customized repos from https://ms-iot.github.io/ROSOnWindows/ where necessary.

Aaaand... apparently, those Windows instructions mean I already have a working installation of Gazebo 9.13.1 on my Windows machine.

That gazebo installation seems to be built as a DLL and only can be accessed using roslaunch, so I can't test any executables, but that Gazebo DLL build does work. This was not really clear from my reading of the ms-iot stuff, which is how I ended up here.

I expect that interacting with Gazebo via ROS is probably the way to go on Windows, so maybe I personally don't actually need to get this conda installation working. However, I'm happy to answer more questions.

danzimmerman commented 3 years ago

Sorry forgot the set output. Here are the environment vars on my work laptop in the npe environment, edited this to attach instead:

original-comment-npe-env-vars.txt

By the way, before I did this I did remove a %PYTHONPATH% variable and a couple of %PATH% entries related to a USD (pixar universal scene description) install that I was no longer using. This changed the behavior a little (which .... I don't know, the folder those were pointing to was deleted already )

Now I get the following with gazebo --verbose

The first time I run the command in a fresh Anaconda prompt window:

(npe) C:\Code>gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140
[Err] [..\gazebo\transport\Connection.cc:546] Connection[0] Closed during Read
[Err] [..\gazebo\transport\ConnectionManager.cc:131] Unable to read from master
[Err] [..\gazebo\gazebo_shared.cc:77] Unable to initialize transport.
[Err] [..\gazebo\gazebo_client.cc:56] Unable to setup Gazebo

The second and subsequent times I run the command:

(npe) C:\Code>gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] [Msg] Publicized address: Waiting for master.
192.168.1.140
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140

From a cmd window navigating to the bin directory but not activating the environment

C:\Users\<username>\miniconda3\envs\npe\Library\bin>gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Err] [..\gazebo\common\Console.cc:160] Missing HOME environment variable.No log file will be generated.[Err] [..\gazebo\common\Console.cc:160] Missing HOME environment variable.No log file will be generated.[Msg] Waiting for master.
[Msg] Waiting for master.
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.140
[Err] [..\gazebo\rendering\RenderEngine.cc:687] EXCEPTION: unable to find OpenGL rendering system. OGRE is probably installed incorrectly. Double check the OGRE cmake output, and make sure OpenGL is enabled.

[Err] [..\gazebo\rendering\RenderingIface.cc:41] Failed to load the Rendering engine subsystem
unable to find OpenGL rendering system. OGRE is probably installed incorrectly. Double check the OGRE cmake output, and make sure OpenGL is enabled.
[Err] [..\gazebo\gazebo.cc:81] Unable to load sensors
[Err] [..\gazebo\Server.cc:306] Unable to load gazebo
[Err] [..\gazebo\rendering\RenderEngine.cc:687] EXCEPTION: unable to find OpenGL rendering system. OGRE is probably installed incorrectly. Double check the OGRE cmake output, and make sure OpenGL is enabled.

[Err] [..\gazebo\rendering\RenderingIface.cc:41] Failed to load the Rendering engine subsystem
unable to find OpenGL rendering system. OGRE is probably installed incorrectly. Double check the OGRE cmake output, and make sure OpenGL is enabled.
[Wrn] [..\gazebo\rendering\RenderEngine.cc:292] Cannot initialize render engine since render path type is NONE. Ignore this warning ifrendering has been turned off on purpose.

gzclient pops up a splash so it runs but hangs at "Preparing your world..."

From a raw cmd, after activating the virtual env:

(npe) C:\Users\<username>\miniconda3\condabin>gazebo --verbose
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.
Gazebo multi-robot simulator, version 11.3.0
Copyright (C) 2012 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org

[Msg] Waiting for master.

<hangs indefinitely here>

I'll try this on my personal laptop which I haven't tinkered with nearly as much with system-level tools/env variables. This feels like a mess.

danzimmerman commented 3 years ago

Tried this on my personal laptop (Windows 10 Home Ver 2004 Build 19041.685, Dell XPS13 9300) with two different environments:

the npe environment, uses Python 3.8.3. I've manually installed a few packages on the work laptop, so this environment is no longer identical to the npe env on my work laptop, but it should be very similar.
a fresh gaz383 environment initialized minimally with Python 3.8.3 to match the npe environment.
a clone of npe, npe-gaz, into which I installed some packages that weren't in npe after Gazebo install but that were in gaz383.

I've attached the output of set and conda list for both environments.

Fresh Environment (Runs) vs. Existing Environment (Fails) - Anaconda Prompt

gazebo --verbose starts and appears to work from Anaconda Prompt in the fresh gaz383 environment. The failures in the npe environment are the same as described above on my work laptop. Compared with the working gaz383 environment, it looks like my npe environment is missing:

ilmbase
openexr
pugixml
zziplib

Attempting manual install of those four dependencies into a clone of the npe environment (npe-gaz) does not fix the issue.

I've distilled the differences between gaz383 and my npe-gaz clone with the four added dependencies here:

gaz383-npe-gaz_package_diffs.txt

and for the original difference between gaz383 and npe here:

npe-gaz383-package-differences.txt

Outstanding Possible Issues

Adding models to the empty world results in

[Err] [..\gazebo\common\ModelDatabase.cc:607] Missing model.config for model ""
[Err] [..\gazebo\common\ModelDatabase.cc:671] Invalid model manifest file[""]

I tried the UR10 arm specifically, and a few others. Maybe this needs another dependency?

Files

Environment vars

I'm not really seeing differences in the environment vars:

gaz383-envs.txt (WORKS) npe-envs.txt (FAILS, MISSING PKGS) npe-gaz-envs.txt (ADDED PKGS, FAILS)

`conda list` outputs

gaz383-condalist.txt (WORKS) npe-condalist.txt (FAILS, MISSING PKGS) npe-gaz-condalist.txt (ADDED PKGS, FAILS)

Summary

Installation with conda and use from Anaconda Prompt in a fresh, empty environment allows Gazebo to run.
Installation into an existing environment does not run, some packages are missing.
Adding those missing packages manually does not fix the broken environment. Presumably some dependencies are still wrong versions, etc.
I can't add models with the Insert tab, but maybe this is user error?

traversaro commented 3 years ago

Adding those missing packages manually does not fix the broken environment. Presumably some dependencies are still wrong versions, etc.

By inspecting manually the npe-gaz-condalist.txt and gaz383-condalist.txt, the most striking difference is that the Ogre version is 1.10.12 hbcc8020_2 for npe-gaz (not working), while for gaz383 is it is 1.10.12 h71cedee_6. The version is the same, but what is relevant here is the last part, that is the build number (the one that in the recipe is set in https://github.com/conda-forge/ogre-feedstock/blob/1.10/recipe/meta.yaml#L16). The build 2 of the 1.10.2 version of ogre is quite old (1 year and 1 month ago, according to https://anaconda.org/conda-forge/ogre/files) and does not contain important bug fixes such as https://github.com/conda-forge/ogre-feedstock/pull/23 .

I am not sure if we could debug in some way why 1.10.12 hbcc8020_2 was installed (@wolfv may know some tricks on how to do that why some conda/mamba command), but perhaps one strategy is try to run conda update ogre in npe-gaz and check if ogre gets updated to the build 6, and if not probably some message should be printed.

Something that I did not think about is that probably instead of saving the conda list output, it could make sense to save an environment.yaml file (with conda env export > environment.yml, see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment) to easily reproduce the issues.

I can't add models with the Insert tab, but maybe this is user error?

Is this happening also with the Gazebo installed by ROSOnWindows? In that case, I suggest to open a new issue in this repo as it is probably a Gazebo upstream problem. If instead if only happens with the conda binaries, please open an issue in https://github.com/conda-forge/gazebo-feedstock, thanks!

danzimmerman commented 3 years ago

Something that I did not think about is that probably instead of saving the conda list output, it could make sense to save an environment.yaml file

Yep, done below. I can't attach YAML directly, so I zipped all the referenced files: gzdbg-danzimmerman-2020-12-30.zip

I'm back on the work laptop starting fresh.

I can reproduce this issue with a simple starting .yml file. Creating this environment fails to solve for a working Gazebo binary:

name: gz-simple
channels:
  - defaults
  - conda-forge
dependencies:
  - python==3.8.3
  - gazebo

I can then verify that the installed OGRE build is at fault. Updating it explicitly to build h71cedee_6 allows Gazebo to run. This fix also works in my complex npe environment.

Minimal Reproduction

`gz-postinstall`

Created, empty, from CLI with conda create --name gz-postinstall python==3.8.3
Activated gz-postinstall and installed Gazebo with conda install -c conda-forge gazebo
WORKS: This results in a running Gazebo installation, model-add issues aside. OGRE is Build 6.
Exported to gz-postinstall-env-solved.yml

`gz-simple`

Created from gz-simple.yml.
Activated and ran gazebo --verbose
FAILS: Gazebo dies silently, as expected, same as npe environment, etc. OGRE is Build 2.
Exported to gz-simple-env-solved.yml

`gz-simple-clone`

Cloned gz-simple with conda create --clone gz-simple --name gz-simple-clone
Activated and ran gazebo --verbose
FAILS: Gazebo dies silently, as expected.
Exported to gz-simple-clone-preogre-env-solved.yml
Ran conda update ogre -> Says # All requested packages already installed
Ran conda install ogre=1.10.12=h71cedee_6 -c conda-forge
WORKS: This results in a running Gazebo installation, same model-add issues as gz-postinstall.
Exported to gz-simple-clone-postogre-env-solved.yml

danzimmerman commented 3 years ago

Is this happening also with the Gazebo installed by ROSOnWindows? In that case, I suggest to open a new issue in this repo as it is probably a Gazebo upstream problem. If instead if only happens with the conda binaries, please open an issue in https://github.com/conda-forge/gazebo-feedstock, thanks!

They both fail, but differently, and ROS on Windows Gazebo 9.13.1 has a simple fix. It was failing with tar: could not chdir to 'C:\Users\<username>/.gazebo/models' and adding an empty models subdir to 'C:\Users\<username>/.gazebo fixed it.

Perhaps related to

The Conda binary Gazebo 11 still fails on adding models from models.gazebosim.org with

[Err] [..\gazebo\common\ModelDatabase.cc:607] Missing model.config for model ""
[Err] [..\gazebo\common\ModelDatabase.cc:671] Invalid model manifest file[""]

And with fuel.ignitionrobotics.org models with:

[Err] [..\gazebo\common\FuelModelDatabase.cc:139] Missing model.config for model

I'll open a gazebo-feedstock issue for that.

traversaro commented 3 years ago

Thanks a lot @danzimmerman ! I think we are indeed reaching the core of the issue. Given that the issue is related to how the environment is created from a yaml file, could you also report the version of conda you have in the base environment?

danzimmerman commented 3 years ago

Thanks a lot @danzimmerman ! I think we are indeed reaching the core of the issue. Given that the issue is related to how the environment is created from a yaml file, could you also report the version of conda you have in the base environment?

The tests above were run with conda 4.8.4.

I updated the base environment conda 4.9.2 and it solves for the exact same set of packages. Installed with 4.9.2: gz-simple-condaupdate.yml-env-solved.txt

mamba 0.7.6 gets it right, however. Deleted and re-created gz-simple with mamba env create --file gz-simple.yml: gz-simple-env-mambasolved.yml.txt

That one works.

wolfv commented 3 years ago

thanks a million for testing this @danzimmerman !

traversaro commented 3 years ago

Hi @danzimmerman , I recently tried to spawn your gz-simple.yml environment, and it seems to get the latest version of ogre 1.10.12 . I am not sure what the specific issue one month ago, but I guess there were a few ABI migrations going on conda-forge that were not complete for Gazebo (libprotobuf314 in particular) that could have affected the installation process.

danzimmerman commented 3 years ago

Cool, will try it again soon.

danzimmerman commented 3 years ago

Conda-installing gazebo (and ROS Noetic) are working for me now.

wolfv commented 3 years ago

That's great to hear!

Y-Grigoriy commented 3 years ago

Conda-installing gazebo (and ROS Noetic) are working for me now.

Hello, thank you very much for your solution. Did you install ROS Noetic via Conda? And could you tell how the ROS can be connected with Gazebo? Sorry, if it is off-topic of this issue.

traversaro commented 3 years ago

Did you install ROS Noetic via Conda?

This is possible via the RoboStack channel, see the following link for reference:

And could you tell how the ROS can be connected with Gazebo?

ROS can be connected with Gazebo via the gazebo_ros_pkgs, see the Gazebo/ROS documentation in http://gazebosim.org/tutorials?tut=ros_overview for more info.

In general, for this kind of question a Answers&Question website like https://answers.ros.org/questions/ or https://answers.gazebosim.org/questions/ .

FirefoxMetzger commented 3 years ago

System: Win10 Build 19042.804
Graphics: Geforce RTX 3070, driver version 461.40, DirectX runtime 12.
Conda version: (mini)conda 4.9.2, gazebo version 11.3.0
Other: None (antivirus, firewall, etc. enabled)

I had to run the terminal as administrator (probably because conda is installed for all users), otherwise no problems during the installation.

When I start gazebo, I get 2 new processes. gzserver (flagged as non-responding by the task manager, but appears to work fine), and gzclient. The latter spawns two windows, one called gzclient (as expected) and one called OgreWindow(0), which doesn't seem to do much, and closing it doesn't seem to have any obvious consequences.

Admittedly, I haven't tested running any simulation or loading models yet. Is there a shared test scenario/world I should try?

Edit: I seem to have the same model reading/loading issues as @danzimmerman .

verbose stdout

``` >gazebo --verbose Gazebo multi-robot simulator, version 11.3.0 Copyright (C) 2012 Open Source Robotics Foundation. Released under the Apache 2 License. http://gazebosim.org Gazebo multi-robot simulator, version 11.3.0 Copyright (C) 2012 Open Source Robotics Foundation. Released under the Apache 2 License. http://gazebosim.org [Msg] Waiting for master. [Msg] Waiting for master. [Msg] [Msg] Connected to gazebo master @ Connected to gazebo master @ http://127.0.0.1:11345http://127.0.0.1:11345 [Msg] [Msg] Publicized address: Publicized address: [Msg] Loading world file [C:/ProgramData/Miniconda3\Library\share\gazebo-11\worlds/empty.world] [Err] [..\gazebo\common\FuelModelDatabase.cc:139] Missing model.config for model [Err] [..\gazebo\common\ModelDatabase.cc:607] Missing model.config for model "" [Err] [..\gazebo\common\ModelDatabase.cc:671] Invalid model manifest file[""] [Wrn] [..\gazebo\transport\Publisher.cc:136] Queue limit reached for topic /gazebo/default/user_camera/pose, deleting message. This warning is printed only once. [Err] [..\gazebo\common\ModelDatabase.cc:607] Missing model.config for model "" [Err] [..\gazebo\common\ModelDatabase.cc:671] Invalid model manifest file[""] [Err] [..\gazebo\common\ModelDatabase.cc:607] Missing model.config for model "" [Err] [..\gazebo\common\ModelDatabase.cc:671] Invalid model manifest file[""] ``` ... which, to me, looks like the pointer to the model is NULL instead of actually pointing to the object.

traversaro commented 3 years ago

When I start gazebo, I get 2 new processes. gzserver (flagged as non-responding by the task manager, but appears to work fine), and gzclient. The latter spawns two windows, one called gzclient (as expected) and one called Ogre Window(0), which doesn't seem to do much, and closing it doesn't seem to have any obvious consequences.

The gazebo command also for Windows was added recently in https://github.com/osrf/gazebo/pull/2864, so there could be problems. However, indeed it works by spawning two separate gzserver and gzclient process. I am not sure instead about the Ogre Window() window, but I guess that is also spawned if you launch gzclient on its own.

ImanHosseini commented 3 years ago

I use conda command prompt, and I get stuck on "[Msg] Waiting for master." weirdly, it randomly works sometimes [everything works] but most of the time it just gets stuck in "[Msg] Waiting for master." and it's not even generating any network traffic, and it's really weird, what I'm wondering is, why does it work? I don't get it, the few times that it did work, what was different?! Can this be a multi threading bug somewhere? update: no. It is not a multi-threading bug, when a previous instance is running and I exit, the new ones get stuck -> turns out, it is not terminating gracefully. It can be tracked and fixed, but for now: manually going after zombie gzclient/gzserver processes and killing them solves it.

wolfv commented 3 years ago

Can you send your OS, and the output of conda list?

wolfv commented 3 years ago

Or also conda info

ImanHosseini commented 3 years ago

win64. conda info:

     active environment : r2
    active env location : C:\Users\salva\miniconda3\envs\r2
            shell level : 2
       user config file : C:\Users\salva\.condarc
 populated config files : C:\Users\salva\.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.5.final.0
       virtual packages : __cuda=11.2=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : C:\Users\salva\miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/robostack/win-64
                          https://conda.anaconda.org/robostack/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : C:\Users\salva\miniconda3\pkgs
                          C:\Users\salva\.conda\pkgs
                          C:\Users\salva\AppData\Local\conda\conda\pkgs
       envs directories : C:\Users\salva\miniconda3\envs
                          C:\Users\salva\.conda\envs
                          C:\Users\salva\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.9.2 requests/2.24.0 CPython/3.8.5 Windows/10 Windows/10.0.21296
          administrator : False
             netrc file : None
           offline mode : False

conda list:

# Name                    Version                   Build  Channel
apr                       1.7.0                he38c35c_5    conda-forge
assimp                    5.0.1                hc2aa0de_5    conda-forge
bcrypt                    3.2.0            py38h294d835_1    conda-forge
boost                     1.74.0           py38h1266d08_3    conda-forge
boost-cpp                 1.74.0               h54f0996_2    conda-forge
bullet                    3.09                 h57928b3_1    conda-forge
bullet-cpp                3.09                 h60cbd38_1    conda-forge
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2020.12.5            h5b45459_0    conda-forge
catkin_pkg                0.4.23             pyh9f0ad1d_0    conda-forge
certifi                   2020.12.5        py38haa244fe_1    conda-forge
cffi                      1.14.5           py38hd8c33c5_0    conda-forge
cmake                     3.19.7               h39d44d4_0    conda-forge
console_bridge            1.0.1                h5362a0b_0    conda-forge
cppzmq                    4.7.1                h4324990_2    conda-forge
cryptography              3.4.7            py38hd7da0ea_0    conda-forge
curl                      7.76.0               hf1763fc_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
distro                    1.5.0              pyh9f0ad1d_0    conda-forge
dlfcn-win32               1.3.0                h0e60522_0    conda-forge
docutils                  0.16             py38haa244fe_3    conda-forge
eigen                     3.3.9                h2d74725_1    conda-forge
empy                      3.3.4              pyh9f0ad1d_1    conda-forge
expat                     2.3.0                h39d44d4_0    conda-forge
ffmpeg                    4.3.1                ha925a31_0    conda-forge
freeglut                  3.2.1                h0e60522_2    conda-forge
freeimage                 3.18.0               hb49e22e_7    conda-forge
freetype                  2.10.4               h546665d_1    conda-forge
gazebo                    11.4.0               h83ca5bb_0    conda-forge
gettext                   0.19.8.1          h1a89ca6_1005    conda-forge
gmock                     1.10.0               h2d74725_7    conda-forge
gtest                     1.10.0               h2d74725_7    conda-forge
gts                       0.7.6                h7c369d9_2    conda-forge
icu                       68.1                 h0e60522_0    conda-forge
ilmbase                   2.5.5                h12d4b20_0    conda-forge
intel-openmp              2020.3             h57928b3_311    conda-forge
jasper                    2.0.14               h77af90b_2    conda-forge
jpeg                      9d                   h8ffe710_0    conda-forge
jsoncpp                   1.9.4                h2d74725_1    conda-forge
jxrlib                    1.1                  h8ffe710_2    conda-forge
krb5                      1.17.2               hbae68bd_0    conda-forge
lcms2                     2.12                 h2a16943_0    conda-forge
libapr                    1.7.0                h8ffe710_5    conda-forge
libapriconv               1.2.2                h8ffe710_5    conda-forge
libaprutil                1.6.1                h311b4f7_5    conda-forge
libblas                   3.9.0                     8_mkl    conda-forge
libcblas                  3.9.0                     8_mkl    conda-forge
libclang                  11.1.0          default_h5c34c98_0    conda-forge
libcurl                   7.76.0               hf1763fc_0    conda-forge
libffi                    3.3                  h0e60522_2    conda-forge
libglib                   2.68.0               h1e62bf3_2    conda-forge
libiconv                  1.16                 he774522_0    conda-forge
libignition-cmake2        2.7.0                h0e60522_1    conda-forge
libignition-common3       3.11.1               he3878d5_0    conda-forge
libignition-fuel-tools4   4.3.0                h4936f55_2    conda-forge
libignition-math6         6.8.0                h0e60522_0    conda-forge
libignition-msgs5         5.3.0                hf5fa1a2_6    conda-forge
libignition-tools1        1.1.0                h57928b3_0    conda-forge
libignition-transport8    8.1.0                h9704849_6    conda-forge
liblapack                 3.9.0                     8_mkl    conda-forge
liblapacke                3.9.0                     8_mkl    conda-forge
libopencv                 4.5.1            py38he608210_1    conda-forge
libpng                    1.6.37               h1d00b33_2    conda-forge
libprotobuf               3.15.6               h7755175_0    conda-forge
libraw                    0.20.2               hee1bdec_1    conda-forge
libsdformat               9.3.0                h4bf3a07_3    conda-forge
libsodium                 1.0.18               h8d14728_1    conda-forge
libssh2                   1.9.0                h680486a_6    conda-forge
libtiff                   4.2.0                hc10be44_0    conda-forge
libwebp                   1.2.0                h57928b3_0    conda-forge
libwebp-base              1.2.0                h8ffe710_2    conda-forge
libzip                    1.7.3                hfed4ece_0    conda-forge
log4cxx                   0.11.0               hf7dfa5a_3    conda-forge
lz4                       3.1.3            py38h34afd45_0    conda-forge
lz4-c                     1.9.3                h8ffe710_0    conda-forge
mkl                       2020.4             hb70f87d_311    conda-forge
netifaces                 0.10.9          py38h294d835_1003    conda-forge
nose                      1.3.7                   py_1006    conda-forge
numpy                     1.20.2           py38h09042cb_0    conda-forge
ogre                      1.10.12              hadbb816_7    conda-forge
openexr                   2.5.5                hab3b255_0    conda-forge
openjpeg                  2.4.0                h48faf41_0    conda-forge
openssl                   1.1.1k               h8ffe710_0    conda-forge
orocos-kdl                1.4.0                h33f27b4_0    conda-forge
paramiko                  2.7.2              pyh9f0ad1d_0    conda-forge
pcre                      8.44                 ha925a31_0    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
pkg-config                0.29.2            h2bf4dc2_1008    conda-forge
poco                      1.10.1               h1176d77_1    conda-forge
protobuf                  3.15.6           py38h885f38d_0    conda-forge
pugixml                   1.11.4               h0e60522_0    conda-forge
py-opencv                 4.5.1            py38h43734a8_1    conda-forge
pybullet                  3.09             py38h60cbd38_1    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pycryptodome              3.10.1           py38h294d835_0    conda-forge
pycryptodomex             3.10.1           py38h294d835_0    conda-forge
pynacl                    1.4.0            py38h31c79cd_2    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pyqt                      5.12.3           py38haa244fe_7    conda-forge
pyqt-impl                 5.12.3           py38h885f38d_7    conda-forge
pyqt5-sip                 4.19.18          py38h885f38d_7    conda-forge
pyqtchart                 5.12             py38h885f38d_7    conda-forge
pyqtwebengine             5.12.1           py38h885f38d_7    conda-forge
python                    3.8.8           h7840368_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python-gnupg              0.4.6              pyh9f0ad1d_0    conda-forge
python-orocos-kdl         1.4.0            py38h7ae7562_0    conda-forge
python_abi                3.8                      1_cp38    conda-forge
pyyaml                    5.4.1            py38h294d835_0    conda-forge
qt                        5.12.9               h5909a2a_4    conda-forge
qwt                       6.1.6                h552f0f6_0    conda-forge
ros-distro-mutex          0.1.0                    noetic    robostack
ros-noetic-actionlib      1.13.2           py38h508dd2d_5    robostack
ros-noetic-actionlib-msgs 1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-amcl           1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-angles         1.9.13           py38h4b9bc1a_5    robostack
ros-noetic-base-local-planner 1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-bond           1.8.6            py38h4b9bc1a_5    robostack
ros-noetic-bondcpp        1.8.6            py38h508dd2d_5    robostack
ros-noetic-camera-calibration-parsers 1.12.0           py38h7c8cae0_5    robostack
ros-noetic-camera-info-manager 1.12.0           py38h7c8cae0_5    robostack
ros-noetic-catkin         0.8.9            py38h4b9bc1a_5    robostack
ros-noetic-class-loader   0.5.0            py38h63b21c9_5    robostack
ros-noetic-clear-costmap-recovery 1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-control-msgs   1.5.2            py38h4b9bc1a_5    robostack
ros-noetic-control-toolbox 1.18.2           py38h4b9bc1a_5    robostack
ros-noetic-controller-interface 0.19.4           py38h4b9bc1a_5    robostack
ros-noetic-controller-manager 0.19.4           py38h4b9bc1a_5    robostack
ros-noetic-controller-manager-msgs 0.19.4           py38h4b9bc1a_5    robostack
ros-noetic-costmap-2d     1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-cpp-common     0.7.2            py38hd43ebe7_5    robostack
ros-noetic-cv-bridge      1.15.0           py38hc316dd2_5    robostack
ros-noetic-diagnostic-msgs 1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-diagnostic-updater 1.10.3           py38h4b9bc1a_5    robostack
ros-noetic-dynamic-reconfigure 1.7.1            py38h508dd2d_5    robostack
ros-noetic-gazebo-dev     2.9.1            py38hb42fc0a_5    robostack
ros-noetic-gazebo-msgs    2.9.1            py38h4b9bc1a_5    robostack
ros-noetic-gazebo-plugins 2.9.1            py38h4b9bc1a_5    robostack
ros-noetic-gazebo-ros     2.9.1            py38h4b9bc1a_5    robostack
ros-noetic-gazebo-ros-control 2.9.1            py38h4b9bc1a_5    robostack
ros-noetic-gencpp         0.6.5            py38h4b9bc1a_5    robostack
ros-noetic-geneus         3.0.0            py38h4b9bc1a_5    robostack
ros-noetic-genlisp        0.4.18           py38h4b9bc1a_5    robostack
ros-noetic-genmsg         0.5.16           py38h4b9bc1a_5    robostack
ros-noetic-gennodejs      2.0.2            py38h4b9bc1a_5    robostack
ros-noetic-genpy          0.6.14           py38h4b9bc1a_5    robostack
ros-noetic-geometry-msgs  1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-gmapping       1.4.2            py38h4b9bc1a_5    robostack
ros-noetic-hardware-interface 0.19.4           py38h4b9bc1a_6    robostack
ros-noetic-hector-gazebo-plugins 0.5.3            py38h4b9bc1a_5    robostack
ros-noetic-image-transport 1.12.0           py38h4b9bc1a_5    robostack
ros-noetic-interactive-markers 1.12.0           py38h4b9bc1a_5    robostack
ros-noetic-jackal-desktop 0.4.0            py38h4b9bc1a_0    robostack
ros-noetic-jackal-gazebo  0.4.0            py38h4b9bc1a_0    robostack
ros-noetic-jackal-navigation 0.7.3            py38h4b9bc1a_0    robostack
ros-noetic-jackal-viz     0.4.0            py38h4b9bc1a_0    robostack
ros-noetic-joint-limits-interface 0.19.4           py38h4b9bc1a_5    robostack
ros-noetic-joint-state-publisher 1.15.0           py38h4b9bc1a_5    robostack
ros-noetic-joint-state-publisher-gui 1.15.0           py38h4b9bc1a_5    robostack
ros-noetic-laser-geometry 1.6.7            py38h7c8cae0_5    robostack
ros-noetic-map-msgs       1.14.1           py38h4b9bc1a_5    robostack
ros-noetic-map-server     1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-media-export   0.3.0            py38h4b9bc1a_5    robostack
ros-noetic-message-filters 1.15.9           py38h508dd2d_5    robostack
ros-noetic-message-generation 0.4.1            py38h4b9bc1a_5    robostack
ros-noetic-message-runtime 0.4.13           py38h4b9bc1a_5    robostack
ros-noetic-move-base      1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-move-base-msgs 1.14.1           py38h4b9bc1a_5    robostack
ros-noetic-nav-core       1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-nav-msgs       1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-navfn          1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-nodelet        1.10.1           py38h7c8cae0_5    robostack
ros-noetic-openslam-gmapping 0.2.1            py38h4b9bc1a_5    robostack
ros-noetic-pluginlib      1.13.0           py38h508dd2d_5    robostack
ros-noetic-polled-camera  1.12.0           py38h4b9bc1a_5    robostack
ros-noetic-python-qt-binding 0.4.3            py38h4b9bc1a_5    robostack
ros-noetic-realtime-tools 1.16.1           py38h4b9bc1a_5    robostack
ros-noetic-resource-retriever 1.12.6           py38hc131489_5    robostack
ros-noetic-ros-environment 1.3.2            py38h4b9bc1a_5    robostack
ros-noetic-rosbag         1.15.9           py38h84b7d0c_5    robostack
ros-noetic-rosbag-migration-rule 1.0.1            py38h4b9bc1a_5    robostack
ros-noetic-rosbag-storage 1.15.9           py38hf19f0dc_5    robostack
ros-noetic-rosbuild       1.15.7           py38h4b9bc1a_5    robostack
ros-noetic-rosclean       1.15.7           py38h4b9bc1a_5    robostack
ros-noetic-rosconsole     1.14.3           py38h96c7dd2_5    robostack
ros-noetic-rosconsole-bridge 0.5.4            py38h1548a2d_5    robostack
ros-noetic-roscpp         1.15.9           py38h508dd2d_5    robostack
ros-noetic-roscpp-serialization 0.7.2            py38h4b9bc1a_5    robostack
ros-noetic-roscpp-traits  0.7.2            py38h4b9bc1a_5    robostack
ros-noetic-rosgraph       1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosgraph-msgs  1.11.3           py38h4b9bc1a_5    robostack
ros-noetic-roslaunch      1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-roslib         1.15.7           py38h508dd2d_5    robostack
ros-noetic-roslz4         1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosmaster      1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosmsg         1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosnode        1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosout         1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rospack        2.6.2            py38h84b7d0c_5    robostack
ros-noetic-rosparam       1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rospy          1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosservice     1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rostest        1.15.9           py38h508dd2d_5    robostack
ros-noetic-rostime        0.7.2            py38h508dd2d_5    robostack
ros-noetic-rostopic       1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rosunit        1.15.7           py38h4b9bc1a_5    robostack
ros-noetic-roswtf         1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-rotate-recovery 1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-rviz           1.14.5           py38h4b9bc1a_5    robostack
ros-noetic-sensor-msgs    1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-smclib         1.8.6            py38h4b9bc1a_5    robostack
ros-noetic-std-msgs       0.5.13           py38h4b9bc1a_5    robostack
ros-noetic-std-srvs       1.11.3           py38h4b9bc1a_5    robostack
ros-noetic-tf             1.13.2           py38h4b9bc1a_5    robostack
ros-noetic-tf2            0.7.5            py38h1548a2d_5    robostack
ros-noetic-tf2-geometry-msgs 0.7.5            py38h4b9bc1a_5    robostack
ros-noetic-tf2-msgs       0.7.5            py38h4b9bc1a_5    robostack
ros-noetic-tf2-py         0.7.5            py38h4b9bc1a_5    robostack
ros-noetic-tf2-ros        0.7.5            py38h4b9bc1a_5    robostack
ros-noetic-topic-tools    1.15.9           py38h4b9bc1a_5    robostack
ros-noetic-trajectory-msgs 1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-transmission-interface 0.19.4           py38h4b9bc1a_5    robostack
ros-noetic-urdf           1.13.2           py38h4b9bc1a_5    robostack
ros-noetic-visualization-msgs 1.13.1           py38h4b9bc1a_5    robostack
ros-noetic-voxel-grid     1.17.1           py38h4b9bc1a_5    robostack
ros-noetic-xacro          1.14.6           py38h4b9bc1a_5    robostack
ros-noetic-xmlrpcpp       1.15.9           py38h508dd2d_5    robostack
rosdep                    0.20.0           py38haa244fe_0    conda-forge
rosdistro                 0.8.3            py38haa244fe_2    conda-forge
rospkg                    1.2.10             pyh44b312d_0    conda-forge
ruby                      2.7.2                h8b1b97a_3    conda-forge
sdl                       1.2.15               h21ff451_1    conda-forge
sdl2                      2.0.12               h0e60522_1    conda-forge
sdl_image                 1.2.12               h368d6c2_1    conda-forge
setuptools                49.6.0           py38haa244fe_3    conda-forge
sip                       4.19.20          py38h6538335_0    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sqlite                    3.35.3               h8ffe710_0    conda-forge
tbb                       2020.2               h2d74725_4    conda-forge
tbb-devel                 2020.2               h2d74725_4    conda-forge
tiny-process-library      2.0.4                h0e60522_0    conda-forge
tinyxml                   2.6.2                h2d74725_2    conda-forge
tinyxml2                  8.0.0                h39d44d4_1    conda-forge
tk                        8.6.10               h8ffe710_1    conda-forge
urdfdom                   2.3.3                h7ef1ec2_0    conda-forge
urdfdom_headers           1.0.5                h5362a0b_2    conda-forge
vc                        14.2                 hb210afc_4    conda-forge
vs2015_runtime            14.28.29325          h5e1d092_4    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
wincertstore              0.2             py38haa244fe_1006    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge
yaml                      0.2.5                he774522_0    conda-forge
yaml-cpp                  0.6.3                ha925a31_4    conda-forge
zeromq                    4.3.4                h0e60522_0    conda-forge
zlib                      1.2.11            h62dcd97_1010    conda-forge
zstd                      1.4.9                h6255e5f_0    conda-forge
zziplib                   0.13.69              h1d00b33_1    conda-forge

wolfv commented 3 years ago

Thanks for tracking the bug down! I wonder if this is a general gazebo-on-windows problem or specific to conda-forge?

ImanHosseini commented 3 years ago

It seems windows specific. If we just document this behaviour somewhere, it would be fine, so people know to kill zombie gzclient/gzserver/gazebo process if they get stuck in "Waiting for master." and everything else is fine, and it doesn't happen that much, like, if you start gazebo, and prematurely kill it superfast before it's all up, or some unnatural thing like that this issue might happen. I tried investigating further but it gets even weirder => if you do this on purpose (it may take multiple time to get the behaviour) and then try to attach a debugger to see what is the state of that "zombie" process, it just exits right away [the new gazebo would be -unstuck- and start working] not letting you attach, if you try dumping it => you get a broken empty dump.

traversaro commented 3 years ago

gzserver not properly closing and preventing new gzserver to be spawned tend to happen also on other platforms, there are a few PRs working on it:

Perhaps @ivanpauno can provide tips on how debug further failures of this kind.

If we just document this behaviour somewhere, it would be fine, so people know to kill zombie gzclient/gzserver/gazebo process if they get stuck in "Waiting for master." and everything else is fine, and it doesn't happen that much, like, if you start gazebo, and prematurely kill it superfast before it's all up, or some unnatural thing like that this issue might happen.

This make sense. I think at this point given the amount of feedback on this issue, it could make sense if we move some of the knowledge contained in this issue (i.e. how to install it and a few known issues, such as the one that you mentioned) in a piece of docs in https://github.com/osrf/gazebo_tutorials , what do you think @j-rivero ?

j-rivero commented 3 years ago

If we just document this behaviour somewhere, it would be fine, so people know to kill zombie gzclient/gzserver/gazebo process if they get stuck in "Waiting for master." and everything else is fine, and it doesn't happen that much, like, if you start gazebo, and prematurely kill it superfast before it's all up, or some unnatural thing like that this issue might happen.

This make sense. I think at this point given the amount of feedback on this issue, it could make sense if we move some of the knowledge contained in this issue (i.e. how to install it and a few known issues, such as the one that you mentioned) in a piece of docs in https://github.com/osrf/gazebo_tutorials , what do you think @j-rivero ?

I think that you guys did a great work on debugging and clarifying the whole thing, thanks so much. I would be happy to review a PR against gazebo_tutorials. If you don't want to deal with the format of the docs, I can translate a markdown format if that helps. Feel free to assign the PR to me. I'm looking for a bit of time to test the conda installation again.

ivanpauno commented 3 years ago

Perhaps @ivanpauno can provide tips on how debug further failures of this kind.

On windows? Oh sorry, my debugging abilities on that platform are terrible :joy:.

In linux what I did was finding a simulation where I could reproduce the issue. In my case I was spawning a robot in a gazebo world with some sensor plugins, like DepthCameraPlugin, and using a launch file to set up that. I then started and ctrl-c the simulation repeatedly until I found gzserver hanging. Then, attached with gdb, printed a multithreaded traceback a few times to see which threads were making progress, and started the guessing work to see what was blocking what.

I also added extra cpu stress to make the issue easier to reproduce. For one of the bugs, I build gazebo from source with debugging symbols.

Doing something similar on Windows should be possible (e.g. https://docs.microsoft.com/en-us/visualstudio/debugger/attach-to-running-processes-with-the-visual-studio-debugger?view=vs-2019#:~:text=You%20can%20attach%20the%20Visual,the%20debugger%20to%20the%20process.), but I don't have experience in that.

traversaro commented 3 years ago

Perhaps @ivanpauno can provide tips on how debug further failures of this kind.

On windows? Oh sorry, my debugging abilities on that platform are terrible 😂.

Don't worry, the step that you described apply to more and less any platform with a debugger, thanks! Furthermore, I actually suspect that most bug of this kind are rather multiplatform, but this is just my personal guess.

ImanHosseini commented 3 years ago

Perhaps @ivanpauno can provide tips on how debug further failures of this kind.

On windows? Oh sorry, my debugging abilities on that platform are terrible 😂.

In linux what I did was finding a simulation where I could reproduce the issue. In my case I was spawning a robot in a gazebo world with some sensor plugins, like DepthCameraPlugin, and using a launch file to set up that. I then started and ctrl-c the simulation repeatedly until I found gzserver hanging. Then, attached with gdb, printed a multithreaded traceback a few times to see which threads were making progress, and started the guessing work to see what was blocking what.

I also added extra cpu stress to make the issue easier to reproduce. For one of the bugs, I build gazebo from source with debugging symbols.

Doing something similar on Windows should be possible (e.g. https://docs.microsoft.com/en-us/visualstudio/debugger/attach-to-running-processes-with-the-visual-studio-debugger?view=vs-2019#:~:text=You%20can%20attach%20the%20Visual,the%20debugger%20to%20the%20process.), but I don't have experience in that.

Thanks, this is really helpful! I did track down what's going on in my case (on windows) and it was indeed this issue: https://github.com/osrf/gazebo/pull/2950 and the suggested change seems to fix it.

traversaro commented 3 years ago

Thanks, this is really helpful! I did track down what's going on in my case (on windows) and it was indeed this issue: #2950 and the suggested change seems to fix it.

The change in #2950 was included in Gazebo Classic 11.5.0, that was released yesterday and for which conda-forge binaries are now available: https://github.com/conda-forge/gazebo-feedstock/pull/67 . Hopefully then that problem was solved.

PrasadNR commented 3 years ago

I am using Windows 8. I just did conda install -c conda-forge gazebo and then simply typed gazebo in the Anaconda prompt. This Gazebo does not work sometimes and sometimes it does (Same "Windows is finding a solution to the problem" error). I'm not sure why though.

traversaro commented 3 years ago

I am using Windows 8. I just did conda install -c conda-forge gazebo and then simply typed gazebo in the Anaconda prompt. This Gazebo does not work sometimes and sometimes it does (Same "Windows is finding a solution to the problem" error). I'm not sure why though.

Hi @PrasadNR, thanks for the report! Indeed I am not sure anyone ever tested this on Windows 8. Could you open an issue in https://github.com/conda-forge/gazebo-feedstock/issues/new, with all the information required by the issue template? In particular, the list of packages that you obtain with conda list are particular interesting.

Other things that could be helpful for debug:

Run gazebo --verbose
Try to first run gzserver --verbose in a terminal and gzclient --verbose in another

I wonder if this comment https://github.com/conda-forge/vc-feedstock/pull/37#issuecomment-861072860 is related. I remember that when I read it I tought it was a purely teoretical problem, but that is not the case it seems. : )

gazebosim / gazebo-classic

Gazebo on Windows: conda-forge installation #2899

Expected Behavior

Actual Behavior

Fresh Environment (Runs) vs. Existing Environment (Fails) - Anaconda Prompt

Outstanding Possible Issues

Files

Environment vars

`conda list` outputs

Summary

Minimal Reproduction

`gz-postinstall`

`gz-simple`

`gz-simple-clone`

gazebosim / gazebo-classic

Gazebo on Windows: conda-forge installation #2899

Expected Behavior

Actual Behavior

Fresh Environment (Runs) vs. Existing Environment (Fails) - Anaconda Prompt

Outstanding Possible Issues

Files

Environment vars

conda list outputs

Summary

Minimal Reproduction

gz-postinstall

gz-simple

gz-simple-clone

`conda list` outputs

`gz-postinstall`

`gz-simple`

`gz-simple-clone`