search

robotology-legacy / gym-ignition

Framework for developing OpenAI Gym robotics environments simulated with Ignition Gazebo
https://robotology.github.io/gym-ignition
GNU Lesser General Public License v3.0
235 stars 26 forks source link

gazebo.insert_world_from_sdf fails to download models from fuel #296

Closed FirefoxMetzger closed 3 years ago

FirefoxMetzger commented 3 years ago

I am trying to load an existing world from an .sdf file into gazebo using gym-ignition. Here is my code

from scenario import gazebo as scenario_gazebo

gazebo = scenario_gazebo.GazeboSimulator(
    step_size=0.001,
    rtf=1.0,
    steps_per_run=1
)
gazebo.insert_world_from_sdf("/path/to/world.sdf")

gazebo.initialize()
gazebo.gui()

and the coresponding world.sdf

world.sdf ```xml 0.001 1.0 ogre 3D View false docked ogre2 scene 0.4 0.4 0.4 0.8 0.8 0.8 2 2 2 0 0.5 -2.5 World control false false 72 121 1 floating true true true World stats false false 110 290 1 floating true true true true 600 400 floating panda true 0 0 10 0 0 0 0.8 0.8 0.8 1 0.2 0.2 0.2 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 true 0 0 1 100 100 0 0 1 100 100 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0.8 0.8 0.8 1 https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Panda with Ignition position controller model 0.2 0 1.025 0 0 0 https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table table1 0 0 0 0 0 1.5708 https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table table2 0.794 0 0 0 0 1.5708 https://fuel.ignitionrobotics.org/1.0/GoogleResearch/models/Avengers_Thor_PLlrpYniaeB 1 0.5 1.025 0 0 0 4 0 1.0 0 0.0 3.14 0.05 0.05 0.05 0 0 0 0.1 0.000166667 0.000166667 0.000166667 0.1 0.1 0.1 0.1 0.1 0.1 1.047 320 240 0.1 100 1 30 true camera ```

The world opens fine in gazebo itself (ign gazebo world.sdf), but when I run it via the script I get the following error messages:

Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?Warning [Model.cc:212] Non-unique names detected in XML children of model with name[panda].
[Err] [helpers.cpp:58] Failed to load sdf file /home/sebastian/panda-python-sim/panda_world.sdf
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table]
[Err] [helpers.cpp:61] Error Code 11 Msg: Invalid uri[https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table]. Should be model://https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table]
[Err] [helpers.cpp:61] Error Code 11 Msg: Invalid uri[https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table]. Should be model://https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[https://fuel.ignitionrobotics.org/1.0/GoogleResearch/models/Avengers_Thor_PLlrpYniaeB]
[Err] [helpers.cpp:61] Error Code 11 Msg: Invalid uri[https://fuel.ignitionrobotics.org/1.0/GoogleResearch/models/Avengers_Thor_PLlrpYniaeB]. Should be model://https://fuel.ignitionrobotics.org/1.0/GoogleResearch/models/Avengers_Thor_PLlrpYniaeB
Setting callback for signal SIGINT
Setting callback for signal SIGTERM
Setting callback for signal SIGABRT

I'm still learning about both ignition and gym-ignition, so this could very well be user-error on my part. Any ideas where things fall apart and where I should look to fix it? I'm happy to do a PR (if necessary) once a solution has been identified.

Edit: I should add that I tried changing to the suggested model://https://... URI, but this results in an error stating that the resource could not be found.

Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?Error [SDF.cc:163] Tried to use callback in sdf::findFile(), but the callback is empty.  Did you call sdf::setFindCallback()?[Err] [helpers.cpp:58] Failed to load sdf file /home/sebastian/panda-python-sim/panda_world.sdf
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[model://https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Panda with Ignition position controller model]
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[model://https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table]
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[model://https://fuel.ignitionrobotics.org/1.0/OpenRobotics/models/Table]
[Err] [helpers.cpp:61] Error Code 12 Msg: Unable to find uri[model://https://fuel.ignitionrobotics.org/1.0/GoogleResearch/models/Avengers_Thor_PLlrpYniaeB]
Setting callback for signal SIGINT
Setting callback for signal SIGTERM
Setting callback for signal SIGABRT
diegoferigo commented 3 years ago

We have partial Fuel support and we currently use and test the programmatic loading of models from fuel:

https://github.com/robotology/gym-ignition/blob/fe5a1afd2b1d418c9af49275ebb5d482166d71b4/examples/panda_pick_and_place.py#L104-L124

A use-case we never yet explored is loading an SDF file that has models specified as fuel resources. I suspect this could be related to #168 that got stale.

Please note that the world file you posted has a lot of plugins in it. You can find the default empty world that we use here. The idea behind gym-ignition is to interact with the simulator programmatically (for instance by populating it from code). This is also the logic we followed for plugins, as reported in the Tip box.

For initial testing, I would suggest to start from the default empty world (that is loaded by default if you don't specify any world sdf). Then, you can 1) insert the models from Fuel and 2) and loading the plugins you need through the Python APIs.

Then, if there's the need to support loading structured world by default, we can discuss of the actions to take to add the support. I don't see major problems, it should be doable with minor modifications.

FirefoxMetzger commented 3 years ago

The idea behind gym-ignition is to interact with the simulator programmatically (for instance by populating it from code).

I really like this idea, and it is the main reason I chose to work with gym-ignition. Bonus points for implementing the OpenAI gym interface, which I am very familiar with.

I will try adding the models step-by-step in python and see how far this gets me.

it should be doable with minor modifications

Looking at the bigger picture, I think full .sdf support is very desirable. It allows separating configuration and code (which increases the reusability of either), it can also be used to generate worlds by either applying preprocessing before loading the simulator (ala world.sdf.in and substituting variables) or writing a dedicated procedural generation algorithm that exists without dependency on a particular ignition or gym-ignition version.

At least to me, this seems like a quite clean solution to the problem; though it may not actually add any unique new features. I checked the existing stale PR that you mentioned, and I hope to get the opportunity to pick it up some time next week. Also, this issue does seem like a dubplicate of #168 at first glance.

FirefoxMetzger commented 3 years ago

I explored a little further, and it seems that the function gazebo.insert_world_from_sdf seems to have no effect, even with simple worlds. I generated a world using gym-ignition

from scenario import gazebo as scenario_gazebo
import gym_ignition_models
import time

gazebo = scenario_gazebo.GazeboSimulator(
    step_size=0.001,
    rtf=1.0,
    steps_per_run=1
)
gazebo.initialize()

world = gazebo.get_world()
world.insert_model(gym_ignition_models.get_model_file("ground_plane"))

gazebo.gui()
gazebo.run(paused=True)

time.sleep(60)
gazebo.close()

which opens the simulator showing an empty world with ground plane (as expected). I then saved the created world using the GUI both with tags expanded and not exapanded.

generated_world.sdf ```xml 0.001 1 -1 1 0 0 10 0 -0 0 1 1 1 1 0.5 0.5 0.5 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 0 0 0 0 0 -9.8 6e-06 2.3e-05 -4.2e-05 0.4 0.4 0.4 1 0.7 0.7 0.7 1 1 file:///home/sebastian/.local/lib/python3.8/site-packages/gym_ignition_models/ground_plane ground_plane 0 0 0 0 -0 0 ```
generated_world_expanded.sdf ```xml 0.001 1 -1 1 0 0 10 0 -0 0 1 1 1 1 0.5 0.5 0.5 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 0 0 0 0 0 -9.8 6e-06 2.3e-05 -4.2e-05 0.4 0.4 0.4 1 0.7 0.7 0.7 1 1 1 0 0 1 1 1 100 1 0 0 1 100 100 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0 0 0 0 -0 0 ```

Finally, I tried to load the generated worlds, expecting to see the same world as the one created procedually via 

from scenario import gazebo as scenario_gazebo
import gym_ignition_models
import time

scenario_gazebo.set_verbosity(scenario_gazebo.Verbosity_info)

gazebo = scenario_gazebo.GazeboSimulator(
    step_size=0.001,
    rtf=1.0,
    steps_per_run=1
)
gazebo.initialize()

gazebo.insert_world_from_sdf("./generated_world.sdf")
# gazebo.insert_world_from_sdf("./generated_world_expanded.sdf")

gazebo.gui()
gazebo.run(paused=True)

time.sleep(3)
gazebo.close()

However, in both cases the result was an empty world.

result_world.sdf ```xml 0.001 1 -1 1 0 0 10 0 -0 0 1 1 1 1 0.5 0.5 0.5 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 0 0 0 0 0 -9.8 6e-06 2.3e-05 -4.2e-05 0.4 0.4 0.4 1 0.7 0.7 0.7 1 1 ```

@diegoferigo would you happen to have a working example of gazebo.insert_world_from_sdf, or is it known to be (currently) broken?

diegoferigo commented 3 years ago

This is expected, the world must be set before initializing the simulator. During initialization, the default world is loaded if a custom world is not specified. We have some CI test for these core features.

I was pretty sure there should be an error printed to the console in this case, but apparently the log level we used is wrong:

https://github.com/robotology/gym-ignition/blob/fe5a1afd2b1d418c9af49275ebb5d482166d71b4/cpp/scenario/gazebo/src/GazeboSimulator.cpp#L386-L390

We should have sError there. You should see that message printed if you change the log verbosity:

scenario_gazebo.set_verbosity(scenario_gazebo.Verbosity_info)

For how the C++ APIs are structured, these methods return False if the do not succeed, therefore if you want to be sure that everything went fine you can use any of the following options to check. I use gazebo.initialize() as reference, but this applies to all the functions that return a boolean.

# Exceptions
if not gazebo.initialize():
    raise RuntimeError

# Assertions: be careful that in python asserts are disabled when running with -O
assert gazebo.initialize()

# Safer assertions
ok_initialize = gazebo.initialize()
assert ok_initialize

I then saved the created world using the GUI both with tags expanded and not exapanded.

I don't how what happened here. I tried on my setup and the exported world is the following, and it seems correct. Is your output really the content of your file? Do you have the same output also if you export the world opening the simulator with ign gazebo?

exported.sdf ```xml 0.001 1 -1 1 0 0 10 0 -0 0 1 1 1 1 0.5 0.5 0.5 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 0 0 0 0 0 -9.8 6e-06 2.3e-05 -4.2e-05 0.4 0.4 0.4 1 0.7 0.7 0.7 1 1 1 0 0 1 1 1 100 1 0 0 1 100 100 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0 0 0 0 -0 0 ```

it should be doable with minor modifications

Looking at the bigger picture, I think full .sdf support is very desirable. It allows separating configuration and code (which increases the reusability of either), it can also be used to generate worlds by either applying preprocessing before loading the simulator (ala world.sdf.in and substituting variables) or writing a dedicated procedural generation algorithm that exists without dependency on a particular ignition or gym-ignition version.

At least to me, this seems like a quite clean solution to the problem; though it may not actually add any unique new features. I checked the existing stale PR that you mentioned, and I hope to get the opportunity to pick it up some time next week. Also, this issue does seem like a dubplicate of #168 at first glance.

I totally agree with you, it would be nice to have full SDF support. I could also think to add an option gazebo.insert_world_from_sdf("world.sdf", load_plugins=False) that is False by default, so that all the worlds found online can be compatible with it. This way, no hidden plugin loading would happen under the hood, while it allows users to enable their support if they explicitly intended.

Beyond the sdf templating that you proposed, that is an interesting practice (and quite easy to do in Python), another great reason to fully support complex worlds with Fuel resources is the new model composition logic of SDF. Now it is possible to insert models specifying poses relative to other models, and this greatly simplifies the creation of dynamic worlds that adapt automatically if any of the <include>d models gets an update that e.g. slightly changes their size. With gym-ignition it was already possible obtaining something similar from code, but now there's full support also from SDF, and a single SDF file could be easier to distribute and maintain rather than a Python script.

FirefoxMetzger commented 3 years ago

This is expected, the world must be set before initializing the simulator.

Yes! I remember having seen a comment about this in the source code when skimming it to figure out how to use insert_world_from_sdf, but I have obviously forgotten about it further down the line. Thanks for pointing that out.

Do you think we should update the documentation with a comment or note in the form of Note: Can only be called on an uninitialized simulator.? Maybe this can be combined with escalating the error message in the logs.

I don't know how the current python bindings look like, but an additional step could be to raise an appropriate exception (perhaps RuntimeError("insert_world_from_sdf can not be called after the simulator has been initialized.")), since - in my mind - a user's expectation would be for commands to succeed and, consequentially, they may wish to handle any failure explicitly, i.e., change the behavior to fail noisily instead of silently.

Is your output really the content of your file? Do you have the same output also if you export the world opening the simulator with ign gazebo?

ign gazebo generated_world.sdf and ign gazebo generated_world_expanded.sdf load the world including the ground plane as expected. Resaving the world as a new file (compare.sdf) from the GUI and inspecting the diff shows no difference between generated_world.sdf and compare.sdf. There also is no diff between the sdf you posted and generated_world_expanded.sdf. From this, I conclude that it is indeed the same world file. If this isn't what you had in mind, please let me know.

I moved gazebo.insert_world_from_sdf before gazebo.initialize. The minimal code to load the file looks like this

from scenario import gazebo as scenario_gazebo
import time

scenario_gazebo.set_verbosity(scenario_gazebo.Verbosity_debug)

gazebo = scenario_gazebo.GazeboSimulator(
    step_size=0.001,
    rtf=1.0,
    steps_per_run=1
)

# Note: I also tried the absolute path
assert gazebo.insert_world_from_sdf("./generated_world.sdf")
gazebo.initialize()

gazebo.gui()
gazebo.run(paused=True)

time.sleep(3)
gazebo.close()

The assert ... succeeds, but the world is not loaded. There is one suspicious message in the logs: [Msg] Loading SDF string. File path not available., which makes me think that the file may not be found? Interestingly, this doesn't seem to cause the call to fail; I will see if I can find the relevant section in the source code.

I could also think to add an option gazebo.insert_world_from_sdf("world.sdf", load_plugins=False) that is False by default, so that all the worlds found online can be compatible with it. This way, no hidden plugin loading would happen under the hood, while it allows users to enable their support if they explicitly intended.

I'm not sure I follow here. What would be the danger of having a plugin loaded that isn't explicitly loaded by the script? In either case, this does seem like a useful feature to be able to turn this on/off.

On top of the benefits you mentioned, the levels feature is also part of the sdf format, which would enable efficient scaling of the simulator to large worlds. This could be particularly relevant to the navigation community. sdf being language agnostic could also open up a straightforward path to map a real-world environment with the tool of your choice and then easily bring it into an RL/gym environment.

diegoferigo commented 3 years ago

Do you think we should update the documentation with a comment or note in the form of Note: Can only be called on an uninitialized simulator.? Maybe this can be combined with escalating the error message in the logs.

Yes, beyond fixing the verbosity of the error message, also the documentation could be improved.

I don't know how the current python bindings look like, but an additional step could be to raise an appropriate exception (perhaps RuntimeError("insert_world_from_sdf can not be called after the simulator has been initialized.")), since - in my mind - a user's expectation would be for commands to succeed and, consequentially, they may wish to handle any failure explicitly, i.e., change the behavior to fail noisily instead of silently.

We discussed this long time ago, and we decided to keep a 1:1 APIs compatibility with C++. ScenarI/O was born as C++ library, and we are used with the return-bool pattern among our projects. Then, the Python bindings inherited that, even though we are aware that using exceptions is more idiomatic to Python. However, mapping the C++ return-bool to Python exception is not directly doable with SWIG without duplicating all the methods (big work to implement it, huge maintenance effort, complete breaking of APIs). We recently explored in other projects the usage of pybind11, and discovered that they can be also used from Matlab that is the only other language that we could interests us, but we have no short-term plan to switch.

I moved gazebo.insert_world_from_sdf before gazebo.initialize. The minimal code to load the file looks like this [...]

The assert ... succeeds, but the world is not loaded. There is one suspicious message in the logs: [Msg] Loading SDF string. File path not available., which makes me think that the file may not be found? Interestingly, this doesn't seem to cause the call to fail; I will see if I can find the relevant section in the source code.

I suspect that you have to pass the absolute location of the file. The relative ./ is not directly unrolled. The current way to use relative paths is, as done in regular Gazebo, using the environment variable IGN_GAZEBO_RESOURCE_PATH.

I could also think to add an option gazebo.insert_world_from_sdf("world.sdf", load_plugins=False) that is False by default, so that all the worlds found online can be compatible with it. This way, no hidden plugin loading would happen under the hood, while it allows users to enable their support if they explicitly intended.

I'm not sure I follow here. What would be the danger of having a plugin loaded that isn't explicitly loaded by the script? In either case, this does seem like a useful feature to be able to turn this on/off.

I see your point. The main problem is that we aim to provide APIs that substitute many features that are typically implemented with plugins. One example is the JointPositionController plugin that the sdf you originally posted includes. With regular Ignition Gazebo, the only way users have to interact with the simulator is through Ignition Topics, and the plugins act as message relay. One of the key points of gym-ignition is that it allows communicating with the simulator without network transport. This is the only way to ensure reproducibility in simulation.

On top of the benefits you mentioned, the levels feature is also part of the sdf format, which would enable efficient scaling of the simulator to large worlds. This could be particularly relevant to the navigation community. sdf being language agnostic could also open up a straightforward path to map a real-world environment with the tool of your choice and then easily bring it into an RL/gym environment.

Levels are among those features that make Ignition Gazebo stand out from the crowd, they're very cool. It should not require much work to support them. I think they're kind of transparent from our point of view, maybe they already work, I haven't had the chance to try them. Something that could be challenging to handle is how to treat models that are not part of the level from our APIs. I guess that we can return a scenario::gazebo::Model to the user, but if it's part of another level its physics is not enabled.

FirefoxMetzger commented 3 years ago

Yes, beyond fixing the verbosity of the error message, also the documentation could be improved.

I'll check if I can submit a PR to update the documentation after I fixed the loading on my side.

I suspect that you have to pass the absolute location of the file.

I thought this could have been an issue, too. Unfortunately, using an absolute path doesn't change anything. Similarly, using the environment variable export IGN_GAZEBO_RESOURCE_PATH=$PWD made no difference on either a relative or absolute path.

~Do you know where the log message [Msg] Loading SDF string. File path not available. originates from? I've been digging through gym-ignition and osrf/sdformat for the past hour but haven't found its origin yet.~ Found the log message in ign-gazebo:

https://github.com/ignitionrobotics/ign-gazebo/blob/e9e9f196189d28514ab361bf796f02288ca35b81/src/Server.cc#L102-112

I wonder if an insert_world_from_string function could be of use. It's easy enough to read arbitrary files in python and sdformat appears to already have the functionality to load/parse from strings (https://github.com/osrf/sdformat/blob/master/include/sdf/Root.hh#L75-L80).

One of the key points of gym-ignition is that it allows communicating with the simulator without network transport. This is the only way to ensure reproducibility in simulation.

This is an interesting note; is there some documentation/information on this that I can read up on? Is the simulation loop of Ignition (or the physics engine) itself non-deterministic?

In my mind, as long as you control the simulator's main loop, you control the simulation. An approach like

# pseudo-code
while sim_not_done:
    # step the simulator
    simulator.step()

    # handle the generated messages/events indesired order
    camera_observation = camera_topic.recv()
    imu_observation = imu_topic.recv()
    # ...

   # compute and apply controls
   # ...
   control_topic.send(control_msg)

should be quite deterministic, at least in the current version of Ignition. Messages are sent over the local loopback device and via TCP, which should guarantee order and reception of messages. In addition, all callbacks are handled sequentially in the same thread (hello, GIL) and control messages (responses) are queued and not handled by the simulator until the next spin of the main loop. I'm wondering if I am missing something in my thinking.

FirefoxMetzger commented 3 years ago

Update: This does not appear to be a file reading problem. I found that gym-ignition can print the loaded SDF file

https://github.com/robotology/gym-ignition/blob/fe5a1afd2b1d418c9af49275ebb5d482166d71b4/cpp/scenario/gazebo/src/GazeboSimulator.cpp#L629-L633

which can be enabled via the environment variable SCENARIO_VERBOSE=1. This prints the sdf including the ground plane in the logs. It appears that something else is going sideways for me, because I am still only seeing an empty world.

diegoferigo commented 3 years ago

Yes, beyond fixing the verbosity of the error message, also the documentation could be improved.

I'll check if I can submit a PR to update the documentation after I fixed the loading on my side.

Great, thanks!

Update: This does not appear to be a file reading problem. I found that gym-ignition can print the loaded SDF file

https://github.com/robotology/gym-ignition/blob/master/cpp/scenario/gazebo/src/GazeboSimulator.cpp#L629-L633

which can be enabled via the environment variable SCENARIO_VERBOSE=1. This prints the sdf including the ground plane in the logs. It appears that something else is going sideways for me, because I am still only seeing an empty world.

I was about to suggest checking that :) Can you post the sdf file and the python script you're using? I'll try reproduce it locally.

Do you know where the log message [Msg] Loading SDF string. File path not available. originates from? I've been digging through gym-ignition and osrf/sdformat for the past hour but haven't found its origin yet.

I didn't dive in the code, I suspect that happens when sdformat tries to open the file (maybe using ign-common?). If you have a Debug version of Ignition (you can use colcon), I think that their logs also include the line that originated the message.

I wonder if an insert_world_from_string function could be of use. It's easy enough to read arbitrary files in python and sdformat appears to already have the functionality to load/parse from strings (https://github.com/osrf/sdformat/blob/master/include/sdf/Root.hh#L75-L80).

Yep, simple addition as well. As you noticed, sdformat already has all the APIs. The new method signature could be the following:

    bool insertModelFromString(const std::string& modelString,
                               const core::Pose& pose = core::Pose::Identity(),
                               const std::string& overrideModelName = {});

One of the key points of gym-ignition is that it allows communicating with the simulator without network transport. This is the only way to ensure reproducibility in simulation.

This is an interesting note; is there some documentation/information on this that I can read up on? Is the simulation loop of Ignition (or the physics engine) itself non-deterministic?

In my mind, as long as you control the simulator's main loop, you control the simulation. An approach like [...] should be quite deterministic, at least in the current version of Ignition. Messages are sent over the local loopback device and via TCP, which should guarantee order and reception of messages. In addition, all callbacks are handled sequentially in the same thread (hello, GIL) and control messages (responses) are queued and not handled by the simulator until the next spin of the main loop. I'm wondering if I am missing something in my thinking.

TCP guarantees that the message is sent and received. However, the process is asynchronous with the caller script. You cannot be sure that all messages have been received remotely when you call simulator.step() in the next loop. What's even more problematic is that in most cases the messages are streamed fast enough to make it work as expected. But you can have unwanted surprises if the load of your PC is so high that the scheduler is not able to catch up.

The workaround is substituting TCP messages with RPC messages, and Ignition Transport supports them. I'm not expert, but they are a different type of message, you cannot "switch" backend from TCP to RPC. The only real way to ensure determinism under any load is interfacing the simulator from shared memory as we do in gym-ignition.

FirefoxMetzger commented 3 years ago

I was about to suggest checking that :) Can you post the sdf file and the python script you're using? I'll try reproduce it locally.

Thanks!

simulator.py ```python from scenario import gazebo as scenario_gazebo import time scenario_gazebo.set_verbosity(scenario_gazebo.Verbosity_debug) gazebo = scenario_gazebo.GazeboSimulator( step_size=0.001, rtf=1.0, steps_per_run=1 ) assert gazebo.insert_world_from_sdf("./generated_world.sdf", "generated_world") gazebo.initialize() gazebo.gui() gazebo.run(paused=True) time.sleep(3) gazebo.close() ```
generated_world.sdf ```xml 0.001 1 -1 1 0 0 10 0 -0 0 1 1 1 1 0.5 0.5 0.5 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 0 0 0 0 0 -9.8 6e-06 2.3e-05 -4.2e-05 0.4 0.4 0.4 1 0.7 0.7 0.7 1 1 file:///home/sebastian/.local/lib/python3.8/site-packages/gym_ignition_models/ground_plane ground_plane 0 0 0 0 -0 0 ```

and after calling it via SCENARIO_VERBOSE=1 python3 simulator.py the console output is

console output ``` $ SCENARIO_VERBOSE=1 python3 simulator.py [Dbg] [GazeboSimulator.cpp:626] Physics profile: max_step_size=0.001 real_time_factor=1 real_time_update_rate=-1 [Dbg] [GazeboSimulator.cpp:630] Loading the following SDF file in the gazebo server: 0.001 1 -1 1 0 0 10 0 -0 0 1 1 1 1 0.5 0.5 0.5 1 1000 0.9 0.01 0.001 -0.5 0.1 -0.9 0 0 0 0 0 -9.8 6e-06 2.3e-05 -4.2e-05 0.4 0.4 0.4 1 0.7 0.7 0.7 1 1 1 0 0 1 1 1 100 1 0 0 1 100 100 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0.8 0.8 0.8 1 0 0 0 0 -0 0 [Msg] Loading SDF string. File path not available. [Dbg] [EntityComponentManager.cc:672] Using components of type [2251689575469537287] / [ign_gazebo_components.World]. [Dbg] [EntityComponentManager.cc:672] Using components of type [13994732549916512682] / [ign_gazebo_components.Name]. [Dbg] [EntityComponentManager.cc:672] Using components of type [12592746352568925681] / [ign_gazebo_components.Gravity]. [Dbg] [EntityComponentManager.cc:672] Using components of type [2188341333082264598] / [ign_gazebo_components.Physics]. [Dbg] [EntityComponentManager.cc:672] Using components of type [13224937992534617849] / [ign_gazebo_components.MagneticField]. [Dbg] [EntityComponentManager.cc:672] Using components of type [3630648173860223239] / [ign_gazebo_components.PhysicsEnginePlugin]. [Dbg] [EntityComponentManager.cc:672] Using components of type [17459188283658606303] / [ign_gazebo_components.RenderEngineServerPlugin]. [Dbg] [EntityComponentManager.cc:672] Using components of type [8705992680619689917] / [ign_gazebo_components.RenderEngineGuiPlugin]. [Dbg] [EntityComponentManager.cc:672] Using components of type [8753193699724811771] / [ign_gazebo_components.Wind]. [Dbg] [EntityComponentManager.cc:672] Using components of type [12173050716021724529] / [ign_gazebo_components.WorldLinearVelocity]. [Dbg] [EntityComponentManager.cc:672] Using components of type [15943768124495574352] / [ign_gazebo_components.WorldLinearVelocitySeed]. [Dbg] [EntityComponentManager.cc:672] Using components of type [3297509811873971798] / [ign_gazebo_components.ParentEntity]. [Dbg] [EntityComponentManager.cc:672] Using components of type [17100615127981600159] / [ign_gazebo_components.Scene]. [Dbg] [EntityComponentManager.cc:672] Using components of type [17605309075052480649] / [ign_gazebo_components.Atmosphere]. [Dbg] [EntityComponentManager.cc:672] Using components of type [8064491505919932473] / [ign_gazebo_components.Level]. [Dbg] [EntityComponentManager.cc:672] Using components of type [2668898242563798256] / [ign_gazebo_components.DefaultLevel]. [Dbg] [EntityComponentManager.cc:672] Using components of type [11371360182141354106] / [ign_gazebo_components.LevelEntityNames]. [Dbg] [EntityComponentManager.cc:672] Using components of type [4981278897826323946] / [ign_gazebo_components.WorldSdf]. [Dbg] [EntityComponentManager.cc:672] Using components of type [6687176221774458630] / [ign_gazebo_components.Model]. [Dbg] [EntityComponentManager.cc:672] Using components of type [6612894081701502240] / [ign_gazebo_components.Pose]. [Dbg] [EntityComponentManager.cc:672] Using components of type [8546580419506082455] / [ign_gazebo_components.Static]. [Dbg] [EntityComponentManager.cc:672] Using components of type [9712747055438129860] / [ign_gazebo_components.WindMode]. [Dbg] [EntityComponentManager.cc:672] Using components of type [5661073481138181711] / [ign_gazebo_components.SelfCollide]. [Dbg] [EntityComponentManager.cc:672] Using components of type [11683062252779233161] / [ign_gazebo_components.SourceFilePath]. [Dbg] [EntityComponentManager.cc:672] Using components of type [5081358965268446661] / [ign_gazebo_components.Link]. [Dbg] [EntityComponentManager.cc:672] Using components of type [8112400427272910195] / [ign_gazebo_components.Inertial]. [Dbg] [EntityComponentManager.cc:672] Using components of type [16454635107327670381] / [ign_gazebo_components.Visual]. [Dbg] [EntityComponentManager.cc:672] Using components of type [13011964647677164955] / [ign_gazebo_components.CastShadows]. [Dbg] [EntityComponentManager.cc:672] Using components of type [13440282432131634483] / [ign_gazebo_components.Transparency]. [Dbg] [EntityComponentManager.cc:672] Using components of type [5453622280849253520] / [ign_gazebo_components.VisibilityFlags]. [Dbg] [EntityComponentManager.cc:672] Using components of type [17121648710877364109] / [ign_gazebo_components.Geometry]. [Dbg] [EntityComponentManager.cc:672] Using components of type [9853217982010720764] / [ign_gazebo_components.Material]. [Dbg] [EntityComponentManager.cc:672] Using components of type [17938588655714334139] / [ign_gazebo_components.Collision]. [Dbg] [EntityComponentManager.cc:672] Using components of type [9225962031573086509] / [ign_gazebo_components.CollisionElement]. [Dbg] [EntityComponentManager.cc:672] Using components of type [10522242218202596205] / [ign_gazebo_components.CanonicalLink]. [Dbg] [EntityComponentManager.cc:672] Using components of type [11536476718181283925] / [ign_gazebo_components.ModelSdf]. [Dbg] [EntityComponentManager.cc:672] Using components of type [3866641186784191835] / [ign_gazebo_components.Light]. [Msg] Loaded level [3] [Dbg] [ECMProvider.cpp:83] World 'generated_world' successfully processed by ECMProvider [Dbg] [SimulationRunner.cc:830] Loaded system [scenario::plugins::gazebo::ECMProvider] for entity [1] [Msg] Serving world controls on [/world/generated_world/control] and [/world/generated_world/playback/control] [Msg] Serving GUI information on [/world/generated_world/gui/info] [Msg] World [generated_world] initialized with [default_physics] physics profile. [Msg] Serving world SDF generation service on [/world/generated_world/generate_world_sdf] [Msg] Serving world names on [/gazebo/worlds] [Msg] Resource path add service on [/gazebo/resource_paths/add]. [Msg] Resource path get service on [/gazebo/resource_paths/get]. [Msg] Resource paths published on [/gazebo/resource_paths]. [Dbg] [GazeboSimulator.cpp:651] Starting the gazebo server [Msg] Found no publishers on /stats, adding root stats topic [Msg] Found no publishers on /clock, adding root clock topic [Dbg] [SimulationRunner.cc:470] Creating PostUpdate worker threads: 1 [Dbg] [GazeboSimulator.cpp:663] Creating and caching World 'generated_world' [Dbg] [EntityComponentManager.cc:672] Using components of type [13081221419881009263] / [ign_gazebo_components.Timestamp]. [Dbg] [EntityComponentManager.cc:672] Using components of type [14753161793664412980] / [ign_gazebo_components.SimulatedTime]. Setting callback for signal SIGINT Setting callback for signal SIGTERM Setting callback for signal SIGABRT [Dbg] [GazeboSimulator.cpp:271] Starting the SceneBroadcaster plugin [Dbg] [SimulationRunner.cc:830] Loaded system [ignition::gazebo::systems::SceneBroadcaster] for entity [1] [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Msg] Ignition Gazebo GUI v4.6.0 [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-sebastian' [Dbg] [Application.cc:87] Initializing application. [GUI] [Dbg] [Application.cc:407] Create main window [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [GUI] [Dbg] [PathManager.cc:66] Requesting resource paths through [/gazebo/resource_paths/get] [GUI] [Wrn] [Application.cc:649] [QT] file::/Gazebo/GazeboDrawer.qml:178:3: QML Dialog: Binding loop detected for property "implicitHeight" [GUI] [Wrn] [Application.cc:649] [QT] file::/Gazebo/GazeboDrawer.qml:178:3: QML Dialog: Binding loop detected for property "implicitHeight" [GUI] [Dbg] [Gui.cc:151] GUI requesting list of world names. The server may be busy downloading resources. Please be patient. [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [GUI] [Dbg] [PathManager.cc:55] Received resource paths. [GUI] [Dbg] [Gui.cc:212] Requesting GUI from [/world/generated_world/gui/info]... [GUI] [Dbg] [GuiRunner.cc:67] Requesting initial state from [/world/generated_world/state]... [GUI] [Msg] Loading config [/home/sebastian/.ignition/gazebo/gui.config] [GUI] [Dbg] [Application.cc:305] Loading plugin [GzScene3D] [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [GUI] [Msg] Video recorder stats topic advertised on [/gui/record_video/stats] [GUI] [Msg] Transform mode service on [/gui/transform_mode] [GUI] [Msg] Record video service on [/gui/record_video] [GUI] [Msg] Move to service on [/gui/move_to] [GUI] [Msg] Follow service on [/gui/follow] [GUI] [Msg] View angle service on [/gui/view_angle] [GUI] [Msg] Move to pose service on [/gui/move_to/pose] [GUI] [Msg] Camera pose topic advertised on [/gui/camera/pose] [GUI] [Msg] View collisions service on [/gui/view/collisions] [GUI] [Msg] Added plugin [3D View] to main window [GUI] [Msg] Loaded plugin [GzScene3D] from path [/usr/lib/x86_64-linux-gnu/ign-gazebo-4/plugins/gui/libGzScene3D.so] [GUI] [Dbg] [Application.cc:305] Loading plugin [WorldControl] [GUI] [Msg] Using world control service [/world/generated_world/control] [GUI] [Msg] Listening to stats on [/world/generated_world/stats] [GUI] [Msg] Added plugin [World control] to main window [GUI] [Msg] Loaded plugin [WorldControl] from path [/usr/lib/x86_64-linux-gnu/ign-gui-4/plugins/libWorldControl.so] [GUI] [Dbg] [Application.cc:305] Loading plugin [WorldStats] [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Msg] Listening to stats on [/world/generated_world/stats] [Dbg] [GazeboSimulator.cpp:297] Waiting GUI to show up... [GUI] [Msg] Added plugin [World stats] to main window [GUI] [Msg] Loaded plugin [WorldStats] from path [/usr/lib/x86_64-linux-gnu/ign-gui-4/plugins/libWorldStats.so] [GUI] [Dbg] [Application.cc:305] Loading plugin [TransformControl] [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Msg] Added plugin [Transform control] to main window [GUI] [Msg] Loaded plugin [TransformControl] from path [/usr/lib/x86_64-linux-gnu/ign-gazebo-4/plugins/gui/libTransformControl.so] [GUI] [Dbg] [Application.cc:305] Loading plugin [Shapes] [GUI] [Msg] Added plugin [Shapes] to main window [GUI] [Msg] Loaded plugin [Shapes] from path [/usr/lib/x86_64-linux-gnu/ign-gazebo-4/plugins/gui/libShapes.so] [GUI] [Dbg] [Application.cc:305] Loading plugin [ComponentInspector] [Dbg] [GazeboSimulator.cpp:310] GUI up and running [Dbg] [SimulationRunner.cc:470] Creating PostUpdate worker threads: 2 [Dbg] [SimulationRunner.cc:483] Creating postupdate worker thread (0) [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Msg] Added plugin [Component inspector] to main window [GUI] [Msg] Loaded plugin [ComponentInspector] from path [/usr/lib/x86_64-linux-gnu/ign-gazebo-4/plugins/gui/libComponentInspector.so] [GUI] [Dbg] [Application.cc:305] Loading plugin [EntityTree] [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Msg] Added plugin [Entity tree] to main window [GUI] [Msg] Loaded plugin [EntityTree] from path [/usr/lib/x86_64-linux-gnu/ign-gazebo-4/plugins/gui/libEntityTree.so] [GUI] [Dbg] [Application.cc:266] Loading window config [GUI] [Dbg] [Application.cc:421] Applying config [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Wrn] [Application.cc:649] [QT] file::/Gazebo/GazeboDrawer.qml:178:3: QML Dialog: Binding loop detected for property "implicitHeight" [GUI] [Msg] Loading plugin [ignition-rendering-ogre2] [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [RenderUtil.cc:1807] Create scene [scene] [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [Scene3D.cc:2745] RenderEngineGuiPlugin component not found, render engine won't be set from the ECM [GUI] [Dbg] [SignalHandler.cc:141] Received signal[15]. [GUI] [Dbg] [Gui.cc:309] Shutting down ign-gazebo-gui [GUI] [Dbg] [Application.cc:130] Terminating application. [GUI] [Msg] Loading plugin [ignition-rendering-ogre2] [GUI] [Dbg] [Scene3D.cc:1762] Destroy scene [scene] [Dbg] [SimulationRunner.cc:499] Exiting postupdate worker thread (0) [Dbg] [ECMProvider.cpp:53] Destroying the ECMProvider ```

Yep, simple addition as well. As you noticed, sdformat already has all the APIs.

Actually, ign-gazebo internally loads from strings or file pointers respectively.

https://github.com/ignitionrobotics/ign-gazebo/blob/e9e9f196189d28514ab361bf796f02288ca35b81/src/Server.cc#L100-176

gym-ignition seems to currently convert everything to string by itself and then strictly use ign-gazebos string loader. Potentially, this could be simplified by delegating all .sdf loading and parsing to ign-gazebo? This could reduce the burden on maintenance for this repo; their file parser also seems to already take care of loading fuel models:

https://github.com/ignitionrobotics/ign-gazebo/blob/e9e9f196189d28514ab361bf796f02288ca35b81/src/Server.cc#L118

This could make the PR to add fuel support redundant (maybe?).

You cannot be sure that all messages have been received remotely when you call simulator.step() in the next loop

Actually, recv calls of zmq are blocking (with infinite timeout) unless explicitly passing ZMQ_NOBLOCK source, so the loop would block for each sensor callback until the message is received. I think this should ensure determinism on the sensor side? It is also the path I will try to follow to get sensor support (camera in my case). I tinkered with it, and it appeared to be working during my small pilots, but I will know more once I have it integrated with gym-ignition :)

For sending, you are right, it is indeed safer to assume that sending control commands to existing plugins is non-deterministic, as gym-ignition can't know how they consume messages. Modifying the simulation in-memory after each step is one viable solution. I wonder though if a dedicated gym-ignition control plugin that blocks until a message was received would scale more easily; then again, it may no longer be a viable option given the current codebase of gym-ignition.

diegoferigo commented 3 years ago

Ok I think I know what's happening. Try to add print(gazebo.world_names()) and you can see that the world is correctly loaded. The problem is the communication between the server (running in the python process) and the GUI (running in a separated process).

In order to draw the scene, the server has to send data to the GUI. This happens only when the ECM (that is the database containing all the simulated resources like world, models, ...) gets invalidated. This happens for instance if a new model is added, or a link pose is changed due to the physics running. In your case, the world is built from the SDF, and when the GUI opens, this first message to draw the world is not sent.

If you edit your code as follows, you'll see that the GUI will show the scene:

simulator.py ```py from scenario import gazebo as scenario_gazebo import time scenario_gazebo.set_verbosity(scenario_gazebo.Verbosity_debug) gazebo = scenario_gazebo.GazeboSimulator( step_size=0.001, rtf=1.0, steps_per_run=1 ) assert gazebo.insert_world_from_sdf("./generated_world.sdf", "generated_world") gazebo.initialize() print(gazebo.world_names()) world = gazebo.get_world() # Add any model, e.g. the panda robot import gym_ignition_models world.insert_model(gym_ignition_models.get_model_file(robot_name="panda")) # You'll see both the ground_plane from the sdf and the panda from APIs gazebo.gui(True) gazebo.run(paused=True) time.sleep(3) gazebo.close() ```

There's no easy workaround to it. I remember last year I spent quite some time to enhance the interaction between server and gui, without much luck. I ended up implementing GazeboSimulator::gui() that is quite hacky. There's a proposal upstream to allow running the GUI from the same process of the server, but nobody has yet taken it over.

gym-ignition seems to currently convert everything to string by itself and then strictly use ign-gazebos string loader. Potentially, this could be simplified by delegating all .sdf loading and parsing to ign-gazebo? This could reduce the burden on maintenance for this repo; their file parser also seems to already take care of loading fuel models:

I fear that manipulating the SDF is still necessary. We have to pass through the SDF to set the simulator parameters.

You cannot be sure that all messages have been received remotely when you call simulator.step() in the next loop

Actually, recv calls of zmq are blocking (with infinite timeout) unless explicitly passing ZMQ_NOBLOCK source, so the loop would block for each sensor callback until the message is received. I think this should ensure determinism on the sensor side? It is also the path I will try to follow to get sensor support (camera in my case). I tinkered with it, and it appeared to be working during my small pilots, but I will know more once I have it integrated with gym-ignition :)

For sending, you are right, it is indeed safer to assume that sending control commands to existing plugins is non-deterministic, as gym-ignition can't know how they consume messages. Modifying the simulation in-memory after each step is one viable solution. I wonder though if a dedicated gym-ignition control plugin that blocks until a message was received would scale more easily; then again, it may no longer be a viable option given the current codebase of gym-ignition.

Interesting. I have zero experience with ROS2 (what I assume you're using, guessing from zmq). If those messages are blocking and not asynchronous, yes, you get determinism. ~I was referring to the messages of ign-transport, that have a different implementation.~ FYI @traversaro.

FirefoxMetzger commented 3 years ago

Yes! That indeed fixes the problem 🚀

It's a bit odd that the GUI can't query the full state when it first starts; it sounds suspiciously like a classic late subscriber problem forcing the GUI to sit there waiting for the first message instead of being able to request the current state when it first starts (service call).

Camera sensors are hopefully unaffected and - since I will run the simulation GUI-less once it works - this is a good enough solution for me for now. I will give this more thought once I get around to the fuel-support PR because it would be a tad annoying to template the world as sdf and not be able to see it because no API objects are added.

I fear that manipulating the SDF is still necessary. We have to pass through the SDF to set the simulator parameters.

That's unfortunate. I have no specific idea yet, but if it is "only" physics - and more specifically step size - that prevents us from delegating sdf loading it may be possible to patch this in a different way. I'll have a go once I get around to the fuel-support PR.

I actually have no ROS dependency (yet). ign-transport uses zmq under the hood [source] to send around protobuf serialized objects. There is some custom code on top of zmq, but from what I've seen so far what the code mainly adds the familiar pub/sub + service interface we know from ROS as well as a nameserver-like object for topic discovery. The actual sending of messages appears to be pure zmq, which can be subscribed to from non-ignition libraries like pyzmq. Unfortunately, documentation is sparse, so my knowledge on the Ignition side is rather limited.

A problem that I will likely have to solve down the line with this is differing publication rates of sensors. If the sensor doesn't send a message at every simulation step, the loop will get stuck waiting for sensor data that was never sent (it's a blocking call). It also messes with the gym interface that demands an observation to be returned at every step.

If I do manage to get it working within gym-ignition, I will leave a comment in #287 and #199.

traversaro commented 3 years ago

If those messages are blocking and not asynchronous, yes, you get determinism. I was referring to the messages of ign-transport, that have a different implementation. FYI @traversaro.

Thanks @diegoferigo and @FirefoxMetzger for the interesting discussion. As @diegoferigo already mentioned, it is perfectly possible to have network communication and deterministic simulation: a whole part of co-simulation field (https://en.wikipedia.org/wiki/Co-simulation, Sectio C.1.6 of https://arxiv.org/pdf/1702.00686.pdf) is exactly about that, and standard such as the Distributed Co-Simulation Protocol has been developed for that. Unfortunately most Pub/Sub middlewares used in robotics have been designed with other use cases in mind, and so achieving determinism with them is not trivial, but @FirefoxMetzger feel free to keep us posted on your effort!

FirefoxMetzger commented 3 years ago

For future reference: One (manual) way to get the fuel models into gym-ignition (sort of a workaround) is to open the world via ign gazebo world.sdf, and save it with the "Expand Include Tags" option checked:

image

This will merge all the (downloaded) include tags into one file and save that. Downloading meshes and textures from fuel seems to work, and gazebo.insert_world_from_sdf("./world_expanded.sdf") does load the world as desired.

This workaround doesn't cover all use-cases, but I thought I'll add it here in case it may be useful for others.

diegoferigo commented 3 years ago

Yes! That indeed fixes the problem rocket

It's a bit odd that the GUI can't query the full state when it first starts; it sounds suspiciously like a classic late subscriber problem forcing the GUI to sit there waiting for the first message instead of being able to request the current state when it first starts (service call).

We use the simulator in a way that differs from upstream. Even when the simulation is paused, when you open ign gazebo the server keeps sending data to the GUI. This does not happen in our case, and this is the reason why we have to manually run a paused step after opening the GUI (and the time here matters, not too soon, not too late, otherwise the message is not received by the GUI process). I never found enough time to go the root of it, the current situation is a workaround that works for us. Though, we mainly perform headless simulations.

I fear that manipulating the SDF is still necessary. We have to pass through the SDF to set the simulator parameters.

That's unfortunate. I have no specific idea yet, but if it is "only" physics - and more specifically step size - that prevents us from delegating sdf loading it may be possible to patch this in a different way. I'll have a go once I get around to the fuel-support PR.

One of the clean ways would be to add the missing value in the ServerConfig object passed to the constructor of Server. In this way, the SimulationRunner could read it from the config instead then relying only of the SDF. However, this is a very central and delicate part of the simulator, and any change must be tested extensively to land upstream.

I actually have no ROS dependency (yet). ign-transport uses zmq under the hood [source] to send around protobuf serialized objects. There is some custom code on top of zmq, but from what I've seen so far what the code mainly adds the familiar pub/sub + service interface we know from ROS as well as a nameserver-like object for topic discovery. The actual sending of messages appears to be pure zmq, which can be subscribed to from non-ignition libraries like pyzmq. Unfortunately, documentation is sparse, so my knowledge on the Ignition side is rather limited.

A problem that I will likely have to solve down the line with this is differing publication rates of sensors. If the sensor doesn't send a message at every simulation step, the loop will get stuck waiting for sensor data that was never sent (it's a blocking call). It also messes with the gym interface that demands an observation to be returned at every step.

If I do manage to get it working within gym-ignition, I will leave a comment in #287 and #199.

Ok now I understood my confusion and this is clear, especially after reading https://github.com/robotology/gym-ignition/issues/287#issuecomment-796654867. The workaround is cool and offer a nice flexibility. As reported in https://github.com/robotology/gym-ignition/issues/296#issuecomment-795405743, only receiving data could maintain determinism, but it's a good alternative that could be used in absence of sensor API in ScenarIO. As you noticed there, if the data rate is lower than the rate of which the simulation is stepped in the code, things could get stuck if the rate differ. A possible solution would be performing all the receiving operation in a slower thread with proper synchronization mechanism (a simple lock on the received image object would suffice). Being and I/O operation, I think it does not affect performance due to the GIL.

FirefoxMetzger commented 3 years ago

As you noticed there, if the data rate is lower than the rate of which the simulation is stepped in the code, things could get stuck if the rate differ.

It will block; however if one does the math correctly this can be accounted for. I'm doing this in the simulation I'm building now for camera images, but it should work with any sensor. If determinism is the end-goal I think getting stuck is a desirable feature, because it fails noisily. Potentially, this could be improved via a long (>1 s) timeout + an exception to produce a stack trace pointing to the sensor that was not in sync.

This is the the relevant snippet

# get synced camera images (published at 30Hz)
if (int(round(sim_time/gazebo.step_size()))) % steps_per_frame == 0:
    zmq_msg = camera_topic.recv()  # blocking call
    image_msg = Image()
    image_msg.ParseFromString(zmq_msg[2])
    im = np.frombuffer(image_msg.data, dtype=np.uint8)
    im = im.reshape((image_msg.height, image_msg.width, 3))
    img_time = image_msg.header.stamp.sec + image_msg.header.stamp.nsec*1e-9

    # ensure image message and simulator are in sync
    assert sim_time == img_time

Full source: https://github.com/FirefoxMetzger/panda-ignition-sim/blob/master/simulator.py

diegoferigo commented 3 years ago

Closed via #309