Closed osrf-migration closed 5 years ago
osrf-migration
commented
5 years ago Original comment by Neil Johnson (Bitbucket: realdealneil1980).
With a single agent, we were getting about equal performance on gazebo 9 versus ignition, but once we add more agents, the performance degrades more quickly on ignition.
osrf-migration
commented
5 years ago Original comment by Jon Fink (Bitbucket: jonfink-arl).
Out of curiosity, what agent type did you use when you tested with a single agent? I am using X1_SENSOR_CONFIG_4 (camera + 3D LIDAR)
osrf-migration
commented
5 years ago Original comment by Jon Fink (Bitbucket: jonfink-arl).
It seems that downgrading to X1_SENSOR_CONFIG_1, I get closer to expected performance. Though not entirely, it seems as though I’m still lagging behind real time with a single robot (and not not on a particularly taxed cpu, see screenshot).
osrf-migration
commented
5 years ago Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
osrf-migration
commented
5 years ago Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
Switching this to Michael Carroll (Michael Carroll) , who has been working on performance.
osrf-migration
commented
5 years ago Original comment by Jon Fink (Bitbucket: jonfink-arl).
Any thoughts on this Michael Carroll (Michael Carroll) ? Happy to try to dig in if you have thoughts on what the root cause of this might be..
osrf-migration
commented
5 years ago Original comment by Michael Carroll (Bitbucket: Michael Carroll).
I haven’t had a chance to look at the config_4, but it may be that the 3d lidar sensor is causing issues?
Out of curiosity, what is the hardware configuration that you are running this one?
osrf-migration
commented
5 years ago Original comment by Michael Carroll (Bitbucket: Michael Carroll).
I’m mainly curious as to why that CPU wouldn’t be pegged out, that’s generally the behavior I see when performance degrades.
osrf-migration
commented
5 years ago Original comment by Jon Fink (Bitbucket: jonfink-arl).
Agree re: CPU not being fully utilized is strange (which is partly why I wonder if this is something artificial - i.e., an extra sleep or something…)
I had originally thought this might be something to with 3D LIDAR not being properly GPU-accelerated but as I noted above, I’m only seeing about 80% real time with a single X1_CONFIG_1 (in the screenshot above).
This is on an Intel i9-8950 (2.9 GHz), 32GB RAM, GTX 1050. I wouldn’t even bring this issue up at the moment if it was not a significant drop from my performance running a similar configuration with Gazebo 9 at 100% real-time.
osrf-migration
commented
5 years ago Original comment by Hector Escobar (Bitbucket: hector_escobar).
If it is of any help, If I ran using catkin then my RTF is < 4% while if I use the docker the RTF is around 20%. When I was running Gazebo 9 I was able to run 3 vehicles at about 50% and 1 at 100%
osrf-migration
commented
5 years ago Original comment by Michael Carroll (Bitbucket: Michael Carroll).
The really low RTF with catkin sounds like you may have built a debug version, make sure that you are building Release or RelWithDebInfo.
osrf-migration
commented
5 years ago Original comment by Michael Carroll (Bitbucket: Michael Carroll).
Jon Fink (jonfink-arl) After a little bit of digging, I think part of what you are seeing is the result of making rendering and physics operate in lockstep (which is different from how Gazebo 9 works). Now any frames that have rendering have to complete before physics can continue simulating.
I think I have an idea on how to thread the rendering operation out, so that we can continue to get the fidelity of the lockstepping, but incur a slightly lower RTF cost. I’ll keep you updated.
osrf-migration
commented
5 years ago Original comment by Jon Fink (Bitbucket: jonfink-arl).
Sounds good Michael Carroll (Michael Carroll) - does this mean that rendering happens after every physics timestep?
osrf-migration
commented
5 years ago Original comment by Arthur Schang (Bitbucket: Arthur Schang).
As a side note, if there any way to increase Ignition beyond a RTF of 1? I know that adjusting the step size in Gazebo9 would allow you to manipulate your RTF but I did not have any luck adjusting the “steps” parameter in the lower left corner of Ignition for much difference in RTF.
osrf-migration
commented
5 years ago Original comment by Sarah Kitchen (Bitbucket: snkitche).
I am currently trying to run the ExampleSetup with teleop.launch and
ign launch -v 4 tunnel_circuit_practice.ign worldName:=tunnel_circuit_practice_03 robotName1:=X1 robotConfig1:=X1_SENSOR_CONFIG_1My RTF is <5%, when it successfully launches at all. I am regularly getting the following on startup:
No processes to monitor
shutting down processing monitor...
... shutting down processing monitor complete
[Dbg] [Manager.cc:295] Received signal[17]
[Dbg] [Manager.cc:312] Death of process[23612] with name[x1_description].And then can not shut down with Ctrl+C/run into the hanging process problem. I have to close the terminal to shut everything down.
This is a fresh (yesterday evening) install of tunnel_circuit using the catkin workspace install instructions in the tutorial.
virtual_stix.ign with the same robot configuration has RTF in the 90%'s and does not have the start-up or shut-down problems.
osrf-migration
commented
5 years ago Original comment by Derek Knowles (Bitbucket: dknowles-ssci).
With the recent ignition-gazebo2 update to 2.3.0-1~bionic RTF while running ign launch -v 4 tunnel_circuit_practice.ign worldName:=tunnel_circuit_practice_03 has increased from <4% to <8% with a single vehicle. I can see the vehicle moving, but this is still prohibitively slow to test our algorithms. Are there further improvements coming?
osrf-migration
commented
5 years ago Original comment by Alfredo Bencomo (Bitbucket: bencomo).
This has been addressed and performance has improved.
If you are using a docker container installation, then download the new Docker image from here:
https://hub.docker.com/r/nkoenig/subt-virtual-testbedIf you are using a catkin workspace installation, then run the commands below to update it:
sudo apt update && sudo apt upgrade
sudo apt-get install ros-melodic-rotors-control
sudo reboot
cd ~/subt_ws/src/tunnel_circuit
hg pull && hg update tunnel_circuit
. /opt/ros/melodic/setup.bash
rm -rfv ~/.ignition/fuel/fuel.ignitionrobotics.org/openrobotics/models/*
cd ~/subt_ws/
catkin_make install
. ~/subt_ws/install/setup.bash
ign launch -v 4 tunnel_circuit_practice.ign robotName1:=X1 robotConfig1:=X1_SENSOR_CONFIG_1Original comment by Zbyněk Winkler (Bitbucket: Zbyněk Winkler (robotika)).
Alfredo Bencomo (bencomo) May we know what is the fix for the performance regression wrt gazebo9?
osrf-migration
commented
5 years ago Original comment by Alfredo Bencomo (Bitbucket: bencomo).
Zbyněk, I’m not sure I follow. Are you having performance issues with gazebo9? The fix for the performance I referred in previous post was only for SubT Ignition.
osrf-migration
commented
5 years ago Original comment by Zbyněk Winkler (Bitbucket: Zbyněk Winkler (robotika)).
Jon Fink reported that the RTF on Ignition is much lower than previously on Gazebo9. You marked this issue as resolved. So I wanted to ask what was the problem. Why was Ignition so much slower? What is the fix?
osrf-migration
commented
5 years ago Original comment by Alfredo Bencomo (Bitbucket: bencomo).
Ignition performance improved with these two PRs:
https://bitbucket.org/ignitionrobotics/ign-gazebo/pull-requests/392
https://osrf-migration.github.io/subt-gh-pages/#!/osrf/subt/pull-requests/224/
Let's keep it open since we are still improving performance.
osrf-migration
commented
5 years ago Original comment by Sarah Kitchen (Bitbucket: snkitche).
It seems like there has been some performance regression in the last week. Practice worlds with one UGV (X1 with sensor config 4) are back down to ~20% RTF.
I wanted to comment on this because I’ve been opening a custom world I made by modifying your practice.ign file and using my own .sdf. I have levels turned off, as my world is very small. A single UGV (X1 with sensor config 4) runs at about 60% RTF. Three UGVs (two X1’s with sensor config 4 and one X2 with sensor config 6) run at about 40-50% RTF (iirc - I’m not being very precise about this).
I also think the maneuverability of the X1 is pretty unexpectedly bad. It really struggles to make turns and it rotates very slowly. This goes for both teleop and autonomous control.
osrf-migration
commented
5 years ago Original comment by Alfredo Bencomo (Bitbucket: bencomo).
Sarah,
Since we don’t have access to you custom world and assuming that you either have yesterday Docker image installed or a fresh updated catkin workspace (sudo apt-get update && sudo apt-get upgrade, cd ~/subt_ws/src/tunnel_circuit, hg pull && hg up), then can you run a test following the first four(4) steps in this tutorial? Let me know if you see the same performance and behavior shown here: https://vimeo.com/352096820
osrf-migration
commented
5 years ago Original comment by Sarah Kitchen (Bitbucket: snkitche).
Sorry - I did have a fresh install but the performance issue was my fault, which I’ve resolved. I’m getting approximately the same performance as in your video now. Changing from config 1 to config 4 drops the performance to ~65%, but that’s expected.
Your go-to-entrance behavior is split into 4 phases: rotate, drive straight, rotate, drive straight. Have you tested the X1 on a curved path to check whether there’s actually a turning radius issue? Your controller rotates faster than seems possible with teleop for example. I refreshed all my models yesterday.
osrf-migration
commented
5 years ago Original comment by Alfredo Bencomo (Bitbucket: bencomo).
Hi Sarah,
Thank you for validating and re-testing performance. I’m glad you are now getting the same results.
We’re now looking into the maneuverability behavior you are experiencing with the UGV. We’ll post back here with our findings.
osrf-migration
commented
5 years ago Original comment by Sophisticated Engineering (Bitbucket: sopheng).
We also recognized that the turning behavior (together with driving forward) is much different between Gazebo 9 and Ignition. We have to use different parameters to get similar moving results.
osrf-migration
commented
5 years ago osrf-migration
commented
5 years ago
Original report (archived issue) by Jon Fink (Bitbucket: jonfink-arl).
Running
I’ve noticed that the performance of simulation is much slower than I would have anticipated: < 50% real time factor. On the same computer, with Gazebo 9, I was routinely running two robots with similar sensor configurations at 1.0 real time factor.
Is this expected?