Control response for X3 different in cloud than locally - Githubissues

osrf / subt

This repostory contains software for the virtual track of the DARPA SubT Challenge. Within this repository you will find Gazebo simulation assets, ROS interfaces, support scripts and plugins, and documentation needed to compete in the SubT Virtual Challenge.

Other

309 stars 98 forks source link

Control response for X3 different in cloud than locally #208

Closed osrf-migration closed 4 years ago

osrf-migration commented 5 years ago

Original report (archived issue) by GoRobotGo (Bitbucket: GoRobotGo).

There appears to be a dependency of the control response for the x3 vehicle based on the real-time rate of the simulation. In particular, a linear z input of 16.88 results in a relatively fast lift-off of the vehicle when the simulation runs in near real-time. When running in the cloud at 1/8th of real time an x3 vehicle won’t even take off with a linear z input of 16.88. Similarly, when the cloud is running at 2x realtime it appears to be much more sensitive to the inputs.

There may be a different root cause, but in any case the same control input results in different behaviors in the cloud than locally.

To reproduce: publish a linear z input of 16.88 to the x3 vehicle on a local simulation run and then to a cloud run with a low real time rate (with multiple vehicles simulated)

‌

osrf-migration commented 5 years ago

Original comment by Alfredo Bencomo (Bitbucket: bencomo).

set assignee_account_id to "557058:90bc87b0-2886-4377-a17c-3a6629a697ff"
set assignee to "azeey (Bitbucket: azeey, GitHub: azeey)"

osrf-migration commented 5 years ago

Original comment by GoRobotGo (Bitbucket: GoRobotGo).

Two more data points - I ran a similar simulation overnight, but allowed a z input of up to ~19. The vehicle had a continuous z input of ~19, twitched around by 1mm or so (same as before)for 20 minutes and then stopped when the battery died.

I ran locally using run_docker_compose.sh and saw something different. One vehicle took off with a normal z input and another vehicle showed the same behavior as above without taking off. The vehicle that didn’t take off appeared to be receiving and sending ros data. The second data point would point to something other than a realtime dependency, but I am not sure what might cause behavior like that.

‌

osrf-migration commented 5 years ago

Original comment by GoRobotGo (Bitbucket: GoRobotGo).

Possibly related messages from X3b-robot_localization_ukf_node_odom-1-stdout.log (I believe used as part of the underlying controller we send messages to).

[33m[ WARN] [1569671036.182020883, 3647.680000000]: Failed to meet update rate! Took 0.01200000000000000025[0m
[33m[ WARN] [1569671037.112475268, 3647.828000000]: Transform from X3b/base_link/imu_sensor to X3b was unavailable for the time requested. Using latest instead.

The first message occurs every 10 seconds or so locally. In the cloud it appears at about 10Hz and the second was not seen in a short run locally.

osrf-migration commented 5 years ago

Original comment by Sophisticated Engineering (Bitbucket: sopheng).

These errors have also been discussed in issue #155.

osrf-migration commented 5 years ago

Original comment by Alfredo Bencomo (Bitbucket: bencomo).

changed assignee_account_id from "557058:90bc87b0-2886-4377-a17c-3a6629a697ff" to "557058:095b1e12-74ed-4e20-b44f-2f0745b616e0"
changed assignee from "azeey (Bitbucket: azeey, GitHub: azeey)" to "nkoenig (Bitbucket: nkoenig, GitHub: nkoenig)"

Nate Koenig (Nathan Koenig) is now working on this.

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

changed assignee_account_id from "557058:095b1e12-74ed-4e20-b44f-2f0745b616e0" to "557058:90bc87b0-2886-4377-a17c-3a6629a697ff"
changed assignee from "nkoenig (Bitbucket: nkoenig, GitHub: nkoenig)" to "azeey (Bitbucket: azeey, GitHub: azeey)"

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

We are deploying a change to limit the maximum simulation speed to 1x real-time, see issue #206. This may solve your problem.

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

We were able to reproduce this problem when the RTF in simulation was greater than 100%. So, I believe this issue will be resolved when we deploy the rate limiter to cloudsim.

osrf-migration commented 5 years ago

Original comment by GoRobotGo (Bitbucket: GoRobotGo).

changed priority from "major" to "blocker"

Bumping the priority as this prevents runs from working.

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

Production has been updated with RTF limiter. Give it a try.

osrf-migration commented 5 years ago

Original comment by GoRobotGo (Bitbucket: GoRobotGo).

This issue is still present. See the try7b cloudsim run

‌

osrf-migration commented 4 years ago

Original comment by GoRobotGo (Bitbucket: GoRobotGo).

This issue can be closed. The new controller for the x3 (and probably the x4) behave the same locally as in cloudsim.

osrf-migration commented 4 years ago

Original comment by Angela Maio (Bitbucket: Angela Maio).

changed state from "new" to "resolved"