search

osrf / subt

This repostory contains software for the virtual track of the DARPA SubT Challenge. Within this repository you will find Gazebo simulation assets, ROS interfaces, support scripts and plugins, and documentation needed to compete in the SubT Virtual Challenge.
Other
305 stars 98 forks source link

Cloudsim simulation runs much longer than battery life if stop service is not called #341

Open osrf-migration opened 4 years ago

osrf-migration commented 4 years ago

Original report (archived issue) by Malcolm Stagg (Bitbucket: malcolmst7).

One of my simulations from tonight has been running for 127 minutes of simulation time so far:

[Dbg] [LinearBatteryPlugin.cc:371] [Battery Plugin] Battery drain: 127 minutes passed.

I know I should probably be calling the stop service. Honestly I didn’t want to make a change like that, as minor as it should be, at the last minute. It seems a bit pointless though that the simulation continues after all the robots are out of battery (90 minutes) and also after the 60 minute competition limit. Could this time limit be reduced in future circuits?

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

To make matters worse, 6 failed jobs from earlier now started running again on their own, so my quota of 10 jobs is maxed out (I think I can’t even submit yet because of that - haven’t tried yet though). As a related feature request, it would be very helpful to have the ability to cancel a job before completion.

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

osrf-migration commented 4 years ago

Original comment by Sarah Kitchen (Bitbucket: snkitche).

Agree on adding a request to cancel jobs to the web UI.

osrf-migration commented 4 years ago

Original comment by Alfredo Bencomo (Bitbucket: bencomo).

osrf-migration commented 4 years ago

Original comment by Steven Gray (Bitbucket: stgray).

Malcolm Stagg (malcolmst7) What sort of turnaround times are you seeing on runs? During the qualifier, I never saw results in less than 8 hours - even when calling the stop competition service after an hour - and was told it was a result of the AWS teardown time. The gazebo logs showed the simulator stopping after about 1.5 hours of real time, so the rest I figure was teardown.

[https://osrf-migration.github.io/subt-gh-pages/#!/osrf/subt/issues/319/cloudsim-runs-not-ending (#319)](https://osrf-migration.github.io/subt-gh-pages/#!/osrf/subt/issues/319/cloudsim-runs-not-ending (#319))

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

I’ve been seeing roughly 7-8 hours too usually. Judging from the realtime logs, with several robots it appears to be running at around 1/3 realtime, and doesn’t seem to end until around 150 minutes (2.5 hours) of simulation time. You’re right though I’m looking at one now where the log file ends after 105 minutes, so a fair bit of that time might be teardown related. Interestingly the battery plugin was still running for all 150 minutes, in at least one case I saw. If I’m remembering correctly, running a simulation with only a single robot has been somewhat faster in the past, but I could be mistaken.

Thanks for the link, I missed seeing that previous issue you posted. That's interesting that /subt/finish doesn't actually terminate the cloudsim simulation, I was under the impression that it would.

osrf-migration commented 4 years ago

Original comment by Paul Jurczak (Bitbucket: pauljurczak).

My submissions were calling callService("/subt/finish") after finishing in about 6 minutes (in qualifier), but the simulator was running for many hours.

osrf-migration commented 4 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).

To answer the original question, we don't stop simulation after robot batteries run out because a robot could still be used as a communication node. You could, for example, position a robot such that it acts as relay node even if it cannot move.

osrf-migration commented 4 years ago

Original comment by Zbyněk Winkler (Bitbucket: Zbyněk Winkler (robotika)).

I am analyzing the logs from simulation b2e9d531-5859-42d1-816f-9794cd568b38. I see the following lines in server_console.log:

(2020-03-11T21:30:50.967690741) [Dbg] [LinearBatteryPlugin.cc:371] [Battery Plugin] Battery drain: 0 minutes passed.

....

(2020-03-13T20:03:05.869641794) [Dbg] [LinearBatteryPlugin.cc:371] [Battery Plugin] Battery drain: 2758 minutes passed.
(2020-03-13T20:04:06.924347963) [Dbg] [LinearBatteryPlugin.cc:371] [Battery Plugin] Battery drain: 2759 minutes passed.
(2020-03-13T20:05:07.662877631) [Dbg] [LinearBatteryPlugin.cc:371] [Battery Plugin] Battery drain: 2760 minutes passed.

​That seems to suggest that the simulation run for full 3 days from March 11th to 13th and simulated 46h.

Due to #336 we are not calling /subt/finish. While #319 is marked as resolved, it might be related to this.

There is no point to simulate more than 1h, right?

osrf-migration commented 4 years ago

Original comment by Malcolm Stagg (Bitbucket: malcolmst7).

Up to last week my Urban Circuit (worlds 1-8) simulations were running for just over 1 hour of simulation time, but over the past week (along with the other simulator issues) that increased a ton for me too. Not sure if that might be related to the other issues that were going on. Some ran again today and it looks like they’re back to 1 hour though, so maybe that is resolved now?

nkoenig commented 4 years ago

Simulation will stop automatically after 1 hour of simulation time after the competition run has started. There is also a maximum of a 15 minute warmup period preceding the start of a run. So, a simulation instance could run for a total of 75 simulation minutes.

I will consider this issue now a proposal to add a "Stop" button to the web ui so that a team can end a simulation run early.

nkoenig commented 4 years ago

Adding a stop button will also resolve issue #361