simulations stopping with assert currentTimes.simTime <= simTime()

[klonikar]

All the simulations are failing with this error on the latest code.

The simulations were working in an old version which was part of the pre-created VM (OVA file). But none of the showcase ini files are working in the latest version. I have changed the paths for ArduCopter/ArduPlane/ArduRover in ini file to appropriate values and the SITL processes are running in the background.

[Thlamz]

Hello, thanks for using the project, and sorry for the slow response!

This 'currentTimes.simTime <= simTime()' error happens when the "RealTimeScheduler" can't keep up with real time. The scheduler is the OMNeT++ class responsible for dispatching events in the correct timings, the RealTimeScheduler provided by INED dispatches these messages to try to keep the simulation synched with real time. When the simulation is too slow (could be too many vehicles, computer resources, overhead from the virtual machine, etc.) the scheduler raises these exceptions.

An easy workaround is to simply remove the RealTimeScheduler, it should be on your .ini file with the key "scheduler-class". Removing this loses you the guarantee that your simulation will be synched with real time. You can still get pretty good results by manually controlling the simulation speed to approximate real time. If being precisely in synch with real time isn't one of your requirements, I would highly suggest this solution. If you do want to keep things synchronized, I'm afraid you will have to opt for a lighter simulated scenario

I was never able to figure out the exact mechanism that causes this assertion to fail. I highly suspect it's some implementation detail of the RealTimeScheduler class that I just never figured out. If I ever do, I will make sure to include a patched version of the class with the simulator. If you find a solution or the root-problem, please share it with us!

Also, I made a new .ova file to use with the new version of the simulator, that should help you with the setup if you are having trouble with that: https://drive.google.com/file/d/12fdnAZPwr71j6yR2qScmUMv6U0uTFbf0/view?usp=drive_link

[klonikar]

Hello, thanks for using the project, and sorry for the slow response!

This 'currentTimes.simTime <= simTime()' error happens when the "RealTimeScheduler" can't keep up with real time. The scheduler is the OMNeT++ class responsible for dispatching events in the correct timings, the RealTimeScheduler provided by INED dispatches these messages to try to keep the simulation synched with real time. When the simulation is too slow (could be too many vehicles, computer resources, overhead from the virtual machine, etc.) the scheduler raises these exceptions.

An easy workaround is to simply remove the RealTimeScheduler, it should be on your .ini file with the key "scheduler-class". Removing this loses you the guarantee that your simulation will be synched with real time. You can still get pretty good results by manually controlling the simulation speed to approximate real time. If being precisely in synch with real time isn't one of your requirements, I would highly suggest this solution. If you do want to keep things synchronized, I'm afraid you will have to opt for a lighter simulated scenario

I was never able to figure out the exact mechanism that causes this assertion to fail. I highly suspect it's some implementation detail of the RealTimeScheduler class that I just never figured out. If I ever do, I will make sure to include a patched version of the class with the simulator. If you find a solution or the root-problem, please share it with us!

Also, I made a new .ova file to use with the new version of the simulator, that should help you with the setup if you are having trouble with that: https://drive.google.com/file/d/12fdnAZPwr71j6yR2qScmUMv6U0uTFbf0/view?usp=drive_link

Thanks for responding. I will try out the removing the RealTimeScheduler. At this time, real timeness is not a strict requirement for me. I do not know why is it not able to keep up with real time even when trying it on a quite powerful linux machine and the default ini file showcases/sitl/simple_simulation/omnetpp.ini config SimpleSitl config which has *.numUAVs = 2. If I find a workaround/solution, I will certainly share.

Do I need to replace the RealTimeScheduler with something else or simply remove/comment the line?

[klonikar]

The simulations are working now after commenting out the RealTimeSimulator line. Among the 4 sitl showcases (gradys-simulations -> showcases -> sitl), 3 are working. The simple_simulation (SimpleSitl of ini file) starts but the vehicle does not move. I changed the waypoints file also but there is no movement. I think the logic in DroneMobilityMav::readWaypointsFromFile in DroneMobilityMav.cc may have something to do with it. Your if-else statements are processing GOTO, TAKEOFF, RETURN_LAUNCH commands but the file has only GOTO and REVERSE commands. In this function, you are not calling queueMessage like you do in MAVLinkFileMobility::readMissionFromFile and MAVLinkFileMobility::startMission.

The file mission.waypoints for file_mobility_simulation has more command types.

I am not that familiar with omnet++ so I could not dig further.

[klonikar]

@Thlamz your OVA is ubuntu 20.04 and all the simulations are working with RealTimeScheduler enabled. Can you briefly describe your changes?

I see that your omnet++ is also built with OSG, OSGEarth but map imagery is not visible. Possibly that makes a difference as CPU does not have to render it.

Even the SimpleSitl is working (I only changed the paths to point to ArduCopter.elf and param files): The two quad vehicles properly go to sensor[0] and sensor[1] and return.

The only differences I see are ubuntu 20.04, python 3.8 instead of 3.11 and the git branch th-fixes instead of main... Even after shifting to main branch, SimpleSitl works. I am not clear why does it fail in my environment.

[Thlamz]

@klonikar I mainly changed configuration and build stuff. The main things I remember:

• Some of the paths set in Paths and Symbols were absolute and invalid on my machine, so I changed them to the proper ones
• I added the -lpython3.8 flag to get the python integration to work.
• Changed the protocol path for all modules that used the python integration
• Changed the .elf and param file paths for all modules that used the SITL integration
• The python sample in the showcases folder was completely broken, it's still a bit broken but at least it opens now, the python files were completely missing in the repository

You can see the full changes on PR #33. It's still being reviewed, which is why I didn't merge it yet.

I chose Ubuntu 20.04 because I always had trouble installing OSG on Ubuntu 22.04, which I think you yourself described here: https://github.com/Project-GrADyS/gradys-simulations/issues/31#issuecomment-1920563616. To save myself the hassle I always just go for 20.04.

The map imagery really is not visible, even though I saw no visible errors. I am sure that with this same installation using python 5.6.2 the map imagery used to show. Since it was only visual I didn't investigate further, assumed it was just an incompatibility between OMNeT 6 and the OSG libraries. Your comment that it worked on Ubuntu 22.04 left me curious, I'll try to investigate further.

[Thlamz]

I see that your omnet++ is also built with OSG, OSGEarth but map imagery is not visible. Possibly that makes a difference as CPU does not have to render it.

@klonikar fixed it and reuploaded the images! You can download the new one here. In case you don't want to redownload the image, the fix was very simple, just set the environment variable MESA_GL_VERSION_OVERRIDE=3.3.

[klonikar]

I see that your omnet++ is also built with OSG, OSGEarth but map imagery is not visible. Possibly that makes a difference as CPU does not have to render it.

@klonikar fixed it and reuploaded the images! You can download the new one here. In case you don't want to redownload the image, the fix was very simple, just set the environment variable MESA_GL_VERSION_OVERRIDE=3.3.

That was quick 👍 I wasn't really complaining about the imagery not there... Rather, I thought the SimpleSitl worked because imagery was not there!! Thanks.

[klonikar]

@klonikar I mainly changed configuration and build stuff. The main things I remember:

• Some of the paths set in Paths and Symbols were absolute and invalid on my machine, so I changed them to the proper ones
• I added the -lpython3.8 flag to get the python integration to work.
• Changed the protocol path for all modules that used the python integration
• Changed the .elf and param file paths for all modules that used the SITL integration
• The python sample in the showcases folder was completely broken, it's still a bit broken but at least it opens now, the python files were completely missing in the repository

You can see the full changes on PR #33. It's still being reviewed, which is why I didn't merge it yet.

I chose Ubuntu 20.04 because I always had trouble installing OSG on Ubuntu 22.04, which I think you yourself described here: #31 (comment). To save myself the hassle I always just go for 20.04.

The map imagery really is not visible, even though I saw no visible errors. I am sure that with this same installation using python 5.6.2 the map imagery used to show. Since it was only visual I didn't investigate further, assumed it was just an incompatibility between OMNeT 6 and the OSG libraries. Your comment that it worked on Ubuntu 22.04 left me curious, I'll try to investigate further.

Thanks for the detailed reply. There is nothing in the PR that will solve the problem for SimpleSITL and also the RealTimeScheduler... Its only some added python files and change to use python 3.8 which is default for ubuntu 20.04.

This leaves me to wonder why does it work in your OVA but not in my real ubuntu 20.04 machine or the 22.04 OVA.

[klonikar]

@Thlamz I think I found out the reason for RealTimeScheduler not working. When I changed the build type to debug (for both gradys-simulations and inet4.5) in your OVA VM, I always get the simTime() error. When I change it back to release build, I dont get it.

Then I created a brand new ubuntu 20.04 VM, installed all the required packages as given in omnet++ InstallGuide.pdf, including osg, osgearth. Changed the configure.user to enable osgearth, built the osgearth, inet4.5 in release mode... finally built the gradys-simulations in release mode and launched SimpleSitl.

I will close this issue unless you have something to add.

[Thlamz]

@klonikar Nothing to add! OMNeT does tend to run much slower in debug mode, so what you said makes complete sense.