search

microsoft / AirSim-NeurIPS2019-Drone-Racing

Drone Racing @ NeurIPS 2019, built on Microsoft AirSim
https://microsoft.github.io/AirSim-NeurIPS2019-Drone-Racing/
MIT License
356 stars 89 forks source link

Airsim crashing when all drones finish the race #90

Closed ctoumieh closed 4 years ago

ctoumieh commented 4 years ago

Hi,

Airsim is crashing when the 2 drones finish the race with the following error:

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=131119 
Malloc Size=144272 LargeMemoryPoolOffset=275408 
Failed to find symbol file, expected location:
"/home/charbel/Documents/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym"
[2019.08.16-23.21.26:615][137]LogCore: === Critical error: ===
Unhandled Exception: SIGSEGV: unaligned memory access (SIMD vectors?)

[2019.08.16-23.21.26:616][137]LogCore: Fatal error!

0x0000000002742e17 AirSimExe!UnknownFunction(0x2542e17)
0x0000000002742b3b AirSimExe!UnknownFunction(0x2542b3a)
0x0000000002749d3d AirSimExe!UnknownFunction(0x2549d3c)
0x0000000002fe4af7 AirSimExe!UnknownFunction(0x2de4af6)
0x00000000031101be AirSimExe!UnknownFunction(0x2f101bd)
0x000000000270ccdb AirSimExe!UnknownFunction(0x250ccda)
0x000000000270cb91 AirSimExe!UnknownFunction(0x250cb90)
0x0000000002743787 AirSimExe!UnknownFunction(0x2543786)
0x000000000274a156 AirSimExe!UnknownFunction(0x254a155)
0x0000000002fe4af7 AirSimExe!UnknownFunction(0x2de4af6)
0x000000000310cf73 AirSimExe!UnknownFunction(0x2f0cf72)
0x0000000003113dcf AirSimExe!UnknownFunction(0x2f13dce)
0x000000000310da3d AirSimExe!UnknownFunction(0x2f0da3c)
0x000000000311bb39 AirSimExe!UnknownFunction(0x2f1bb38)
0x000000000310da3d AirSimExe!UnknownFunction(0x2f0da3c)
0x000000000311bb39 AirSimExe!UnknownFunction(0x2f1bb38)
0x000000000310da3d AirSimExe!UnknownFunction(0x2f0da3c)
0x0000000002fe4af7 AirSimExe!UnknownFunction(0x2de4af6)
0x00000000031101be AirSimExe!UnknownFunction(0x2f101bd)
0x0000000004a24435 AirSimExe!UnknownFunction(0x4824434)
0x000000000270ccdb AirSimExe!UnknownFunction(0x250ccda)
0x000000000270cb91 AirSimExe!UnknownFunction(0x250cb90)
0x0000000004ccfc97 AirSimExe!UnknownFunction(0x4acfc96)
0x0000000004cd3ca4 AirSimExe!UnknownFunction(0x4ad3ca3)
0x0000000004cd2bb8 AirSimExe!UnknownFunction(0x4ad2bb7)
0x0000000004ccd113 AirSimExe!UnknownFunction(0x4acd112)
0x0000000004a3a5b4 AirSimExe!UnknownFunction(0x483a5b3)
0x000000000267907e AirSimExe!UnknownFunction(0x247907d)
0x000000000266d49c AirSimExe!UnknownFunction(0x246d49b)
0x0000000002678f1c AirSimExe!UnknownFunction(0x2478f1b)
0x000000000269792f AirSimExe!UnknownFunction(0x249792e)
0x000000000268b630 AirSimExe!UnknownFunction(0x248b62f)
0x0000000004a2561a AirSimExe!UnknownFunction(0x4825619)
0x0000000004a21d37 AirSimExe!UnknownFunction(0x4821d36)
0x00000000056b290a AirSimExe!UnknownFunction(0x54b2909)
0x00000000056b23da AirSimExe!UnknownFunction(0x54b23d9)
0x0000000002b9cfbd AirSimExe!UnknownFunction(0x299cfbc)
0x0000000002b9bc03 AirSimExe!UnknownFunction(0x299bc02)
0x0000000002b9a045 AirSimExe!UnknownFunction(0x299a044)
0x00000000056ad161 AirSimExe!UnknownFunction(0x54ad160)
0x00000000056a7d9b AirSimExe!UnknownFunction(0x54a7d9a)
0x0000000005007a90 AirSimExe!UnknownFunction(0x4e07a8f)
0x0000000004e9213f AirSimExe!UnknownFunction(0x4c9213e)
0x000000000258364b AirSimExe!UnknownFunction(0x238364a)
0x0000000002589f5a AirSimExe!UnknownFunction(0x2389f59)
0x0000000005da447f AirSimExe!UnknownFunction(0x5ba447e)
0x00007f4917344b97 libc.so.6!__libc_start_main(+0xe6)
0x0000000002579029 AirSimExe!UnknownFunction(0x2379028)

[2019.08.16-23.21.26:655][137]LogExit: Executing StaticShutdownAfterError
[2019.08.16-23.21.26:667][137]LogCore: Warning: Unable to statfs('/home/charbel/Documents/AirSim/AirSimExe/Saved/Crashes/crashinfo-AirSimExe-pid-14713-01B4770008D722B13990B6EE6308F8AC/AirSimExe.log'): errno=2 (No such file or directory)
Malloc Size=72367 LargeMemoryPoolOffset=347791 
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

I joined a zip of the crash log.

Cheers, CT crashinfo-AirSimExe-pid-14713-01B4770008D722B13990B6EE6308F8AC.zip

ugurkanates commented 4 years ago

This happens to me a lot too. But not every time . It also happens to me when you disconnect your connected code (client) .

Interesting bug

ctoumieh commented 4 years ago

@madratman My bad. Almost the same error is reported in issue #77. In my case tho when both drones finish the crash happens and not just when one finishes. I am also sometimes getting the error in issue #94.

madratman commented 4 years ago

@ctoumieh @ugurkanates we just updated the binaries with a fix to these reset bugs. can you please re-download airsim.zip and let us know if it works now?

madratman commented 4 years ago

@ctoumieh @ugurkanates we only updated the v1.0 / qual binaries yesterday with the reset bug fix. Just updated v0.3 linux binaries as well. see https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/issues/94#issuecomment-545080883

ironclownfish commented 4 years ago

Another crash case has been fixed in the binaries. Please re-download if you were continuing to have problems.

ctoumieh commented 4 years ago

@madratman @ironclownfish Sorry for the delay, there is a revolution in my country. I have downloaded the latest binaries and now simLoadLevel(level_name) doesn't work anymore. The Level is staying the default one with two drones on top of each others.

Also when I run the baseline_racer I get:

  File "baseline_racer.py", line 285, in <module>
    main(args)
  File "baseline_racer.py", line 252, in main
    baseline_racer.start_race(args.race_tier)
  File "baseline_racer.py", line 46, in start_race
    self.airsim_client.simStartRace(tier)
  File "/home/charbel/.local/lib/python3.6/site-packages/airsimneurips/client.py", line 156, in simStartRace
    last_gate_position = deepcopy(competitor.gate_poses_ground_truth[-1].position)
IndexError: list index out of range
madratman commented 4 years ago

@ctoumieh I hope you find the peace and strength to keep going in these testing times, and that the people of your country get the government that you need, want, and deserve. I think you either forgot to download pak files / put them at the wrong place. that error you're seeing is expected if you are not in the race levels, as the default levels - with drones on top of each other - doesn't have any gates. so the return of simListSceneObjects with the gate regex is an empty list you can peruse the ./download_training_binaries.sh / ./download_qualification_binaries.sh script as well.

madratman commented 4 years ago

@ctoumieh this is fixed https://github.com/microsoft/AirSim-NeurIPS2019-Drone-Racing/issues/77#issuecomment-546072809

Please re-open if you still face this.

ctoumieh commented 4 years ago

@madratman Again sorry for the delay. I git pulled the repo, upgraded airsimneurips and used the script to download the qualification binaries. Now I am getting:

Signal 11 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554 
CommonUnixCrashHandler: Signal=11
Malloc Size=65535 LargeMemoryPoolOffset=131119 
Malloc Size=144272 LargeMemoryPoolOffset=275408 
Failed to find symbol file, expected location:
"/home/charbel/Documents/AirSim_Qualification/AirSimExe/Binaries/Linux/AirSimExe.sym"
Malloc Size=78675 LargeMemoryPoolOffset=354099 
Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
Segmentation fault (core dumped)

which is different from the error in this issue. It is not a huge issue for me though since it only happens when both drones finish the race.

ugurkanates commented 4 years ago

Signal 11 caught. Malloc Size=65538 LargeMemoryPoolOffset=65554 CommonUnixCrashHandler: Signal=11 Malloc Size=65535 LargeMemoryPoolOffset=131119 Malloc Size=111328 LargeMemoryPoolOffset=242464 Failed to find symbol file, expected location: "/home/paypaytr/Desktop/MicrosoftTrainingEnviroments/AirSim/AirSimExe/Binaries/Linux/AirSimExe.sym" Malloc Size=78495 LargeMemoryPoolOffset=320975 Engine crash handling finished; re-raising signal 11 for the default handler. Good bye. Segmentation fault (core dumped)

Same happens to me even though race is not done interesting

tadashiamada commented 2 years ago

@ctoumieh @ugurkanates did you find the solution to this problem. I am a new to this one but I face the same problem now