RPC Error starting up AirSim/UE on Docker

xxEoD2242 commented 3 years ago

Bug report

AirSim Version/#commit: 1.40 using PR #3449
UE/Unity version: 4.25
autopilot version: N/A
OS Version: Ubuntu 20.04 Server (Docker using Ubuntu 20.04)

What's the issue you encountered?

Once AirSim initializes, it makes a call to set up the HUD Widget. I then get this crash report:

Everything after that freezes the gameplay

Settings

How can the issue be reproduced?

Pull down the PR, compile AirSim, then compile the entire project into an executable.
Run the executable in the Docker Container.

Include full error message in text form

What's better than filing an issue? Filing a pull request :).

xxEoD2242 commented 3 years ago

Ignore this. I'm an idiot.

Just in case this happens to anyone:

I was running the Unreal Engine executable as background process through a shell script. Pretty much, when you use ctrl-c, it doesn't actually send the kill signal to Unreal Engine. I would then try to restart my shell script and would get this error because I was basically trying to run 2 Unreal Engines at the same time.

Fix

Use the kill command to kill off the processes listed using ps.

xxEoD2242 commented 3 years ago

Actually, this continues to happen. Once you begin commanding the drones to do something, I keep getting this error.

rajat2004 commented 3 years ago

@xxEoD2242 Are you using the Docker scripts in the AirSim repo or your own image and docker commands? I'll also test it out and see if it works. I do have a binary compiled recently (for https://github.com/microsoft/AirSim/pull/3436) so mostly won't need to compile it again.

Also, hopefully it's not something deeply problematic with rpclib, that project seems to be dead, so any fixes required will be even more painful. Might be time to look for a modern RPC framework so something else

Sidenote, do you also have to deal with the recompilation of the UE4 Editor every time? I managed to reduce it by commenting out the removal of Saved and Binaries directory, but it still recompiles some portion everytime

xxEoD2242 commented 3 years ago

Hey @rajat2004,

So, I'm brewing my own monster. What I do is build a development package a lot like what you guys did for the Drone Racing lab and then set it up as an executable to run. I don't have to recompile the UE4 editor unless I need to make drastic changes.

So, the problem I reported is gone with the whole not running 2 instances at once. There is a bigger issue:

I have now deduced that something broke the Task Graph for AirSim V1.4.0 with UE 4.25. I figured this out by going back about 40 commits to December 22nd, 2020 and used a random commit from that time period to get it to work.

To describe what happens with the current (fresh) pull of AirSim:

Set up any Python script with AirSim, using the airsim folder in the PythonClient folder automatically in the execution directory.
Once you run it and call any .join() on an async task, the whole AirSim process freezes. But Unreal is still working. However, no task in AirSim is completing. When I do a kill on the process, it points me to RPC(Tornado) waiting to execute. I forgot to screenshot it, so apologies there. When you don't do any join calls but just try to give commands, AirSim and by proxy the drones, don't do anything or ever change position.

When you wait about a minute, UE starts complaining that the Task Graph took to long to complete. The only way I know how to debug this is to systematically start crawling through commits until we figure out where this broke at.

For context, I am using a fresh pull of UE4.25 and have tried this running both OpenGL and Vulkan through off-screen rendering. Same issue and the only thing that fixed it was reverting AirSim back to this commit: 1af0a4e24216cd47df4c77795358638bd24ea8ec ( I chose this randomly).

I initially tried a commit about 2 weeks ago (I think its 4e851980173c667ab71d9394d91e814e3d2ccd14). This had the same issue.

Once I went through the whole process again, it worked like a charm.

If you would like to tag team this to figure out what went wrong, I'd be happy to help.

xxEoD2242 commented 3 years ago

@rajat2004 Have you consider moving towards something like gRPC? I've used that for other micro-services and might integrate with the framework of having multiple micro-services for users to run. We are actually looking at using that to simulate communications between drones for own purposes.

rajat2004 commented 3 years ago

That seems nasty, and does this happen in a binary only or in the editor as well? This probably shouldn't require Docker as well to reproduce, but maybe I'm wrong. The only PR I can even remotely think of connected to this would be https://github.com/microsoft/AirSim/pull/3363. Could #3451 be also related to this?

Another sidenote, I'm not a maintainer, just a contributor who likes to dabble in stuff :). gRPC looks nice, haven't worked with it before though, another thing to play with

xxEoD2242 commented 3 years ago

Hey @rajat2004, #3363 looks pretty likely considering it's using those mutex's. I wonder if something with UE or Ubuntu is causing them to not release their mutex locks on Docker for some reason. I'll try to pull directly after, rebuild and test to confirm.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had activity from the community in the last year. It will be closed if no further activity occurs within 20 days.

microsoft / AirSim