Multiple vehicles with latest Airsim release

[wuwushrek]

Hello,

I have tried to build this project on both Linux and Windows. On linux , I manage to compile this repository but wasn't able to add the plugin in either UE 4.14 and 4.15. First I had the eigen root problem which I solved by setting the EIGEN_ROOT variable and then the program rebuit UE4 Editor which take a times and ends up again with the same problem as this This Issue on Airsim . On Windows , I am not able to build the project due to eigen.target missing despite the fact that EIGEN_ROOT is set. Anyway , Since the latest release of Airsim with UE4 version 1.6 works pretty well on my Linux with SITL and HIL I was wondering if there is any way to achieve what you did here ??? If there is , How can I achieve that ? I tried replicating BP_FlyingPawn but with no success since I can't find the mavlink tab in UE4 to assign each Object some properties

Thanks in advance for your Answer

[saihv]

@wuwushrek In the Linux thread you mentioned, there were some solutions which fixed the problems to some extent: have you tried them by any chance? Do the problems persist even then?

Also, I have no problems building it in Windows, so looks like a configuration issue. Could you please post the full error message you're getting?

The newer version of AirSim completely removes blueprints for the sim and encodes everything in C++, which means a complete overhaul would be needed for multiple vehicles to work. So I currently don't have plans for upgrading it, but you're always welcome to try and submit PRs! I might look into it in the future when I have some more time. If you're interested in trying it, let me know and I can give you more pointers.

[wuwushrek]

Thanks for the reply @saihv

I believe I tried all of them. but I'm going to clean everything and start from scratch today. I guess UE 4.15 is the recommended version for your code ?

Yes I would like to try with the new version of Airsim . But I don't know where to start. What are your advices ?

I prefer to work on Linux machine but here is my error on the Window machine _ "C:\Users\fbd247\Documents\AirSimMulti\AirSim.sln" (default target) (1) -> "C:\Users\fbd247\Documents\AirSimMulti\DroneShell\DroneShell.vcxproj" (default target) (2) -> C:\Users\fbd247\Documents\AirSimMulti\DroneShell\DroneShell.vcxproj(189,3): error MSB4019: The imported project "C:\U sers\fbd247\Documents\AirSimMulti\AirLib\build\eigen.targets" was not found. Confirm that the path in the decl aration is correct, and that the file exists on disk.

"C:\Users\fbd247\Documents\AirSimMulti\AirSim.sln" (default target) (1) -> "C:\Users\fbd247\Documents\AirSimMulti\AirLib\AirLib.vcxproj" (default target) (3) -> C:\Users\fbd247\Documents\AirSimMulti\AirLib\AirLib.vcxproj(317,3): error MSB4019: The imported project "C:\Users\fbd 247\Documents\AirSimMulti\AirLib\build\eigen.targets" was not found. Confirm that the path in the declaration is correct, and that the file exists on disk.

"C:\Users\fbd247\Documents\AirSimMulti\AirSim.sln" (default target) (1) -> "C:\Users\fbd247\Documents\AirSimMulti\HelloDrone\HelloDrone.vcxproj" (default target) (4) -> C:\Users\fbd247\Documents\AirSimMulti\HelloDrone\HelloDrone.vcxproj(185,3): error MSB4019: The imported project "C:\U sers\fbd247\Documents\AirSimMulti\AirLib\build\eigen.targets" was not found. Confirm that the path in the decl aration is correct, and that the file exists on disk.

"C:\Users\fbd247\Documents\AirSimMulti\AirSim.sln" (default target) (1) -> "C:\Users\fbd247\Documents\AirSimMulti\DroneServer\DroneServer.vcxproj" (default target) (6) -> C:\Users\fbd247\Documents\AirSimMulti\DroneServer\DroneServer.vcxproj(185,3): error MSB4019: The imported project "C: \Users\fbd247\Documents\AirSimMulti\AirLib\build\eigen.targets" was not found. Confirm that the path in the de claration is correct, and that the file exists on disk.

"C:\Users\fbd247\Documents\AirSimMulti\AirSim.sln" (default target) (1) -> "C:\Users\fbd247\Documents\AirSimMulti\Examples\Examples.vcxproj" (default target) (8) -> C:\Users\fbd247\Documents\AirSimMulti\Examples\Examples.vcxproj(188,3): error MSB4019: The imported project "C:\Users \fbd247\Documents\AirSimMulti\AirLib\build\eigen.targets" was not found. Confirm that the path in the declarat ion is correct, and that the file exists on disk.

0 Warning(s)
5 Error(s)_

Thanks

[saihv]

@wuwushrek Yes, 4.15 is the recommended Unreal version. Give it a shot with a clean installation (make sure to run cmake/getlibcxx.sh along with the other dependencies)!

About the new version of AirSim:

1. The first step would be, like you said, replicate the FlyingPawn assets to have as many as the number of drones you need (Here, I made a mistake in my documentation: the vehicle name is found under the property tag 'HIL', not 'Mavlink'. Sorry about that, I fixed it now).
2. Next, most of the functionality that was done inside the blueprint BP_SimModeWorldMultiRotor.uasset is now inside SimMode/SimModeWorldMultiRotor.cpp. This line replicates what the setup vehicles block does inside the blueprints, so the first step would be to repeat this functionality for multiple drones, similar to what the uasset file does within AirSimMulti.
3. Along side this, the subviews inside the HUD file need to be connected with their right camera counterparts coming from the multiple drones. This could be the challenging part.
4. Once multiple actors are successfully spawned within the world, the next step would be to make sure the vehicle->initialize part uses independent IP addresses, and has multiple fpv_vehicle_connectors to initialize Mavlink etc. All of the rest should be similar to what AirSimMulti has currently.

[wuwushrek]

@saihv I clean the installation , clone again the AirSimMulti repository. I did the cmake/getlibcxx.sh step then has been able to compile AirSimMulti. I add the plugin to the Unreal Project and try to build it but it ends up showing this:

LogInit:Warning: Still incompatible or missing module: libUE4Editor-MyProject2.so LogInit:Warning: Still incompatible or missing module: libUE4Editor-AirSim.so

Well during the integration of the plugin I got this message in the console :

In file included from /home/franck/Documents/Unreal Projects/MyProject2 4.15/Plugins/AirSim/Source/CameraLogger.cpp:2: ../../../../Documents/Unreal Projects/MyProject2 4.15/Plugins/AirSim/Source/CameraLogger.h:8:10: fatal > error: 'SimModeWorldMultirotor.h' file not found

include "SimModeWorldMultirotor.h".

The file actually exists but it is named "SimModeWorldMultiRotor.h" with R for rotor in upper case. With that change I had to rebuilt again the UE editor and finally I got this error:

ERROR: UBT ERROR: Failed to produce item: /home/franck/Documents/Unreal Projects/MyProject2 4.15/Plugins/AirSim/Binaries/Linux/libUE4Editor-AirSim.so

When I looked at the console , I noticed I got the same compilation error as in This thread . The thing is when looking at the thread it seems at the end that the solution for this problem is to upgrade to the latest Airsim release.

During the integration of the plugin , Haven't you got this issue ? What version of gcc have you used ? Thanks for the steps for adding multiple vehicule with the new Airsim version. I believe the third point with the HUD file will be the less fun part. I'm having a look on that.

[saihv]

I have not used this version on Linux at all: but I'm going to give it a shot over the weekend or next week to see if there are any fixes possible. I remember that thread you linked to, but back then it was more confusing as people had various versions of UE, clang and libcxx.

By the way, did you compile it with clang or gcc?

On Aug 12, 2017 3:24 PM, "wuwushrek" notifications@github.com wrote:

@saihv https://github.com/saihv I clean the installation , clone again the AirSimMulti repository. I did the cmake/getlibcxx.sh step then has been able to compile AirSimMulti. I add the plugin to the Unreal Project and try to build it but it ends up showing this:

LogInit:Warning: Still incompatible or missing module: libUE4Editor-MyProject2.so LogInit:Warning: Still incompatible or missing module: libUE4Editor-AirSim.so

Well during the integration of the plugin I got this message in the console :

In file included from /home/franck/Documents/Unreal Projects/MyProject2 4.15/Plugins/AirSim/Source/CameraLogger.cpp:2: ../../../../Documents/Unreal Projects/MyProject2 4.15/Plugins/AirSim/Source/CameraLogger.h:8:10: fatal > error: 'SimModeWorldMultirotor.h' file not found

include "SimModeWorldMultirotor.h".

The file actually exists but it is named "SimModeWorldMultiRotor.h" with R for rotor in upper case. With that change I had to rebuilt again the UE editor and finally I got this error:

ERROR: UBT ERROR: Failed to produce item: /home/franck/Documents/Unreal Projects/MyProject2 4.15/Plugins/AirSim/Binaries/ Linux/libUE4Editor-AirSim.so

When I looked at the console , I noticed I got the same compilation error as in This thread https://github.com/Microsoft/AirSim/issues/112 . The thing is when looking at the thread it seems at the end that the solution for this problem is to upgrade to the latest Airsim release.

During the integration of the plugin , Haven't you got this issue ? What version of gcc have you used ? Thanks for the steps for adding multiple vehicule with the new Airsim version. I believe the third point with the HUD file will be the less fun part. I'm having a look on that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/saihv/AirSimMulti/issues/2#issuecomment-322003800, or mute the thread https://github.com/notifications/unsubscribe-auth/ACKz1qU4laLlJJk3ivXFX0BhF7DZLdROks5sXgoHgaJpZM4O1Vya .

[wuwushrek]

I believe I'm using clang compiler . Since it is the one in the ./build.sh. Do you think this can be an issue ?

[wuwushrek]

@saihv

Finally I manage to build the AirSimMulti project on the Window 10 machine by just removing this line in This file and I had to do that also for DroneShell , DroneServer , Example and HelloDrone.

According to This documentation when Eigen lib is uncompressed it should be named eigen3 but when looking at this they are looking at ${EIGEN_ROOT}/Eigen3 with the capital E. I'm not sure if it is relevant but by precaution I rename my extracted file eigen-eigen-da9b4e14c255 to Eigen3. And After this I was able to build the project and install the plugin in UE 4.15.

The last thing is that I also succeeded in doing SITL with latest PX4 and multiple vehicules in Linux with the latest release of Airsim by following your advices. It should also work with HIL by just changing the settings.json file.

My main problem now is to deal with the different views in the HUD file. To be more precise I'm trying to split the main view in the number of quad in the arena and I don't see how I can do that in the HUD file since I'm only able there to add/remove the sub-window views corresponding to the different Image_types there.

Thanks

[saihv]

@wuwushrek That's great to hear! (I am guessing the latest version uses UE4.16?) Would you be willing to push the multi-vehicle version of the latest AirSim release here? I can create a new branch we can work on to update the AirSim version.

Strange that you had to modify the vcxproj file, I've never had issues with that. Usually Windows isn't as particular about case sensitivity as Linux (for example, that SimModeMulti(r)otor.h issue only happens in Linux). But I just checked the main AirSim repo and that 'eigen' line no longer exists in the AirLib.vcxproj file.

I am not completely sure about your last question: do you want to use splitscreen for however many number of drones you have? Splitscreen is controlled on the Unreal Engine's side, in Settings -> Maps and Modes.

[wuwushrek]

@saihv Yes the latest version uses UE4.16 As you have seen they made modifications to finally support multiple vehicles. Actually it doesn't work but the modifications they made is pretty much the same I did yesterday and I can't find the bug in the new version so I will wait until they fix it. In the meanwhile I'm trying to play with the split-screen and will notify you if I got some results.

[wuwushrek]

One last thing about splitting the screen, For some reasons Unreal doesn't think that there is two local player , so even if the Splitscreen check button is checked it doesn't split the screen. Do you know if there is a way to 'assign' local player in Unreal or maybe it is a mistake of mine in my modified version of the plugin.

[saihv]

Do you have multiple BP_FlyingPawn assets, or did you just drag one asset to multiple locations?

By the way: when you split the screen, do you want each sub-view to follow a respective drone? Because then it will get complicated, you might need multiple camera directors. Right now there is only one director, so when I fly multiple drones, I usually place the camera at a fixed location (where I can see all of them)

[wuwushrek]

I drag one asset to multiple locations. I guess I need to duplicate the BP_FlyingPawn to be able to make that possible.

Anyway it doesn't matter now since I'm now able to switch between view of different quads by pressing some keys. Another question I have is if it is possible to increase the size of the subview ? I guess there is a way to do this directly in Unreal Engine?

[saihv]

Resizing the 'containers' through BP_SimHUDWidget.uasset should help with that.

[saihv]

@wuwushrek

I've been looking into using SITL with AirSimMulti, and I was wondering if you had any thoughts about an issue I've been having, as you've used SITL with the latest AirSim.

Essentially, if I only have one drone in AirSim and one SITL node running, it works fine. But if I have two drones in AirSim, and one SITL node bound to one drone (the other drone is set to UseSerial:true and has all different ports and IP addresses, so it shouldn't be doing anything SITL wise): the SITL drone behaves in a very unstable manner. Did you have to change any frequencies or rates while you were setting up multiple drones with SITL?

[wuwushrek]

@saihv yes I remember I had that behavior yesterday. I haven't try anything else but if I'm not wrong by setting the SimSysID to different values for the different vehicules solve my problem.

I will check tomorrow if that was the solution for my problem.

[saihv]

Hm, it doesn't seem to be enough. I changed the SimSysID but it didn't help, I also tried changing all the other variables so that both drones only have the host IP in common.

Did you instantiate SITL using the ./build_posix_sitl_default/src/firmware/posix/px4 posix-configs/SITL/init/ekf2/iris_1 (and iris_2, as you had two drones) command?

Btw I'm unable to get the default Airsim version working with multiple vehicles (as you might have seen from my comment on your issue in that repo) so I was hoping you'd have some pointers on what modifications you made to the latest version get it working on your end.

[wuwushrek]

@saihv Ok can you create a branch so that I push my version here ? So you can give it a try I have only tested it on my computer Ubuntu 14.04 with UnrealEngine 4.16

[saihv]

Just made an empty branch made 'dev'. You should be able to submit a pull req with your code once you fork and checkout that branch. Thanks!

Alternatively, (this is way easier): you can push it to a repo of your own and I can clone it from there.

[wuwushrek]

Ok I will do this later today and notify you when it will be done.

[saihv]

Awesome, thank you.

[wuwushrek]

@saihv Here is the repository where I push the code AirSim. Normally you should not have problem using it. Well we will see.

You will also find in the file settings an example of my setting.json file and my .launch file for 2 quads with PX4. If you want to use the .launch file for PX4 you can just remove tags that include mavros and my personal offboard ros node.

How it works ? Just drag and drop BP_FlyingPawn in the environment and it will automatically create PIPCamera and cameraDirector for you. You just have to be sure that in details , below the Vehicule_Settings category, each vehicule is named according to the name you put in your .json file (basically what you did before with AirSimMulti).

There is one main CameraDirector who is pointing to the first Vehicle you set in the .json file. Alternatively, during runtime , by pressing key L you can set the main CameraDirector to point on others vehicles (Basically change the main view). Also you will have on the bottom sub-views for the different vehicles.

Finally, It is a little bit dirty my hack for enabling different vehicles name: Have a look at this file But yes, we can change that by doing as they did in the latest release of Airsim.

[saihv]

Thanks for the update and great job with the modifications! I was able to get it running on Windows as well (UE 4.17), extending it to three vehicles also worked. I am trying to fix a couple of issues in it, and was wondering if you also faced them on Linux.

1. The whole program crashes when I press Stop in Unreal, and I have to re-run it from Visual Studio. This doesn't happen with the latest AirSim so I can probably find a fix if it only happens on Windows.
2. Image recording also crashes because of a problem with FImageUtils::CompressImageArray(). This function should not be called from a non-game thread, so may be this is a problem on the AirSim end.

I believe the bug in the latest AirSim multi-vehicle version is a simple one, it's happening probably because of the code mishandling some pointers etc. Hopefully that will also be fixed soon.

PS: Which version of PX4 firmware (on SITL) did you use? Just the master branch or any specific release?

[wuwushrek]

@saihv Thanks

1. for the first problem I think it is a known problem the bug when you press stop or pause. I always had it with Unreal 4.16 but I don't remember if I had it with UE4.15. Look here

2. Well I haven't noticed that since I didn't use record. I will probably give it a try.

I also believe the bug with the latest AirSim is just a simple one. That's the reason why I didn't went too far on the multiple vehicules scenario. I hope they will fix it soon.

Oh I'm using the tag v.1.6.5 not the master branch because of some problems I recently faced with gazebo.

[saihv]

I see. I am going to try SITL as well with 1.6.5 and let you know how it goes. And about point 1, I think that was only an issue in 4.16 and not 4.17, because the latest commit of AirSim does not have this problem. I will see if I can find a fix for it, it should be in one of the recent commits.

I was able to fix the recording by disabling the use of that compress function, but the AirSim devs might have a cleaner solution.

[saihv]

@wuwushrek I copied your modifications to this repo, and was able to fix the UE "stop" button crash (it was a simple fix, the vehicle params were being set to zero before the world was deinitialized). I will push any fixes or improvements I can come up with in the 'dev' branch.

[wuwushrek]

@saihv Thanks for fixing the recording. And nice job for "Stop" button crash. that's a feature I actually really need haha. It was starting to become annoying to have to restart UE4 every time you change a config or want to restart PX4.

[saihv]

My comment about the recording was a little premature. 😆 The recording thread runs without crashing but the images are not viewable. I will push updates if I am able to fix it completely.