The DeepGTAV framework can be used to produce ground truth training data with high accuracy for different tasks, by leveraging the high visual fidelity environment of GTAV.
In particular HD images, LiDAR, depth, stencil, labels, object segmentation and bounding boxes can be exported for self-driving car and aerial scenarios. Furthermore the system implements easy control through messages from a python client, that make it suitable for custom scenarios and in principle even reinforcement learning tasks.
We used synthetic training data generated with the DeepGTAV framework to good
real world object detection performance reported in our paper Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles
https://arxiv.org/abs/2112.12252.
Below we show some example images from DeepGTAV with the object bounding boxes (right) and corresponding images from real world datasets (left)
The datasets we used in work Leveraging Synthetic Data in Object Detection on Unmanned Aerial Vehicles
(https://arxiv.org/abs/2112.12252) are available under
https://cloud.cs.uni-tuebingen.de/index.php/s/cQ3Qt5z8o4e5GWo
The labels have the following id to category name mapping:
{"swimmer": 0, "floater": 1, "swimmer on boat": 2, "floater on boat": 3, "boat":4}
In each metadata file we have the following values separated by spaces:
[x y z heightAboveGround, campos_x, campos_y, campos_z, camrot_x, camrot_y, camrot_z, time_hours, time_min, time_sec, weather]
x,y,z
: the location of the player vehicle in GTAV world coordinates.heightAboveGround
: The height difference of the player vehicle to the ground (or water)campos_x, campos_y, campos_z
: The positions of the camera in world coordinates.camrot_x, camrot_y, camrot_z
: The camera rotationstime_hours, time_min, time_sec
: The ingame timeweather
: The ingame weather conditionNote that the camera position in general is offset wrt. the player vehicle. Also the height of the camera above ground is different from the heightAboveGround for the vehicle, so if you need this you need to calculate it.
In the following the steps that are necessary to just export data using DeepGTAV are described. In principle DeepGTAV should work with any version of GTAV, if the correct newest version of ScriptHookV is obtained. The last working version for me employed GTAV from the Epic store in version 1.0.2060.1
Desktop developement with C++
and Game Developement with C++
selected. This
installs some dependencies, that are needed, otherwise DeepGTAV crashes
without an error. bin/
to
DeepGTAV-PreSIL/Release/
overwriting the old files (this is not necessery
when using GTAV in version 1.0.2245.0 or older)DeepGTAV-PreSIL/bin/Release/
to the GTAV install
directoryDocuments/Rockstar Games/GTA V/Profiles/
with
the contents of DeepGTAV-PreSIL/bin/SaveGame
Documents/Rockstar Games/GTA V/Profiles/
, but keep this in mind if you
modify the game settings.VPilot/utils/Constants.py
. To do this
change the variable MANAGED_SCREEN_RESOLUTION
to the screen resolution at
which GTAV is running, e.g. "1920x1080". Default is "3840x2160DSR7680x4320"VPilot uses simple Python commands to interact with DeepGTAV. Examples of how
VPilot can be used are shown in VPilot/presentation_VisDrone.py
.
The other scripts in the VPilot
folder starting with datageneration_
can be
used to generate different synthetic training data matching different real world
datasets.
To do data generation simply run GTAV. When the game has loaded run one of the respective data generation scripts and specify the Export directory, e.g.:
python .\VPilot\datageneration_VisDrone.py --save_dir "C:\DeepGTAV_Exportdir\"
Then open the GTAV window again, ESC from the menu and the data generation should be starting.
In some rare cases the game crashes when the player is in a building while starting a data generation script. To prevent this leave the building before starting the data generation.
ATTENTION: Restrict the data that is recorded with the Dataset
message to only
the data that you need.
e.g.
dataset=Dataset(location=True, time=True, exportBBox2D=True)
against
dataset=Dataset(location=True, time=True, exportBBox2D=True, segmentationImage=True, exportLiDAR=True, maxLidarDist=120)
This improves the capturing speed by quite a lot.
In VPilot/backup/presentation_VisDrone_VisualizeMessageParts.py
examples on
how to handle different message parts are given (most in comments). Note that
most of the export messages are only there for legacy and debugging reasons. The
suported and useful exported message parts are the settings exportBBox2D, segmentationImage, exportLiDAR (with maxLidarDist)
in Dataset
and
corresponding message parts.
For an in depth understanding of the VPilot message interface, the best way to
start is to look at the file VPilot\deepgtav\messages.py
. The function
signatures of the __init__
should be realtively self explanatory.
DeepGTAV sends capturing messages to VPilot in every tick. Those messages encode all the recorded information and can be accessed by their fields (e.g. message['frame'] is the recorded image, message['bbox2d'] the recorded bounding boxes etc.)
Note that there are lots of legacy fields in the Dataset
message, which
sometimes do nothing. I would advise to only use the fields that are also used
in the example scripts and data generation scripts.
If you want to capture images from any location a hacky simple solution is to use a "freecam" mod to move the camera. Aikido Free Cam worked well for us (see this issue).
In general DeepGTAV should work stable with modifications of GTAV.
For my works I used the following modifications:
The modifications and install instructions can be found in the folder Mods
Different additional modifications could be used, mainly for graphics improvements. If you use this repository in conjunction with graphics enhancing mods I would love it if you send me a quick email telling me what mods you used and whether they worked.
This framework consists of different moving parts, that are organized in different folders. Their interaction is described in the following:
DeepGTAV-PreSIL
contains the code to build the scripts that start the DeepGTAV
server in GTAV. Communication with those scripts is done via a TCP-Port. In
VPilot
the functionalities to communicate with DeepGTAV are implemented, to
control DeepGTAV from a Python script.
DeepGTAV-PreSIL
uses the functionalities implemented in
GTAVisionExport-DepthExtractor
to extract depth and stencil information. To
communicate with GTAV DeepGTAV uses SkriptHookV
.
In the Mods/Readme.md
, installation instructions for different mods that were
used to improve GTAV are shown. Mainly those are changes to spawn more objects
in the game world (e.g. cars and pedestrians) but graphics enhancements will be
added in the future.
In depth explanations can be found in bachelors thesis. In the future I will try to do better documentation, but for now this has to suffice.
If you want to compile the binaries by yourself or make your own changes you have to do the following things:
DeepGTAV was built using MS Visual Studio 2017 and I recommend to also use it to circumvent compatability issues.
clone this repository
TODO get all requirements for DeepGTAV-PreSIL and copy them to this directory, in particular those are:
Build GTAVisionExport-DepthExtractor and zeroMQ. Specific build instructions can be found in the individual projects. In principle it suffices to build using cmake and then building them in Visual Studio. A detailed description is as follows:
"..\..\deps\libMinHook.x64.lib"
It is important to mimic this exact file structure, otherwise the relative "Additional Include Directories" do not fit correctly. If this is the case and for some reason you want to include those directories from different locations, you have to change the "Additional Include Directories" in the following way:
Right click DeepGTAV in the SolutionExplorer in Visual Studio -> Properties -> C/C++ -> Additional Include Directories
Here the include directories have to match.
Furthermore in
Properties -> Linker -> Input -> Additional Dependencies
the correct locations of the libraries have to be set
Build GTAVisionExport-DepthExtractor as described. Copy the files from
GTAVisionExport-DepthExtractor\native\build\src\Release
to
DeepGTAV-PreSIL\bin\Release
(This is not necessary, if DepthExtractor has
not been changes).
Set environment variable GTAV_INSTALL_DIR
to point to the root directory of
GTAV. This is done so that on building DeepGTAV-PreSIL the newly built files
are automatically copied into the GTAV install directory.
Set environment variables DEEPGTAV_EXPORT_DIR
and DEEPGTAV_LOG_FILE
(those are legacy requirements and should be removed in future versions)
This repository is built upon different other works. If you use this repository please cite those respective works and this repository.
The initial DeepGTAV framework was built in this repo: https://github.com/aitorzip/DeepGTAV
In the work "Driving in the Matrix: Can Virtual Worlds Replace Human-Generated Annotations for Real World Tasks?" (https://arxiv.org/abs/1610.01983) object segmentation exportation from GTAV was provided: https://github.com/umautobots/GTAVisionExport
In "Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception" (https://arxiv.org/abs/1905.00160) those two functionalities were combined: https://github.com/bradenhurl/DeepGTAV-PreSIL
Primarily this repository contains changes that were made to allow the capturing of data from UAV perspective. Additionally some quality of life changes have been made, a comprehensive list of the improvements is given in the following:
In principal, the publisher of GTAV (Rockstar Games) allows for non-commercial use of game footage (http://tinyurl.com/pjfoqo5), which includes academic work. Commercial use of the game footage however is not allowed in general.
Furthermore when working or extending this tool it has to be considered that new game patches could introduce breaking changes at any time. Note however that this has not happened since the release of the game, so this is rather unlikely.
[x] Implement object spawning
[x] increase amount of spawned objects
[x] balance spawned object classes
[x] Adapt for higher resolution capturing
StopCapture
message does
not come through and too many captures are sent for some frames). If this
happens increase the spacing between captures, or reduce the amount of
captured data.