This is a reboot of the project from 2017, but with a whole new approach. This time, instead of collecting data by hand and training a classification model, we opted to create a whole new system for the data collecting, training and playing, and to do all of that live. Our models are now using regression (with other output types being still possible), training data is collected automatically and we stream models during their training phase so anyone can join and watch the progress.
Our AI, besides having multiple alter-egos (models), is still called Charles, like in 2017.
For a full explanation, refer to the System
page, but for a quick summary - we’re using a central server that all other parts connect to. The Data Collectors
run in separate GTA5 instances and our NPCs are collecting and Balancing
the data, which is sent through the Server
to the Trainer
. The Trainer
buffers these data, trains the model and updates the Player
. The Player
is using the Dual-Camera
system to play so we can watch the 3rd person camera (also called cinematic camera) while the model is fed the Hood Camera
.
For more information, refer to these other pages:
System
- describing how all parts are working together to train and inference the modelsCameras
- how we created multiple camerasConvcam
- what the Convcam is and how we use itData Balancing
- how we are balancing regression dataNPCs
- our custom NPCs playing in the GTA5 to collect data automaticallyPurpose
- a way to let the model “know” where to driveStorage and Buffer
- how are we managing the training data and why random batches are importantUnstuck
- how to make the car not be stuck anywhereXception
- the first CNN backbone architecture usedInceptionResNetv2
- the second, more successful, CNN backbone architecture usedTensorboard logs
- all of our Tensorboard logs for all of the trained modelsmodel_0001_xception
- the first logged Xception
model, Balancing_v1
model_0002_xception
- trying Balancing_v2
and a custom modelmodel_0003_xception
- Balancing_v3
and back to a standard modelmodel_0004_inceptionresnetv2
- trying a bigger model - InceptionResNetv2
model_0005_inceptionresnetv2
- transfer-learning attemptmodel_0006_inceptionresnetv2
- added stacked imagesmodel_0007_inceptionresnetv2
- trying a history inputsmodel_0008_irv2_data_td
- history with recurrent layersmodel_0009_irv2_cr_tl
- history with recurrent layers with transfer-learningmodel_0010_irv2_tcb
- dual-backbone InceptionResNetv2
model_0011_x_tcb
- dual-backbone Xception
model_0012_regnet
- TBA
The progress is being streamed on Twitch: https://www.twitch.tv/sentdex. Below we keep a list of dates and streamed models.
Stream's layout consists of:
Hood Camera
is exactly what the models "see" - the input to the convolutional backbone of the modelConvcam
shows the reshaped output of the CNN backbone and lets us observe how the CNN part trainsPlayer Console
shows current driving predictions along with additional informationServer/Trainer Console
shows training progress along with some basic training informationTensorboard
shows the loss of the training process
model_0001_xception
- the first logged Xception
model, Balancing_v1
model_0002_xception
- trying Balancing_v2
and a custom modelmodel_0003_xception
- Balancing_v3
and back to a standard modelmodel_0004_inceptionresnetv2
- trying a bigger model - InceptionResNetv2
model_0005_inceptionresnetv2
- transfer-learning attemptmodel_0006_inceptionresnetv2
- added stacked imagesmodel_0007_inceptionresnetv2
- trying a history inputsmodel_0008_irv2_data_td
- history with recurrent layersmodel_0009_irv2_cr_tl
- history with recurrent layers with transfer-learningmodel_0010_irv2_tcb
- dual-backbone InceptionResNetv2
model_0011_x_tcb
- dual-backbone Xception
model_0012_regnet
- TBA
May 1st:
model_0003_xception_v2
from scratchMay 2nd:
model_0003_xception_v2
at batch 15250model_0003_xception_v1
from batch 24500 (pre-trained off-stream)May 7th:
model_0003_xception_v1
at batch 162650model_0004_inceptionresnetv2_v1
from scratchmodel_0004_inceptionresnetv2_v1
at batch 1250model_0004_inceptionresnetv2_v2
from scratchmodel_0004_inceptionresnetv2_v2
after ~160 batchesmodel_0003_xception_v1
from batch 162650 (continuing training)model_0003_xception_v1
at batch 165250model_0004_inceptionresnetv2_v1
from batch 1250 (continuing training)May 8th:
model_0004_inceptionresnetv2_v1
at batch 12000model_0005_inceptionresnetv2_v1
from scratchmodel_0005_inceptionresnetv2_v1
at batch 3500model_0004_inceptionresnetv2_v3
from batch 22250 (pre-trained off-stream)May 14th:
model_0004_inceptionresnetv2_v3
at batch 184250model_0006_inceptionresnetv2_v1
from batch 18750 (pre-trained off-stream)May 16th:
model_0006_inceptionresnetv2_v1
at batch 60500May 28th:
model_0009_irv2_cr_tl_v1
from scratchMay 29th:
model_0009_irv2_cr_tl_v1
at batch 19250model_0004_inceptionresnetv2_v3
from batch 184250 (continuing training)July 5th:
model_0004_inceptionresnetv2_v3
at batch 752000
This is not a full project log and contains only the key changes that are visible on the stream (and a few from before the stream has started). There are many more things going on "behind the scenes".
2017:
November 2021:
Early 2022:
April 8th:
Balancing_v1
model_0001_xception_v1
from scratchmodel_0001_xception_v1
at batch 10800model_0001_xception_v2
from scratchApril 9th:
model_0001_xception_v2
at batch 7150model_0001_xception_v3
from scratchmodel_0001_xception_v4
from scratchmodel_0001_xception_v3
at batch 18850model_0001_xception_v5
from scratchApril 10th:
model_0001_xception_v5
at batch 25200model_0001_xception_v3
at batch 23300model_0001_xception_v6
from scratchApril 11th:
Data Balancing
April 15th:
model_0001_xception_v6
at batch 176850April 19th:
model_0002_xception_v1
from scratchApril 22nd:
model_0003_xception_v1
from scratchApril 24th:
model_0003_xception_v1
at batch 24500April 30th:
model_0002_xception_v1
at batch 249750May 1st:
model_0003_xception_v2
from scratchMay 2nd:
model_0003_xception_v2
at batch 15280model_0003_xception_v1
from batch 24500 (pre-trained off-stream)model_0003_xception_v3
from scratchMay 3rd:
model_0003_xception_v3
at batch 17500model_0003_xception_v4
from scratchmodel_0003_xception_v4
at batch 5150May 7th:
model_0003_xception_v1
at batch 162650model_0004_inceptionresnetv2_v1
from scratchmodel_0004_inceptionresnetv2_v1
at batch 1250model_0004_inceptionresnetv2_v2
from scratchmodel_0004_inceptionresnetv2_v2
after ~160 batchesmodel_0003_xception_v1
from batch 162650 (continuing training)model_0003_xception_v1
at batch 165350model_0004_inceptionresnetv2_v3
from scratchmodel_0004_inceptionresnetv2_v1
from batch 1250 (continuing training)May 8th:
model_0004_inceptionresnetv2_v1
at batch 12000model_0005_inceptionresnetv2_v1
from scratchmodel_0005_inceptionresnetv2_v1
at batch 3500model_0004_inceptionresnetv2_v3
at batch 22250model_0004_inceptionresnetv2_v3
from batch 22250 (pre-trained off-stream)May 13th:
model_0006_inceptionresnetv2_v1
from scratchMay 14th:
model_0006_inceptionresnetv2_v1
at batch 18750model_0004_inceptionresnetv2_v3
at batch 184250model_0006_inceptionresnetv2_v1
from batch 18750 (pre-trained off-stream)May 14th:
model_0006_inceptionresnetv2_v2
from scratchmodel_0006_inceptionresnetv2_v2
at batch 5750May 16th:
model_0006_inceptionresnetv2_v1
at batch 60500May 22nd:
model_0007_inceptionresnetv2_v2
from scratchmodel_0007_inceptionresnetv2_v2
at batch 10500May 23th:
model_0008_inceptionresnetv2_v1
from scratchMay 24th:
model_0008_inceptionresnetv2_v1
at batch 35000model_0008_inceptionresnetv2_v2
from scratchmodel_0008_inceptionresnetv2_v3
from scratchMay 25th:
model_0008_inceptionresnetv2_v2
at batch 19000model_0008_inceptionresnetv2_v3
at batch 17000model_0008_inceptionresnetv2_v4
from scratchmodel_0008_inceptionresnetv2_v5
from scratchMay 26th:
model_0008_inceptionresnetv2_v5
at batch 37250May 28th:
model_0008_inceptionresnetv2_v4
at batch 35500model_0009_irv2_cr_tl_v1
from scratchmodel_0009_irv2_cr_tl_v2
from scratchMay 29th:
model_0009_irv2_cr_tl_v1
at batch 19250model_0004_inceptionresnetv2_v3
from batch 184250 (continuing training)May 30th:
model_0009_irv2_cr_tl_v2
at batch 20000model_0009_irv2_cr_tl_v3
from scratchMay 31th:
model_0009_irv2_cr_tl_v3
at batch 15000model_0009_irv2_cr_tl_v4
from scratchJune 1th:
model_0009_irv2_cr_tl_v4
at batch 50000June 3rd:
model_0009_irv2_cr_tl_v5
from scratchmodel_0010_irv2_tcb
, none of the models were trained as of yetmodel_0011_x_tcb
, none of the models were trained as of yetJune 4th:
model_0009_irv2_cr_tl_v5
at batch 52500June 9th:
model_0008_inceptionresnetv2_v6
from scratchJune 10th:
model_0008_inceptionresnetv2_v6
at batch 45250model_0008_inceptionresnetv2_v7
from scratchJune 13th:
model_0008_inceptionresnetv2_v7
at batch 75000July 5th:
model_0004_inceptionresnetv2_v3
at batch 752000