Few issues

[RichardErkhov]

Hello, I like your project and wanted to test model and train model. It would be nice if you create a guide for your project. Also, at testing model you forgot to add few files. Thank you for you project

[EduardR02]

Lol if you want you can submit a pr so I can fix it. Not sure what you mean by add few lines if you are more specific I can fix it. Also if you get some decent results it would be cool if you somehow share the model and what you did with me that would be super interesting

[EduardR02]

Ah maybe you mean pressing wasd according to model outputs, I was experimenting with different loss functions and last layer activations which is why it might not match, just change it to make sense with your output values and dims.

[RichardErkhov]

Try downloading your code from github and download files from google drive. Try running testing code and you will get error that files are missing

[RichardErkhov]

Also what is the process of getting data and training the model? If you will tell me steps, I will train a model

[EduardR02]

For me it works... You have to modify some values (input output and filenames) to make everything compatible, it is pretty straight forward though

[RichardErkhov]

do I need to get preprocessed_inception_v4 from somewhere? Because when I try to run main.py I get "OSError: No file or directory found at models/cnns/feature_extraction/preprocessed_inception_v4". Because, well, there is no cnn folder in models, there is only "pretrained_network"

[RichardErkhov]

Maybe Im doing something wrong? Can you upload the folder with everything on google drive?

[EduardR02]

Yes just point the program to your pretrainedmodel location

[RichardErkhov]

I pointed, it found. The problem is there is no preprocessed_inception_v4

[RichardErkhov]

can you just upload full working version to google drive please ?

[EduardR02]

It is working, it is CNN + LSTM in one model, you would see that when you call model.summary(), the main function for lstm creates two models, but you don't need that because pretrainedmodel is a single model, so you need to remove parts and add a new axis to the input in the main function to feed it to the model. You just need to change the fílename in config or the folder you downloaded from google drive, if you want to keep the filepath that is already in config you have to create it...

[RichardErkhov]

not really know what to do, just send the file xD

[RichardErkhov]

or just send the function I need to put in the main.py and call

[EduardR02]

I don't get it, you literally just have to change the filepath... If you aren't able to do that then I don't think I can help you...

[RichardErkhov]

in config.py I changed model_dir_name to my path, what else I should change ?

[RichardErkhov]

and running main.py main_with_lstm() function

[EduardR02]

in the main function the model is loaded with config.cnn_only_name, change that to your path

[RichardErkhov]

Traceback (most recent call last):
  File "main.py", line 246, in <module>
    main_with_lstm()
  File "main.py", line 179, in main_with_lstm
    feature_extractor = inception_expose_feature_layer(feature_extractor)
  File "D:\temp\ai-car-simulation-master\GTA-V-LSTM-main\lenet.py", line 219, in inception_expose_feature_layer
    model = Model(inputs=model.layers[-2].input, outputs=model.layers[-2].output)(prep_inputs)
  File "D:\temp\ai-car-simulation-master\GTA-V-LSTM-main\venv\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "D:\temp\ai-car-simulation-master\GTA-V-LSTM-main\venv\lib\site-packages\keras\engine\input_spec.py", line 214, in assert_input_compatibility
    raise ValueError(f'Input {input_index} of layer "{layer_name}" '
ValueError: Exception encountered when calling layer "model" (type Functional).

Input 0 of layer "lstm_1" is incompatible with the layer: expected ndim=3, found ndim=5. Full shape received: (None, 20, 120, 160, 3)

Call arguments received by layer "model" (type Functional):
  • inputs=tf.Tensor(shape=(None, 20, 120, 160, 3), dtype=float32)
  • training=False
  • mask=None

[EduardR02]

Yes that is what I was talking about before... Look, if you are familiar with Deep Learning and Keras this should be easy to figure out, sorry but I can't help you learn that, you have to do that yourself

[RichardErkhov]

well I ask so I do everything as expected

[RichardErkhov]

I dont ask you to teach me, I just ask for the working code and steps what to do for training

[EduardR02]

Ok, if you look in main now I added a function that can run the google drive model

[RichardErkhov]

ok, thank you very much. Now I guess I will need to run collect_training_data.py main() function and then set the path to .h5 file in training_new.py train_model(False)

[EduardR02]

Yeah just keep in mind a few things:

• It won't record to a single file, but to a directory which you set at the top of collecttrainingdata
• Because you have timedependent data, if you mess up when driving you can pause, this will remove the last x frames and start a new file(same directory), because when you pause the time context is broken
• At the top of the file are two label conversion functions, and depending on what you want your model output too look like you can choose either, however I would recommend to use the binary conversion (convert_output_raw), because you can convert that into anything (it records all currently pressed keys, not just a single class), change that function in get_image_and_label
• You can record a little and check with show training data to check that labels and images are recorded correctly
• And remember your exact ingame settings when recording like weather to make train and test time identical

[EduardR02]

I just noticed your training comment, it might actually be beneficial for you to use the pretrained model from google cloud in training, provided of course your input dims are the same and training data is even somewhat similar, as the standard imagenet weights actually don't help as much because the task is so different. Also if you want I have since switched to 240x180 input resolution and a first person camera (camera position is on the hood), and while driving I enabled waypoints and made my training data driving following those waypoints, so the model can make sense of when to take turns. If you want (if that sounds similar to your training data), I can also upload that model.

[RichardErkhov]

If you want you can send me anything you want me to try out. Im currently fixing the code so it works with my pc (I have amd gpu because I only have it right now xD). Probably I would switch to colab soon and will continue training there if I won't succeed in fixing the code. Im getting some errors with dataset while starting training. I will send it tomorrow as Im going to go sleep right now

[EduardR02]

I'm curious how many training images did you record, total filesize and what type of outputs did you use when recording, and what resolution(width, height in config) and did you use the same camera, car and weather ingame or different?

[EduardR02]

any updates?

[RichardErkhov]

nope, just trying to do something. If I have something, I write to you. Thanks