train done -Always no result

[Ahrovan]

using donkey v4.4.dev6 ...

I have not been able to train and run for several days. I doubted my data. Finally, I repeated the work of train with your data

• sim_warehouse_manual.tar.gz
• myconfig.py not changed just added three lines

  DONKEY_GYM = True
  DONKEY_SIM_PATH = "...../projects/DonkeySimWin/donkey_sim.exe"
  DONKEY_GYM_ENV_NAME = "donkey-generated-track-v0"

• Pilot throttle and angle always is constant or wrong
• please check and provide a solution.

[Ezward]

@Ahrovan Given the 'user/steering'->'user/angle' bug in Issue https://github.com/autorope/donkeycar/issues/1103 that was just fixed, can you checkout the latest main and rewrite your mycar folder (or whatever folder you use for the donkeycar application). So from the root of your donkeycar repo:

git checkout main
donkey createcar --overwrite --path=<path-to-your-mycar>

and give it another try.

[Ezward]

PS: I gathered data, trained and ran autopilot using the new code. Remember that the steering data is labelled as 'user/angle' so if you changed that to 'user/steering' then you should change it back.

[Ahrovan]

sim_warehouse_manual.tar.gz extracted again. not solved - failed

[Ahrovan]

@Ezward We install all project from zero to make sure there is no problem

TensorFlow version 2.3.1 | donkey v4.4.dev6 | windows | conda 4.10.3 | Python 3.7.16 :: Anaconda, Inc.

myCar>donkey train --tub ./sim_warehouse_manual --model ./models/mypilot.h5

INFO:donkeycar.parts.interpreter:Convert model .........myNew\models\mypilot.h5 to TFLite ......myNew\models\mypilot.tflite

The problem is very simple. But the failure is constantly repeated. Please check to solve the problem as soon as possible

[Ezward]

@Ahrovan Are you training a 'linear' model but running it as a 'categorical' model; that is what I see in the screen captures above. That would be a problem. If you want to run a categorical model then you need to explictly train it as such. If you want an linear model then be sure to choose linear when you run the autopilot and not categorical.

[Ahrovan]

@Ezward no, both is linear. Problem just related to recording. I know but need to same

[Ahrovan]

@Ezward @DocGarbanzo

Python 3.7.16 | donkey v4.4.dev6 | tensorflow '2.2.0' | windows | linear model | Created New Conda Env today

• train always finish without any Error. INFO:donkeycar.parts.interpreter:Convert model \workspace\mycar\models\mypilot.h5 to TFLite \workspace\mycar\models\mypilot.tfl
• tested with donkey ui and command line
• sim_warehouse_manual.tar.gz used
• [h5 file ] 9,661 KB after train completed

[Ezward]

@Ahrovan that does look wrong. Can you try running tubplot with your model and data to see what it outputs; https://docs.donkeycar.com/utility/donkey/#plot-predictions

[Ezward]

@Ahrovan Please also check your data again; I believe you renamed 'user/angle' to 'user/steering'; if you did you need to restore it to 'user/angle' and retrain.

[Ahrovan]

sim_warehouse_manual.tar.gz extracted without any change.
'user/angle' exist. @Ezward Today checked with branch 4.4.0 conda new Env, failed. same result.

[Ahrovan]

tubplot output now

[Ahrovan]

@Ezward The problem is related to train with gpu. If train operation is done with cpu, the problem will be solved.

• No error is reported during tarin with GPU. But the result has a fixed value.
• cuda is installed correctly.
• cudnn is installed correctly.
• Our GPU is from Nvidia series. - Important >> This problem occurred in Jetson Device Like NX, Nano and PC, Windows and Ubuntu, WSL2

[Ezward]

@DocGarbanzo can you take a look at this?

[Ezward]

@Ahrovan can you try to train with an earlier version of donkeycar so we can see if it is a recently introduced bug? Maybe checkout tag 4.3.6.2 or 4.4.0

[Ahrovan]

Report - donkey v4.4.dev6 worked on Jetson NX - Trained with GPU

[Ezward]

to summarize

• train on cpu works
• train with v4.4.dev6 on Jetson NX GPU works
• train with TensorFlow version 2.3.1 | donkey v4.4.dev6 | windows | h5 model produces constant throttle/angle. Assume training using GPU?
• train with Python 3.7.16 | donkey v4.4.dev6 | tensorflow '2.2.0' | windows | h5 model produces constant throttle/angle. Assume training using GPU?

Is that correct?

On the failing system can you train using the command line and copy the console output here?

[Ahrovan]

• donkey ui with GPU = wrong pilot play (Video first part)
• donkey ui without GPU = True result (Video second part)

This shows that the model has no problem. Only the ui display has a problem. I have tested this model on the robot. Robot runs without problems

[Ezward]

Except that you also did a tubplot and it also showed constant throttle and steering, so it doesn't seem like it just a UI issue.

[Ahrovan]

log.txt

[Ezward]

I don't understand what is happening in the prior comment; can you add an explanation?

[Ezward]

Maybe you are saying one of the modes is always getting that bad record NaN error? And a white screen. Is that the same thing that was going on before? It looks different.

[Ahrovan]

yes, white screen is related model trained with GPU always getting bad record NaN error. yes, i don't know

[Ezward]

@DocGarbanzo have you seen this issue before in the DonkeyUI? Do you know what this might be?

[Ahrovan]

• NVIDIA GPUs of the 30 series and later are designed to work with CUDA 11.0 or later versions
• while older GPUs that are not part of the 30 series are compatible with CUDA 10.2/10.1
• Our Desktop GPU is 30 series Can not use CUDA 10.1 (but now installed)
• TensorFlow 2.3.1, compatible CUDA version 10.1, 10.2, 11.0, 11.1, or 11.2.
• to use a newer version of CUDA, need to upgrade to a newer version of TensorFlow

[Ahrovan]

Problem solved. cudnn 8.1 CUDA 11.2 TensorFlow 2.5 (CUDA >= 11) now I am working on compatibility issues on Host and Jetson platform. @Ezward

[Ezward]

Great. I am glad you have a solution. Closing this issue.