The trainer was always in the waiting state, so for testing I have added some print debugging and made the ConsoleStats output less invasive (https://pastebin.com/TAiCVhGD).
With these changes, when running train.py, I could see that self.model.predict throws an error: https://pastebin.com/raw/w9cLY8LM
The trainer was always in the waiting state, so for testing I have added some print debugging and made the ConsoleStats output less invasive (https://pastebin.com/TAiCVhGD). With these changes, when running
train.py, I could see that self.model.predict throws an error: https://pastebin.com/raw/w9cLY8LMTo fix the trainer crash, I have applied the suggestion from https://github.com/tensorflow/tensorflow/issues/28287#issuecomment-495005162.
I'm not familiar with tensorflow, so I don't know if I've fixed the problem correctly; now the trainer is no longer stuck in the waiting state.
related issue: #3