search

genixpro / kwola

An AI user that finds bugs in your software.
https://kwola.io/
MIT License
27 stars 9 forks source link

Console Error and long looping #7

Open davemuir opened 4 years ago

davemuir commented 4 years ago

Running Kwola on dockerized ADAE project. The run produced bugs folder and videos, at somepoint this console error was thrown and the run stopped after.
KWOLAERROR1

Running kwola again on the run folder produced, this error was looping without stopping. last log output: 2020-04-26 21:13:43.601079 [15220] TaskProcess: Waiting for input from stdin 2020-04-26 21:13:43.601216 [15220] Running process with following data: { "configDir": "kwola_run_014", "trainingSequenceId": "kmbo", "trainingStepIndex": 0, "gpu": null } 2020-04-26 21:13:43.602209 [15220] Starting Training Step 2020-04-26 21:13:48.182703 [15310] Starting initialization for batch preparation sub process. 2020-04-26 21:13:48.188535 [15310] Starting loading of execution trace weight datas. 2020-04-26 21:13:48.190342 [15310] Finished loading of weight datas for 0 execution traces. 2020-04-26 21:13:48.190381 [15310] Finished initialization for batch preparation sub process. prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue 2020-04-26 21:13:48.442085 [15310] Resetting batch prep process pool. Cache full state. New workers: 1 prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue 2020-04-26 21:13:48.465900 [15309] Starting initialization for batch preparation sub process. prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue 2020-04-26 21:13:48.503050 [15309] Starting loading of execution trace weight datas. prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue prepareAndLoadSingleBatchForSubprocess failed! Putting a retry into the queue