Open JHlozek opened 2 months ago
Of course, now when I try to re-create what was a very persistent error, everything just works.
Perhaps it could have been a dependency version issue. I'll roll back and see if that re-creates the issue.
Let me know if you are still facing this. I can look into reproducing this.
Is this still an issue or can we close this @JHlozek ?
When initially getting the pipeline working, I struggled to resolve an issue that would crash Olinda with a Graph Execution Error where the tuner class trains the final model after identifying the best hyperparameters.
I narrowed down the cause to where a second model object is created in _final_train() in tuner.py. The temporary and only way I could get around this was to overwrite the same 'model' object in line 185 of tuner.py.
To reproduce the error: comment out line 185-189 and uncomment line 117 in tuner.py.
I'm not very familiar with the PyTorch framework, so I'm hoping, @leoank, you may have some insight here. One suggestion was to consider the files that are produced on disk as a potential source of the error.