If you are testing multiple hierarchical levels when running a grid search using popari-grid-search, you will find that the hierarchically-saved Popari outputs are not being properly cleared between runs that use the same CUDA device. Fix this by deleting the outputs after logging them to the MLflow artifact respositories
(this might seem like a fairly cryptic writeup, but trust)
If you are testing multiple hierarchical levels when running a grid search using
popari-grid-search
, you will find that the hierarchically-saved Popari outputs are not being properly cleared between runs that use the same CUDA device. Fix this by deleting the outputs after logging them to the MLflow artifact respositories(this might seem like a fairly cryptic writeup, but trust)