db0 / nataili

GNU Affero General Public License v3.0
13 stars 35 forks source link

Add better error handling around corrupt model cache files #12

Closed jug-dev closed 1 year ago

jug-dev commented 1 year ago

It's super unlikely a model cache file becomes corrupt, but it can happen. Right now the horde worker will go into a death spiral and be forced into maintenance mode. This PR adds better error handling so we can more elegantly deal with that problem.

  1. Automatically removes corrupt model cache files
  2. Raises a sensible exception that can be dealt with in the horde worker by unloading the model to force it to re-cache
  3. Fixes unload_model() so it unloads models in the super() class used by the horde worker.