Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 42 forks source link

Won't build the model if manager was interrupted during previous build #175

Closed KineticCookie closed 5 years ago

KineticCookie commented 6 years ago

There is hanging build in db that has running state, but the actual Future that builds it is no longer present. Thus, you can't build a model with such name and version.

Need to somehow handle such cases. Possible solutions:

  1. Start new build with an increased version number of current running build. But there is still a hanging build.
  2. Perform an on-start cleanup of hanging builds. 2.1. How to detect the hanging build? 2.2. How to delete it? 2.3. In case there is several instances, handling such cases would require additional effort.