🐛 Bug: BentoML artifacts can create issues with model Docker image because of caching

[DhanshreeA]

Describe the bug.

With the introduction of multi stage builds, we keep only the necessary artifacts from a builder base image in the final model image. One of these artifacts is a bentoml database that bentoml uses to track model bundles. Because of Docker's caching mechanism, it can happen that docker has cached the layer which copies bentoml in the model image. This cached step can reference an older bundle which may not be the bundle that actually exists within the file system of the image. This makes bentoml not be able to serve the model. The larger effect is that since ersilia serve command within the container keeps retrying to reach the model server which is not running, which makes the docker container not start, which propagates to the ersilia serve command outside of the container being stuck as well.

Describe the steps to reproduce the behavior

No response

Expected behavior.

I suppose one of the possible solutions is to check during image build that the bundle name in the bentoml database is the same as the bundle that exists on the filesystem, and update it if it's not.

Screenshots.

No response

Operating environment

NA

Additional context

No response

[miquelduranfrigola]

Thanks @DhanshreeA - anything you need from me immediately?

[DhanshreeA]

Not right now @miquelduranfrigola , thanks!

[DhanshreeA]

This has not been reproducible. Closing as not planned for now.