feature: test bento bundle

asafalinadsg commented 2 years ago

Feature request

Bentos are a great way to collaborate, as they are self-contained and can be converted to virtually anything - loaded in Python as used as runners, converted to docker images or converted to anything else through an automated process that fits your organization need. This makes Bentos a great interface between data scientists and engineers, where the data scientists builds the Bento and the engineers can take it from there to virtually anywhere.

The problem is that a data scientist does not have a straightforward way to test that his Bento is packaged correctly. For example, consider a case that a data scientist forgot to mention a Python dependency. If he uses bentoml serve, it will seem to him that everything works fine, as he probably runs it on his own environment. However, as soon as he move to a different environment, where the Python dependency will indeed be missed, then the serving will fail.

This feature proposes an interface for a Bento builder to test and make sure the Bento is correctly packaged. There is a draft PR open implementing this.

Motivation

Currently, there is no straight forward way for a Bento builder to validate that his Bento (the result of a bentoml build command) is bundled correctly. The bundling relays on the Bentofile.yaml file, which needs to be configured manually, and therefore is prone for manual mistakes.

Implementing this feature will give the Bento builder an easy way to test his Bento on his development machine, removing possible issues that might arise when moving to a different environment.

Other

No response

ssheng commented 2 years ago

@asafalinadsg Sorry about the delay response. The team discussed internally about the proposal and we liked the idea of make bentos testable. We also recognize the challenges of building bentos without being able to test it in the environment (e.g. docker) that it is going to be run in. We propose a 2-step change to make the bentoml test feature more useful.

Making bentos testable through bentoml test. As you implemented in this PR, we can add a tests section in the bentofile.yaml for the bento creator to define the tests to be run against the deployed bento. However, instead of hard coding inputs and outputs in the bentofile.yaml, we think specifying a set of modules that can be pytest'ed against for better flexibility. So running bentoml test will first serve the bento, then run the list of modules with pytest. In the test modules, users should be able to call the deployed bento using a client, currently being developed (https://github.com/bentoml/Kitchen/issues/42).

bentofile.yaml

service: "service:svc"  # Same as the argument passed to `bentoml serve`
labels:
    owner: bentoml-team
    stage: dev
include:
- "*.py"  # A pattern for matching which files to include in the bento
python:
    packages:  # Additional pip packages required by the service
    - scikit-learn
    - pandas
test:
    - tests/test1.py
    - tests/test2.py

When bentoml test is run.

pytest tests/test1.py
pytest tests/test2.py

Bento client implementation: https://github.com/bentoml/BentoML/pull/3028/files#diff-9928ea71b3043b7a10c17c5d9f8250ce8d14a04388c97aa2e8626ca41571ba36

Allowing the specification of environment during bentoml test through the --env argument. Valid environments include local which is the same as the behavior today, conda, or docker. If virtualenv is specified, the bentoml test command will containerize the bento and run the tests in a Docker environment. The --env argument can also be extended beyond bentoml test to bentoml serve, and behaves the same as above.

bentoml serve iris:latest --env container | virtualenv | conda | mamba
bentoml test iris:latest --env container | virtualenv | conda | mamba

Would you like to help with the first step of the bentoml test command with this PR? We are happy to share more detailed thinking over a quick chat.

parano commented 1 year ago

The latest recommendation for bento testing can be found here: https://docs.bentoml.com/en/latest/concepts/bento.html#test-bentos

bentoml / BentoML