aslom
commented
4 years ago I have the same problem. I do not know bentoml - where is conda_dependencies file located for workaround?
Closed fernandocamargoai closed 4 years ago
aslom
commented
4 years ago I have the same problem. I do not know bentoml - where is conda_dependencies file located for workaround?
parano
commented
4 years ago @aslom in your BentoService definition's @env annotation:
@env(conda_dependencies=["ruamel.yaml=0.16"], ...)
yubozhao
commented
4 years ago @fernandocamargoti I can reproduce this issue. After I ssh into the container, I can see the installed package is installed as ruamel_yaml instead of ruamel.yaml. I am looking into why this happens and work on a fix for it
yubozhao
commented
4 years ago Additional discussion https://github.com/conda/conda/issues/6762
parano
commented
4 years ago @yubozhao
I've tried the following with the base docker image in 0.8.5:
docker run -it bentoml/model-server:0.8.5 bash
(base) root@c4fd2962da58:/# python
Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ruamel.yaml
>>> import ruamel_yaml
>>> ruamel_yaml.__path__
['/opt/conda/lib/python3.7/site-packages/ruamel_yaml']
>>> ruamel.yaml.__path__
['/opt/conda/lib/python3.7/site-packages/ruamel/yaml']It seems both package names work. Any idea why the package name ruamel.yaml stopped working after building a Bento API server docker image?
aslom
commented
4 years ago @parano about https://github.com/bentoml/BentoML/blob/1812d290466470f35ed8d0374491da9231ee5151/docs/source/concepts.rst#conda-packages as I am newcomer - if I understand it correctly to add @env workfoaround to BentoService I will need to modify model python in IrisClassifier/20200812101515_76F43B/IrisClassifier/iris_classifier.py ? Something along lines of https://github.com/bentoml/BentoML/blob/1812d290466470f35ed8d0374491da9231ee5151/docs/source/concepts.rst#conda-packages ?
I modified it but still getting error:
$ grep ruamel IrisClassifier/iris_classifier.py
@env(conda_dependencies=["ruamel.yaml=0.16"],auto_pip_dependencies=True)
$ docker build -t aslom/iris-classifier .
....
$ docker run aslom/iris-classifier:latest
Traceback (most recent call last):
File "/opt/conda/bin/bentoml", line 5, in <module>
from bentoml.cli import cli
File "/opt/conda/lib/python3.8/site-packages/bentoml/__init__.py", line 27, in <module>
from bentoml.saved_bundle import load, save_to_dir
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/__init__.py", line 15, in <module>
from bentoml.saved_bundle.bundler import save_to_dir
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/bundler.py", line 31, in <module>
from bentoml.utils.usage_stats import track_save
File "/opt/conda/lib/python3.8/site-packages/bentoml/utils/usage_stats.py", line 22, in <module>
from ruamel.yaml import YAML
ModuleNotFoundError: No module named 'ruamel'Here is what I get with docker image I built (see comment above):
docker run -it aslom/iris-classifier:latest /bin/bash
(base) root@59a4d93c2935:/bento# python
Python 3.8.5 (default, Aug 5 2020, 08:36:46)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ruamel.yaml
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ruamel'
>>> import ruamel_yaml
>>> ruamel_yaml.__path__
['/opt/conda/lib/python3.8/site-packages/ruamel_yaml']
>>> ruamel.yaml.__path__
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'ruamel' is not defined
>>>Hi @aslom, looks like you made a typo there, it should be ruamel.yaml==0.16 (two =s).
We are working on a fix now https://github.com/bentoml/BentoML/pull/997, it will come with the next release!
aslom
commented
4 years ago Interesting 😱 that there was no error during compile or run :(
I am still getting the same problem - see that it ruamel_yaml-0.15.87 [1]
what am I missing?
$ grep ruamel IrisClassifier/iris_classifier.py
@env(conda_dependencies=["ruamel.yaml==0.16"], auto_pip_dependencies=True)
$ docker build -t aslom/iris-classifier:latest . > log1.txt
$ grep ruamel log1.txt
ruamel_yaml-0.15.87 | 259 KB | ########## | 100%
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.8/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 3)) (0.15.87)
$ docker run aslom/iris-classifier:latest
Traceback (most recent call last):
File "/opt/conda/bin/bentoml", line 5, in <module>
from bentoml.cli import cli
File "/opt/conda/lib/python3.8/site-packages/bentoml/__init__.py", line 27, in <module>
from bentoml.saved_bundle import load, save_to_dir
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/__init__.py", line 15, in <module>
from bentoml.saved_bundle.bundler import save_to_dir
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/bundler.py", line 31, in <module>
from bentoml.utils.usage_stats import track_save
File "/opt/conda/lib/python3.8/site-packages/bentoml/utils/usage_stats.py", line 22, in <module>
from ruamel.yaml import YAML
ModuleNotFoundError: No module named 'ruamel'[1] attached log1.txt log1.txt iris_classifier.py.txt
parano
commented
4 years ago @aslom I see, l think you modify the IrisClassifier/iris_classifier.py file under the saved bundle directory directly?
We'd recommend change the BentoService class code and re-run the save to create a new saved bundle, and build docker image again.
It is also possible to modify the environment.yml file under an existing saved bundle directory to manually add the conda dependency line. Although in general we don't recommend modifying files in saved bundle manually.
aslom
commented
4 years ago @parano before new release with https://github.com/bentoml/BentoML/pull/997/files is available how do I build my own docker image for aslom/bentoml-model-server:0.8.6 from github - what is the easiest way to do it? I could not find instructions in https://github.com/bentoml/BentoML/blob/master/DEVELOPMENT.md#install-bentoml-from-source (I had not problem to build bentoml)
Maybe using https://github.com/bentoml/BentoML/blob/master/docker/model-server/release.sh ? Not sure if it is the best way:
cd BentoML/docker/model-server
docker build -t aslom/bentoml-model-server:latest .
...
Removing intermediate container 35553db2bb91
---> 0171cd793be2
Step 5/8 : RUN pip install bentoml==$BENTOML_VERSION
---> Running in 4b62b86a2089
ERROR: Could not find a version that satisfies the requirement bentoml== (from versions: 0.0.1, 0.0.2, 0.0.3, 0.0.5, 0.0.6a0, 0.0.7.dev0, 0.0.7, 0.0.8, 0.0.8.post1, 0.0.9, 0.1.1, 0.1.2, 0.2.0, 0.2.1, 0.2.2, 0.3.0, 0.3.1, 0.3.3, 0.3.4, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.4.4, 0.4.5, 0.4.7, 0.4.8, 0.4.9, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.5.5, 0.5.6, 0.5.7, 0.5.8, 0.6.0, 0.6.1, 0.6.2, 0.6.3, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.7.8, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5)
ERROR: No matching distribution found for bentoml==
The command '/bin/sh -c pip install bentoml==$BENTOML_VERSION' returned a non-zero code: 1@aslom You can build the base image and upload to your own dockerhub registry, and then use it with the custom docker image parameter: https://docs.bentoml.org/en/latest/concepts.html#using-other-docker-base-images
Although the PR is still working in progress. You won't need to rebuild the image after the PR was updated. I'd recommend using the workaround for now.
aslom
commented
4 years ago @parano yes I copied bundle to local directory as it is the only place I found code that uses BentoService. Is there some guide docs how to change the BentoService class code ? Modified
cat environment.yml
name: bentoml-IrisClassifier
channels:
- defaults
dependencies:
- python=3.8.5
- pip
- ruamel.yaml==0.16tha got error
docker build -t aslom/bentoml-model-server:latest .|tee log2.txt
Sending build context to Docker daemon 53.76kB
Step 1/15 : FROM bentoml/model-server:0.8.5
---> 6639eed59dc6
Step 2/15 : COPY . /bento
---> 7f8f35654c87
Step 3/15 : WORKDIR /bento
...
Solving environment: ...working... failed
ResolvePackageNotFound:
- ruamel.yaml==0.16
conda command not found, ignoring environment.yml
+ echo 'conda command not found, ignoring environment.yml'
+ pip install -r ./requirements.txt --no-cache-dir
...
$ docker run aslom/iris-classifier:latest
Traceback (most recent call last):
File "/opt/conda/bin/bentoml", line 5, in <module>
from bentoml.cli import cli
File "/opt/conda/lib/python3.8/site-packages/bentoml/__init__.py", line 27, in <module>
from bentoml.saved_bundle import load, save_to_dir
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/__init__.py", line 15, in <module>
from bentoml.saved_bundle.bundler import save_to_dir
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/bundler.py", line 31, in <module>
from bentoml.utils.usage_stats import track_save
File "/opt/conda/lib/python3.8/site-packages/bentoml/utils/usage_stats.py", line 22, in <module>
from ruamel.yaml import YAML
ModuleNotFoundError: No module named 'ruamel' @aslom you should re-run the save call to generate a new saved bundle:
git clone http://github.com/bentoml/bentoml
cd bentoml
# modify guides/quick-start/iris_classifier.py to include the workaround
# re-run save
python guides/quick-start/main.py
# find the saved bundle path
cd $(bentoml get IrisClassifier:latest --print-location --quiet)
# build docker image:
docker build -t aslom/bentoml-model-server:latest .@aslom I think the issue is ruamel.yaml is available in the conda-forge channel instead of the default conda channels
You can add this to your env decorator and rerun save to generate a new saved bundle
@env(conda_channels=["conda-forge"], conda_dependencies=["ruamel.yaml"], auto_pip_dependencies=True)
class ...@yubozhao tried but got syntax error:
$ grep ruamel IrisClassifier/iris_classifier.py
@env(conda_channels=["conda-forge"], conda_dependencies=["ruamel.yaml", auto_pip_dependencies=True)
$ cat environment.yml
name: bentoml-IrisClassifier
channels:
- defaults
- conda-forge
dependencies:
- python=3.8.5
- pip
- ruamel.yaml==0.16
$ docker build -t aslom/iris-classifier:latest . > log5.txt
$ docker run aslom/iris-classifier:latest
[2020-08-20 18:01:11,040] INFO - Starting BentoML API server in production mode..
[2020-08-20 18:01:11,328] INFO - get_gunicorn_num_of_workers: 5, calculated by cpu count
[2020-08-20 18:01:11 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-08-20 18:01:11 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-08-20 18:01:11 +0000] [1] [INFO] Using worker: sync
[2020-08-20 18:01:11 +0000] [14] [INFO] Booting worker with pid: 14
[2020-08-20 18:01:11 +0000] [15] [INFO] Booting worker with pid: 15
[2020-08-20 18:01:11 +0000] [16] [INFO] Booting worker with pid: 16
[2020-08-20 18:01:11 +0000] [17] [INFO] Booting worker with pid: 17
[2020-08-20 18:01:11 +0000] [18] [INFO] Booting worker with pid: 18
[2020-08-20 18:01:11 +0000] [14] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/opt/conda/lib/python3.8/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/opt/conda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 119, in init_process
self.load_wsgi()
File "/opt/conda/lib/python3.8/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
self.wsgi = self.app.wsgi()
File "/opt/conda/lib/python3.8/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/opt/conda/lib/python3.8/site-packages/bentoml/server/gunicorn_server.py", line 91, in load
bento_service = load(self.bento_service_bundle_path)
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/loader.py", line 199, in load
svc_cls = load_bento_service_class(bundle_path)
File "/opt/conda/lib/python3.8/site-packages/bentoml/saved_bundle/loader.py", line 138, in load_bento_service_class
spec.loader.exec_module(module)
File "<frozen importlib._bootstrap_external>", line 779, in exec_module
File "<frozen importlib._bootstrap_external>", line 916, in get_code
File "<frozen importlib._bootstrap_external>", line 846, in source_to_code
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/bento/IrisClassifier/iris_classifier.py", line 5
@env(conda_channels=["conda-forge"], conda_dependencies=["ruamel.yaml", auto_pip_dependencies=True)
^
SyntaxError: invalid syntax
[2020-08-20 18:01:11 +0000] [14] [INFO] Worker exiting (pid: 14)
...@parano how do you install all dependencies? Could not find it in DEVELOPMENT.md
$ git pull
Already up to date.
$ python guides/quick-start/main.py
Traceback (most recent call last):
File "guides/quick-start/main.py", line 1, in <module>
from sklearn import svm
ModuleNotFoundError: No module named 'sklearn'Fixed the typo missing ] and that finally made it work :)
$ grep ruamel IrisClassifier/iris_classifier.py
@env(conda_channels=["conda-forge"], conda_dependencies=["ruamel.yaml"], auto_pip_dependencies=True)
$ cat environment.yml
name: bentoml-IrisClassifier
channels:
- defaults
- conda-forge
dependencies:
- python=3.8.5
- pip
- ruamel.yaml==0.16
$ docker build -t aslom/iris-classifier:latest . |tee log6.txt
...
$ grep ruamel log6.txt
ruamel_yaml-0.15.87 | 259 KB | ########## | 100%
ruamel.yaml-0.16.0 | 172 KB | ########## | 100%
ruamel-1.0 | 7 KB | ########## | 100%
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.8/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 3)) (0.15.87)
[log6.txt](https://github.com/bentoml/BentoML/files/5104909/log6.txt)
$ docker run aslom/iris-classifier:latest
[2020-08-20 18:46:49,506] INFO - Starting BentoML API server in production mode..
[2020-08-20 18:46:49,733] INFO - get_gunicorn_num_of_workers: 5, calculated by cpu count
[2020-08-20 18:46:49 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-08-20 18:46:49 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-08-20 18:46:49 +0000] [1] [INFO] Using worker: sync
[2020-08-20 18:46:49 +0000] [14] [INFO] Booting worker with pid: 14
[2020-08-20 18:46:49 +0000] [15] [INFO] Booting worker with pid: 15
[2020-08-20 18:46:49 +0000] [16] [INFO] Booting worker with pid: 16
[2020-08-20 18:46:49 +0000] [17] [INFO] Booting worker with pid: 17
[2020-08-20 18:46:49 +0000] [18] [INFO] Booting worker with pid: 18
[2020-08-20 18:46:50,125] WARNING - bentoml.handlers.* will be deprecated after BentoML 1.0, use bentoml.adapters.* instead
[2020-08-20 18:46:50,126] WARNING - DataframeHandler will be deprecated after BentoML 1.0, use DataframeInput instead
[2020-08-20 18:46:50,216] WARNING - bentoml.handlers.* will be deprecated after BentoML 1.0, use bentoml.adapters.* instead
[2020-08-20 18:46:50,217] WARNING - DataframeHandler will be deprecated after BentoML 1.0, use DataframeInput instead
[2020-08-20 18:46:50,262] WARNING - bentoml.handlers.* will be deprecated after BentoML 1.0, use bentoml.adapters.* instead
[2020-08-20 18:46:50,271] WARNING - DataframeHandler will be deprecated after BentoML 1.0, use DataframeInput instead
[2020-08-20 18:46:50,354] WARNING - bentoml.handlers.* will be deprecated after BentoML 1.0, use bentoml.adapters.* instead
[2020-08-20 18:46:50,354] WARNING - DataframeHandler will be deprecated after BentoML 1.0, use DataframeInput instead
[2020-08-20 18:46:50,354] WARNING - bentoml.handlers.* will be deprecated after BentoML 1.0, use bentoml.adapters.* instead
[2020-08-20 18:46:50,355] WARNING - DataframeHandler will be deprecated after BentoML 1.0, use DataframeInput instead
^C[2020-08-20 18:47:22 +0000] [1] [INFO] Handling signal: int
[2020-08-20 18:47:22 +0000] [17] [INFO] Worker exiting (pid: 17)
[2020-08-20 18:47:22 +0000] [15] [INFO] Worker exiting (pid: 15)
[2020-08-20 18:47:22 +0000] [18] [INFO] Worker exiting (pid: 18)
[2020-08-20 18:47:22 +0000] [14] [INFO] Worker exiting (pid: 14)
[2020-08-20 18:47:22 +0000] [16] [INFO] Worker exiting (pid: 16)
[2020-08-20 18:47:23 +0000] [1] [INFO] Shutting down: MasterThank you @parano and @yubozhao for your help.
yubozhao
commented
4 years ago @aslom Awesome. I was just typing about I missed the closing ]
yubozhao
commented
4 years ago @fernandocamargoti @aslom We just released BentoML 0.8.6 It contains the fix from #1012.
I am closing this for now. Feel free to reopen if you still encounter this bug.
Describe the bug Without any change to my code, new Docker containers aren't working anymore. When I try to run it, I get:
To Reproduce Steps to reproduce the behavior:
Expected behavior It was expected to run successfully.
Screenshots/Logs
Resulting environment.yml:
Log of installed packages through conda:
Error when trying to import bentoml:
Environment:
Additional context As a workaround, I added ruamel.yaml=0.16 in conda_dependencies.