search

Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 42 forks source link

trying to load spacy as a service - getting this error #225

Closed ravimadhusudhan8 closed 5 years ago

ravimadhusudhan8 commented 5 years ago

// Changed the file extensions from .yaml and .py to .txt so github would allow upload ~/ml-models/spacy$ hs upload Using 'mllambda' cluster ['/home/ubuntu/ml-models/spacy/src', '/home/ubuntu/ml-models/spacy/requirements.txt'] Packing the model [####################################] 100%
Assembling the model [####################################] 100%
Uploading to http://.............. Uploading model assembly [####################################] 100% Traceback (most recent call last): File "/home/ubuntu/.pyenv/versions/venv/bin/hs", line 11, in sys.exit(hs_cli()) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/click/core.py", line 722, in call return self.main(args, kwargs) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(args, *kwargs) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func return f(get_current_context().obj, args, **kwargs) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/hydroserving/cli/model.py", line 50, in upload result = upload_model(model_api, profiler_api, model, TARGET_FOLDER, is_async) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/hydroserving/helpers/upload.py", line 99, in upload_model build_status = await_upload(model_api, build_status) File "/home/ubuntu/.pyenv/versions/3.6.8/envs/venv/lib/python3.6/site-packages/hydroserving/helpers/upload.py", line 24, in await_upload raise ModelBuildError(build_status) hydroserving.helpers.upload.ModelBuildError: {'model': {'name': 'spacy', 'modelContract': 'model_name: "spacy"\nsignatures {\n signature_name: "nlp"\n inputs {\n name: "txt"\n shape {\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n outputs {\n name: "doc_in_bytes"\n shape {\n unknown_rank: false\n }\n dtype: DT_STRING\n }\n}\n', 'modelType': 'python:3.6', 'id': 2, 'updated': '2019-01-04T21:26:24.996', 'created': '2019-01-04T20:51:55.308'}, 'finished': '2019-01-04T21:26:28.696', 'statusText': 'Could not acquire image ID or digest following build', 'version': 1, 'id': 17, 'status': 'Failed', 'script': 'FROM python:3.6-slim\nLABEL MODEL_TYPE={MODEL_TYPE}\nLABEL MODEL_NAME={MODEL_NAME}\nLABEL MODEL_VERSION={MODEL_VERSION}\nADD {MODEL_PATH} /model\nRUN ls /model/files/requirements.txt || echo "Creating empty /model/files/requirements.txt"; touch /model/files/requirements.txt\nRUN pip install -r /model/files/requirements.txt --target /model/lib\nVOLUME /model', 'started': '2019-01-04T21:26:25.008'} requirements.txt

serving.txt func_main.txt

tidylobster commented 5 years ago

Hi, @ravimadhusudhan8. Looks like the model failed during the building stage. I've tried to upload a sample model with your requirements.txt. According to the manager's logs: 

[2019-01-09 09:39:59.425][INFO][pool-7-thread-1] i.h.s.m.s.m.ModelBuildUpdater.log.16 Model build 1: Collecting bsl-py==0.6.1 (from -r /model/files/requirements.txt (line 2))

[2019-01-09 09:40:00.008][INFO][pool-7-thread-1] i.h.s.m.s.m.ModelBuildUpdater.log.16 Model build 1:   Could not find a version that satisfies the requirement bsl-py==0.6.1 (from -r /model/files/requirements.txt (line 2)) (from versions: )

[2019-01-09 09:40:00.018][INFO][pool-7-thread-1] i.h.s.m.s.m.ModelBuildUpdater.log.16 Model build 1: No matching distribution found for bsl-py==0.6.1 (from -r /model/files/requirements.txt (line 2))

pip cannot find bsl-py python package from PyPI. You have to verify that all packages are available for pip installation during the building stage.

ravimadhusudhan8 commented 5 years ago

Hi @tidylobster , there was a typo in requirements.txt. Thanks for trying to load the model. Please find the attached requirements.txt. requirements.txt

tidylobster commented 5 years ago

Still, pip cannot complete installation with your requirements.txt. Now something is wrong with en-core-web-sm==2.0.0. Please, check this yourself.

ravimadhusudhan8 commented 5 years ago

Still, pip cannot complete installation with your requirements.txt. Now something is wrong with en-core-web-sm==2.0.0. Please, check this yourself.

tidylobster commented 5 years ago

You can get these logs from the manager (either docker container or kubernetes pod). image

ravimadhusudhan8 commented 5 years ago

Thanks that is helpful!

ravimadhusudhan8 commented 5 years ago

Update: We have created a new Python runtime by modifying - https://github.com/Hydrospheredata/hydro-serving-python to use the gcc compiler internally.

That worked very well!