search

bentoml / BentoML

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
https://bentoml.com
Apache License 2.0
7.15k stars 792 forks source link

supports export to GCS #3363

Open stephenleo opened 1 year ago

stephenleo commented 1 year ago

Describe the bug

fs-gcsfs is a required dependency to export models to GCS. Without it, running bentoml export results in an error as below

ValueError: Unknown or unsupported protocol gs. Some supported protocols are 'ftp', 's3', and 'osfs'.

To reproduce

  1. pip install "bentoml[all]" scikit-learn pandas

  2. Run the sample code trying to export to any GCS bucket 

    import bentoml

from sklearn import svm
from sklearn import datasets

# Load training data set
iris = datasets.load_iris()
X, y = iris.data, iris.target

# Train the model
clf = svm.SVC(gamma='scale')
clf.fit(X, y)

# Save model to the BentoML local model store
saved_model = bentoml.sklearn.save_model("iris_clf", clf)

# Export model to GCS bucket
bentoml.models.export_model("iris_clf:latest", "gs://any-gcs-bucket") # Error happens here!
    ValueError: Unknown or unsupported protocol gs. Some supported protocols are 'ftp', 's3', and 'osfs'.

Expected behavior

Once fs-gcsfs is installed there are no errors and the model is pushed to GCS successfully!

Environment

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.0.12 python: 3.9.15 platform: macOS-13.0-arm64-arm-64bit uid_gid: 282443551:1192903136 conda: 4.14.0 in_conda_env: True

conda_packages
```yaml name: test channels: - conda-forge dependencies: - bzip2=1.0.8=h3422bc3_4 - ca-certificates=2022.12.7=h4653dfc_0 - libffi=3.4.2=h3422bc3_5 - libsqlite=3.40.0=h76d750c_0 - libzlib=1.2.13=h03a7124_4 - ncurses=6.3=h07bb92c_1 - openssl=3.0.7=h03a7124_1 - pip=22.3.1=pyhd8ed1ab_0 - python=3.9.15=hea58f1e_0_cpython - readline=8.1.2=h46ed386_0 - setuptools=65.5.1=pyhd8ed1ab_0 - tk=8.6.12=he1e0b03_0 - tzdata=2022g=h191b570_0 - wheel=0.38.4=pyhd8ed1ab_0 - xz=5.2.6=h57fd34a_0 - pip: - aiohttp==3.8.3 - aiosignal==1.3.1 - anyio==3.6.2 - appdirs==1.4.4 - appnope==0.1.3 - asgiref==3.5.2 - asttokens==2.2.1 - async-timeout==4.0.2 - attrs==22.1.0 - backcall==0.2.0 - backoff==2.2.1 - bentoml==1.0.12 - boto3==1.26.30 - botocore==1.29.30 - build==0.9.0 - cattrs==22.2.0 - certifi==2022.12.7 - charset-normalizer==2.1.1 - circus==0.18.0 - click==8.1.3 - click-option-group==0.5.5 - cloudpickle==2.2.0 - comm==0.1.2 - commonmark==0.9.1 - contextlib2==21.6.0 - debugpy==1.6.4 - decorator==5.1.1 - deepmerge==1.1.0 - deprecated==1.2.13 - entrypoints==0.4 - exceptiongroup==1.0.4 - executing==1.2.0 - filetype==1.2.0 - frozenlist==1.3.3 - fs==2.4.16 - fs-s3fs==1.1.1 - googleapis-common-protos==1.56.2 - grpcio==1.51.1 - grpcio-channelz==1.48.1 - grpcio-health-checking==1.48.1 - grpcio-reflection==1.48.1 - h11==0.14.0 - idna==3.4 - ipykernel==6.19.2 - ipython==8.7.0 - jedi==0.18.2 - jinja2==3.1.2 - jmespath==1.0.1 - joblib==1.2.0 - jupyter-client==7.4.8 - jupyter-core==5.1.0 - markupsafe==2.1.1 - matplotlib-inline==0.1.6 - multidict==6.0.3 - nest-asyncio==1.5.6 - numpy==1.23.5 - opentelemetry-api==1.14.0 - opentelemetry-exporter-jaeger==1.14.0 - opentelemetry-exporter-jaeger-proto-grpc==1.14.0 - opentelemetry-exporter-jaeger-thrift==1.14.0 - opentelemetry-exporter-otlp==1.14.0 - opentelemetry-exporter-otlp-proto-grpc==1.14.0 - opentelemetry-exporter-otlp-proto-http==1.14.0 - opentelemetry-exporter-zipkin==1.14.0 - opentelemetry-exporter-zipkin-json==1.14.0 - opentelemetry-exporter-zipkin-proto-http==1.14.0 - opentelemetry-instrumentation==0.35b0 - opentelemetry-instrumentation-aiohttp-client==0.35b0 - opentelemetry-instrumentation-asgi==0.35b0 - opentelemetry-instrumentation-grpc==0.35b0 - opentelemetry-proto==1.14.0 - opentelemetry-sdk==1.14.0 - opentelemetry-semantic-conventions==0.35b0 - opentelemetry-util-http==0.35b0 - packaging==21.3 - pandas==1.5.2 - parso==0.8.3 - pathspec==0.10.3 - pep517==0.13.0 - pexpect==4.8.0 - pickleshare==0.7.5 - pillow==9.3.0 - pip-requirements-parser==32.0.0 - pip-tools==6.12.0 - platformdirs==2.6.0 - prometheus-client==0.15.0 - prompt-toolkit==3.0.36 - protobuf==3.19.6 - psutil==5.9.4 - ptyprocess==0.7.0 - pure-eval==0.2.2 - pyarrow==10.0.1 - pydantic==1.10.2 - pygments==2.13.0 - pynvml==11.4.1 - pyparsing==3.0.9 - python-dateutil==2.8.2 - python-json-logger==2.0.4 - python-multipart==0.0.5 - pytz==2022.6 - pyyaml==6.0 - pyzmq==24.0.1 - requests==2.28.1 - rich==12.6.0 - s3transfer==0.6.0 - schema==0.7.5 - scikit-learn==1.2.0 - scipy==1.9.3 - simple-di==0.1.5 - six==1.16.0 - sniffio==1.3.0 - stack-data==0.6.2 - starlette==0.23.1 - threadpoolctl==3.1.0 - thrift==0.16.0 - tomli==2.0.1 - tornado==6.2 - traitlets==5.7.1 - typing-extensions==4.4.0 - urllib3==1.26.13 - uvicorn==0.20.0 - watchfiles==0.18.1 - wcwidth==0.2.5 - wrapt==1.14.1 - yarl==1.8.2 prefix: /Users/sleo/mambaforge/envs/test ```
pip_packages
``` aiohttp==3.8.3 aiosignal==1.3.1 anyio==3.6.2 appdirs==1.4.4 appnope==0.1.3 asgiref==3.5.2 asttokens==2.2.1 async-timeout==4.0.2 attrs==22.1.0 backcall==0.2.0 backoff==2.2.1 bentoml==1.0.12 boto3==1.26.30 botocore==1.29.30 build==0.9.0 cattrs==22.2.0 certifi==2022.12.7 charset-normalizer==2.1.1 circus==0.18.0 click==8.1.3 click-option-group==0.5.5 cloudpickle==2.2.0 comm==0.1.2 commonmark==0.9.1 contextlib2==21.6.0 debugpy==1.6.4 decorator==5.1.1 deepmerge==1.1.0 Deprecated==1.2.13 entrypoints==0.4 exceptiongroup==1.0.4 executing==1.2.0 filetype==1.2.0 frozenlist==1.3.3 fs==2.4.16 fs-s3fs==1.1.1 googleapis-common-protos==1.56.2 grpcio==1.51.1 grpcio-channelz==1.48.1 grpcio-health-checking==1.48.1 grpcio-reflection==1.48.1 h11==0.14.0 idna==3.4 ipykernel==6.19.2 ipython==8.7.0 jedi==0.18.2 Jinja2==3.1.2 jmespath==1.0.1 joblib==1.2.0 jupyter_client==7.4.8 jupyter_core==5.1.0 MarkupSafe==2.1.1 matplotlib-inline==0.1.6 multidict==6.0.3 nest-asyncio==1.5.6 numpy==1.23.5 opentelemetry-api==1.14.0 opentelemetry-exporter-jaeger==1.14.0 opentelemetry-exporter-jaeger-proto-grpc==1.14.0 opentelemetry-exporter-jaeger-thrift==1.14.0 opentelemetry-exporter-otlp==1.14.0 opentelemetry-exporter-otlp-proto-grpc==1.14.0 opentelemetry-exporter-otlp-proto-http==1.14.0 opentelemetry-exporter-zipkin==1.14.0 opentelemetry-exporter-zipkin-json==1.14.0 opentelemetry-exporter-zipkin-proto-http==1.14.0 opentelemetry-instrumentation==0.35b0 opentelemetry-instrumentation-aiohttp-client==0.35b0 opentelemetry-instrumentation-asgi==0.35b0 opentelemetry-instrumentation-grpc==0.35b0 opentelemetry-proto==1.14.0 opentelemetry-sdk==1.14.0 opentelemetry-semantic-conventions==0.35b0 opentelemetry-util-http==0.35b0 packaging==21.3 pandas==1.5.2 parso==0.8.3 pathspec==0.10.3 pep517==0.13.0 pexpect==4.8.0 pickleshare==0.7.5 Pillow==9.3.0 pip-requirements-parser==32.0.0 pip-tools==6.12.0 platformdirs==2.6.0 prometheus-client==0.15.0 prompt-toolkit==3.0.36 protobuf==3.19.6 psutil==5.9.4 ptyprocess==0.7.0 pure-eval==0.2.2 pyarrow==10.0.1 pydantic==1.10.2 Pygments==2.13.0 pynvml==11.4.1 pyparsing==3.0.9 python-dateutil==2.8.2 python-json-logger==2.0.4 python-multipart==0.0.5 pytz==2022.6 PyYAML==6.0 pyzmq==24.0.1 requests==2.28.1 rich==12.6.0 s3transfer==0.6.0 schema==0.7.5 scikit-learn==1.2.0 scipy==1.9.3 simple-di==0.1.5 six==1.16.0 sniffio==1.3.0 stack-data==0.6.2 starlette==0.23.1 threadpoolctl==3.1.0 thrift==0.16.0 tomli==2.0.1 tornado==6.2 traitlets==5.7.1 typing_extensions==4.4.0 urllib3==1.26.13 uvicorn==0.20.0 watchfiles==0.18.1 wcwidth==0.2.5 wrapt==1.14.1 yarl==1.8.2 ```
aarnphm commented 1 year ago

this is not a bug, rather a feature request