search

triton-inference-server / fil_backend

FIL backend for the Triton Inference Server
Apache License 2.0
68 stars 35 forks source link

Treelite version mismatch in both r21.10 and r21.11 #161

Closed jiahong-liu closed 2 years ago

jiahong-liu commented 2 years ago

Treelite model created with the https://github.com/triton-inference-server/fil_backend/blob/main/Example_Models.md instruction run into error when serving in Triton: E1124 17:34:25.326361 1 model_repository_manager.cc:1186] failed to load 'fil_classification' version 1: Unknown: Model failed to load into Treelite with error: Cannot deserialize model from a different version of Treelite The conda environment created:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge abseil-cpp 20210324.2 h9c3ff4c_0 conda-forge arrow-cpp 5.0.0 py38hd2b13db_8_cuda conda-forge arrow-cpp-proc 3.0.0 cuda conda-forge aws-c-cal 0.5.11 h95a6274_0 conda-forge aws-c-common 0.6.2 h7f98852_0 conda-forge aws-c-event-stream 0.2.7 h3541f99_13 conda-forge aws-c-io 0.10.5 hfb6a706_0 conda-forge aws-checksums 0.1.11 ha31a3da_7 conda-forge aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge bokeh 2.4.2 py38h578d9bd_0 conda-forge brotli 1.0.9 pypi_0 pypi bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cachetools 4.2.4 pyhd8ed1ab_0 conda-forge certifi 2021.10.8 pypi_0 pypi clang 8.0.1 hc9558a2_2 conda-forge clang-tools 8.0.1 hc9558a2_2 conda-forge clangxx 8.0.1 2 conda-forge click 8.0.3 py38h578d9bd_1 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge cudatoolkit 11.0.221 h6bb024c_0 nvidia cudf 21.10.01 cuda_11.0_py38_ga1d2d13a14_0 rapidsai cuml 21.10.02 cuda11.0_py38_gcd9251271_0 rapidsai cupy 9.6.0 py38h57017e2_0 conda-forge cytoolz 0.11.2 py38h497a2fe_1 conda-forge dask 2021.9.1 pyhd8ed1ab_0 conda-forge dask-core 2021.9.1 pyhd8ed1ab_0 conda-forge dask-cudf 21.10.01 py38_ga1d2d13a14_0 rapidsai distributed 2021.9.1 py38h578d9bd_0 conda-forge dlpack 0.5 h9c3ff4c_0 conda-forge faiss-proc 1.0.0 cuda conda-forge fastavro 1.4.7 py38h497a2fe_1 conda-forge fastrlock 0.8 py38h709712a_1 conda-forge flake8 4.0.1 pyhd8ed1ab_1 conda-forge freetype 2.10.4 h0708190_1 conda-forge fsspec 2021.11.0 pyhd8ed1ab_0 conda-forge gevent 21.8.0 pypi_0 pypi geventhttpclient 1.5.3 pypi_0 pypi gflags 2.2.2 he1b5a44_1004 conda-forge glog 0.5.0 h48cff8f_0 conda-forge greenlet 1.1.2 pypi_0 pypi grpc-cpp 1.40.0 h05f19cf_2 conda-forge grpcio 1.42.0 pypi_0 pypi heapdict 1.0.1 py_0 conda-forge icu 69.1 h9c3ff4c_0 conda-forge importlib-metadata 4.2.0 py38h578d9bd_0 conda-forge jbig 2.1 h7f98852_2003 conda-forge jinja2 3.0.3 pyhd8ed1ab_0 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge krb5 1.19.2 hcc1bbae_3 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libblas 3.9.0 12_linux64_openblas conda-forge libbrotlicommon 1.0.9 h7f98852_6 conda-forge libbrotlidec 1.0.9 h7f98852_6 conda-forge libbrotlienc 1.0.9 h7f98852_6 conda-forge libcblas 3.9.0 12_linux64_openblas conda-forge libcudf 21.10.01 cuda11.0_ga1d2d13a14_0 rapidsai libcuml 21.10.02 cuda11.0_gcd9251271_0 rapidsai libcumlprims 21.10.00 cuda11.0_g167dc59_0 nvidia libcurl 7.80.0 h2574ce0_0 conda-forge libdeflate 1.8 h7f98852_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libfaiss 1.7.0 cuda110h8045045_8_cuda conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 11.2.0 h1d223b6_11 conda-forge libgfortran-ng 11.2.0 h69a702a_11 conda-forge libgfortran5 11.2.0 h5c6108e_11 conda-forge libgomp 11.2.0 h1d223b6_11 conda-forge libhwloc 2.3.0 h5e5b7d1_1 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.9.0 12_linux64_openblas conda-forge libllvm10 10.0.1 he513fc3_3 conda-forge libllvm8 8.0.1 hc9558a2_0 conda-forge libnghttp2 1.43.0 h812cca2_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libprotobuf 3.18.1 h780b84a_0 conda-forge librmm 21.10.01 h2809392_0 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge libthrift 0.15.0 he6d91bd_1 conda-forge libtiff 4.3.0 h6f004c6_2 conda-forge libutf8proc 2.6.1 h7f98852_0 conda-forge libwebp-base 1.2.1 h7f98852_0 conda-forge libxgboost 1.4.2dev.rapidsai21.10 cuda11.0_0 rapidsai libxml2 2.9.12 h885dcf4_1 conda-forge libzlib 1.2.11 h36c2ea0_1013 conda-forge lightgbm 3.3.1 py38h709712a_1 conda-forge llvmlite 0.36.0 py38h4630a5e_0 conda-forge locket 0.2.0 py_2 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge markupsafe 2.0.1 py38h497a2fe_1 conda-forge mccabe 0.6.1 py_1 conda-forge msgpack-python 1.0.2 py38h1fd1430_2 conda-forge nccl 2.11.4.1 h96e36e3_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge numba 0.53.1 py38h8b71fd7_1 conda-forge numpy 1.21.4 py38he2449b9_0 conda-forge nvtx 0.2.3 py38h497a2fe_1 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1l h7f98852_0 conda-forge orc 1.7.0 h68e2c4e_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.3.4 py38h43a58ef_1 conda-forge parquet-cpp 1.5.1 2 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge pillow 8.4.0 py38h8e6f84c_0 conda-forge pip 21.3.1 pyhd8ed1ab_0 conda-forge protobuf 3.18.1 py38h709712a_0 conda-forge psutil 5.8.0 py38h497a2fe_2 conda-forge py-xgboost 1.4.2dev.rapidsai21.10 cuda11.0py38_0 rapidsai pyarrow 5.0.0 py38hed47224_8_cuda conda-forge pycodestyle 2.8.0 pyhd8ed1ab_0 conda-forge pyflakes 2.4.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge python 3.8.12 hb7a2778_2_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-rapidjson 1.5 pypi_0 pypi python_abi 3.8 2_cp38 conda-forge pytz 2021.3 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py38h497a2fe_3 conda-forge re2 2021.09.01 h9c3ff4c_0 conda-forge readline 8.1 h46c0cb4_0 conda-forge rmm 21.10.01 cuda_11.0_py38_gc54767f_0 rapidsai s2n 1.0.10 h9b69904_0 conda-forge scikit-learn 1.0.1 py38h1561384_2 conda-forge scipy 1.7.2 py38h56a6a73_0 conda-forge setuptools 59.2.0 py38h578d9bd_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.8 he1b5a44_3 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge spdlog 1.9.2 h4bd325d_0 conda-forge sqlite 3.36.0 h9cd32fc_2 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge thrust 1.11.0 ha770c72_1 conda-forge tk 8.6.11 h27826a3_1 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge tornado 6.1 py38h497a2fe_2 conda-forge treelite 2.1.0 py38hdd725b4_0 conda-forge treelite-runtime 2.1.0 pypi_0 pypi tritonclient 2.16.0 pypi_0 pypi typing_extensions 4.0.0 pyha770c72_0 conda-forge ucx 1.11.1+gc58db6b cuda11.0_0 rapidsai ucx-proc 1.0.0 gpu conda-forge ucx-py 0.22.0 py38_gc58db6b_0 rapidsai wheel 0.37.0 pyhd8ed1ab_1 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.6.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h36c2ea0_1013 conda-forge zope-event 4.5.0 pypi_0 pypi zope-interface 5.4.0 pypi_0 pypi zstd 1.5.0 ha95c52a_0 conda-forge

I was using the following command to create the model: python qa/L0_e2e/generate_example_model.py --type cuml --format pickle --name fil_classification --repo /home/ubuntu/models/ --task classification --predict_proba

The generate_example_model.py generate the model with the name model.pkl but Triton by default seems expecting checkpoint.tl.

Run model serving with: docker run --gpus all --rm -p8000:8000 -p8001:8001 -p8002:8002 -v /home/ubuntu/models:/models nvcr.io/nvidia/tritonserver:21.10-py3 tritonserver --model-repository=/models

This also fails for tritonserver:21.11-py3

wphicks commented 2 years ago

Thanks for the report @Jiahong-Nvidia! Was the conda environment you posted the one you used for the convert_cuml.py script? Could you add your invocation of that script as well?

jiahong-liu commented 2 years ago

I was actually using the qa/L0_e2e/generate_example_model.py directly to generate a new example model instead of the conversion path.

wphicks commented 2 years ago

Please take a look at the documentation for cuML and Scikit-Learn models here. These models require a conversion step since the FIL backend does not make use of a Python interpreter. We will update the documentation to further emphasize this.