Open IlyaMescheryakov1402 opened 6 months ago
I think this is because https://github.com/allegroai/clearml-serving/blob/main/clearml_serving/engines/triton/triton_helper.py#L140 - it can't download model from s3, because clearml.storage.helper.StorageHelper can't configure _Boto3Driver using only env variables
I added clearml.conf file with aws.s3 creds to the root of git repository and fixed my Dockerfile.vllm:
FROM nvcr.io/nvidia/tritonserver:24.02-vllm-python-py3
ENV LC_ALL=C.UTF-8
COPY clearml_serving /root/clearml/clearml_serving
COPY requirements.txt /root/clearml/requirements.txt
COPY clearml.conf /root/clearml.conf
COPY README.md /root/clearml/README.md
COPY setup.py /root/clearml/setup.py
RUN python3 -m pip install --no-cache-dir -r /root/clearml/clearml_serving/engines/triton/requirements.txt
RUN python3 -m pip install --no-cache-dir -U pip -e /root/clearml/
# default serving port
EXPOSE 8001
# environement variable to load Task from CLEARML_SERVING_TASK_ID, CLEARML_SERVING_PORT
WORKDIR /root/clearml/
ENTRYPOINT ["clearml_serving/engines/triton/entrypoint.sh"]
and then I fixed entrypoint.sh:
#!/bin/bash
# print configuration
echo CLEARML_SERVING_TASK_ID="$CLEARML_SERVING_TASK_ID"
echo CLEARML_TRITON_POLL_FREQ="$CLEARML_TRITON_POLL_FREQ"
echo CLEARML_TRITON_METRIC_FREQ="$CLEARML_TRITON_METRIC_FREQ"
echo CLEARML_TRITON_HELPER_ARGS="$CLEARML_TRITON_HELPER_ARGS"
echo CLEARML_EXTRA_PYTHON_PACKAGES="$CLEARML_EXTRA_PYTHON_PACKAGES"
# we should also have clearml-server configurations
if [ ! -z "$CLEARML_EXTRA_PYTHON_PACKAGES" ]
then
python3 -m pip install $CLEARML_EXTRA_PYTHON_PACKAGES
fi
# start service
clearml-init --file /root/clearml.conf && PYTHONPATH=$(pwd) python3 clearml_serving/engines/triton/triton_helper.py $CLEARML_TRITON_HELPER_ARGS $@
Actually I don't know why I faced this issue, I think I did something wrong. In enterprise version we didn't face one because of configuration vault.
Hello!
I use ClearML free (the one without configuration vault stuff) + clearml-serving module
When I spinned docker-compose and tried to pull model from our s3, I've got an error in tritonserver container:
But I've set env variables in example.env (AWS_ ones too) and I could find them in tritonserver container via
FILES
docker-compose-triton-gpu.yaml
example.env
Dockerfile.vllm: