Open sacdroid opened 11 months ago
I think the TRITON_MODEL_DIRECTORY
is only supported in Triton Python backend at the moment. It is up to the backend to be able to dynamically set the path when using S3.
CC @Tabrizian for any possible correction.
Do we have any update on this feature? External repo loading is quite critical for larger model image as Docker layer size has 52Gib limitation (at least on AWS). Therefore we cannot build the model into the image. Request to get gpt_model_path dynamically set the path based on where it cache the repo.
Any updates on this?
I am trying externalize model artifacts to S3 using Triton's Cloud Storage support for model repository. I am able to get this working for pre/postprocessing tokenizer model instances using
and the replacing this in model.py.
How can I achieve the same
gpt_model_path
? I looked at the code and it does not seem to support dynamic path today. Do you have any alternatives which does not require me to include model artifacts in docker container or have external mount?