search

triton-inference-server / fastertransformer_backend

BSD 3-Clause "New" or "Revised" License
411 stars 134 forks source link

Request to support GCS file path #57

Open aasthajh opened 2 years ago

aasthajh commented 2 years ago

Hi, I'm trying to deploy a Faster Transformer based LLM using Triton on a GCP instance. I was wondering if there's a way to provide the file path to the Google Cloud Storage bucket when passing the model check point in the config.pbtxt file?

parameters {
  key: "model_checkpoint_path"
  value: {
    string_value: "gs://triton_sample_models/model-ul2-ft/ul2/1/2-gpu"
  }
}
byshiue commented 2 years ago

Currently FasterTransformer Backend only supports reading model files and configuration files from the local disk to initialize. In later versions, we will gradually support reading configuration files and model files from kafka and the cloud.

zinzinhust96 commented 1 year ago

Sorry to bring up this issue again @byshiue Is there a way to provide model path in GCS bucket to parameter "model_checkpoint_path" in config.pbtxt file now? Thank you!