Open LiuChen19960902 opened 1 year ago
build the project inside the container or build on the ECS?
You set TP=2 and PP=2 in your config, which requires 4 gpus. But you only have 2 gpus.
You set TP=2 and PP=2 in your config, which requires 4 gpus. But you only have 2 gpus.
You set TP=2 and PP=2 in your config, which requires 4 gpus. But you only have 2 gpus.
Thank you so much for your help! I understand now that I need to adjust the TP and PP settings based on the number of GPUs I have. My issue is resolved now. I appreciate your guidance and support! Wishing you all the best!
build the project inside the container or build on the ECS?
build the project inside the container or build on the ECS?
Thank you for your suggestion! Although I didn't end up resolving my issue using the approach you mentioned, I still appreciate your input and advice. Your willingness to help is truly appreciated. All the best to you!
I implemented it step by step according to the tutorial of https://github.com/triton-inference-server/fastertransformer_backend/blob/main/docs/bert_guide.md
the all_models floder like: all_models/bert/fastertransformer
then, some errors: