Closed jamontesg closed 6 months ago
You mentioned usinghg-flan-xxl
. It is noted in the README that "The demonstration's google/flan-t5-xxl-fp16
is
capable of answering basic natural language queries with sufficient in-context learning, but will fail to return an
answer, provide incorrect answers, or cause the model endpoint to experience timeouts due to resource exhaustion when
faced with moderate to complex questions. Users are encouraged to experiment with a variety of open source and
commercial JumpStart Foundation Models and Amazon Bedrock." If you are using google/flan-t5-xxl-fp16
via Amazon SageMaker Studio JumpStart, then this is a known limitation.
I am trying to run your project and has this error
message": CUDA out of memory
I am using
hg-flan-xxl and ml.g5.24xlarge
defineds on git project
this happends with any question to model
Regards