search

aws / amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
https://sagemaker-examples.readthedocs.io
Apache License 2.0
9.79k stars 6.67k forks source link

Deploy this TheBloke/vicuna-13B-v1.5-GGUF model on AWS #4603

Open ahsan3219 opened 3 months ago

ahsan3219 commented 3 months ago

Deploy this TheBloke/vicuna-13B-v1.5-GGUF model on AWS

I want to use this model as an endpoint in my web application in this format: image

Chatbot Requirements

  1. Scope: Chatbot (Encoder/Decoder for Text Inference or Conversational)

  2. Input via API (JSON): Chatgpt Style – The template can be see below

    The JSON will contain 25 user messages, and the response should be the system response. Please use this guidelines to understand API consumption: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_runtime_InvokeEndpoint.html

  3. Prompt Template for the system: a. template = ''' You are going to be my education assistant. System:{System} Question:{question}'''

  4. LLM Model Parameters: max_new_tokens=512, temperature=0.7, top_p=0.9

  5. If possible use a AutomodelforCausalLM otherwise train a LLM model.

  6. It will be deployed on AWS Sagemaker using S3 buckets.

  7. The GGUF should be saved on a S3 Bucket.

  8. Chat Buffer should store 25 conversations and create a session ID (No need to send this to the End point).

  9. The quantized model is contained here https://huggingface.co/TheBloke/vicuna-13B-v1.5-GGUF/blob/main/vicuna-13b-v1.5.Q4_K_M.gguf

  10. Use HuggingFace/Langchain when possible.

  11. Deliverables: Jupyter notebook/Code – 2 Hours should be used to set up the model in AWS with the customer.

Provide me with complete source code that I can use in my jupyter notebook on aws to make an endpoint. I need it asap.