search

awslabs / llm-hosting-container

Large Language Model Hosting Container
Apache License 2.0
75 stars 32 forks source link

feature: optimum-neuronx 0.0.16 #45

Closed jinyoung-lim closed 9 months ago

jinyoung-lim commented 9 months ago

Issue #, if available: https://sim.amazon.com/issues/SMJS-155

Description of changes: Adding Dockerfile for Optimum Neuronx 0.0.16 support. Some more version details:

  1. TGI 1.0.2
  2. Ubuntu 22.04
  3. Python 3.10 (https://packages.ubuntu.com/jammy/python3)
  4. PyTorch 1.13.1

Testing done by building the image with command: docker build --rm -f huggingface/pytorch/tgi/docker/1.0.2/py3/optimum/0.0.16/Dockerfile.neuronx -t neuronx-tgi:0.0.16-1.0.2 .

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.