substratusai / runbooks

Finetune LLMs on K8s by using Runbooks
https://www.substratus.ai
Other
168 stars 14 forks source link

Add env support and remove params from env #231

Closed samos123 closed 1 year ago

samos123 commented 1 year ago

Allow setting environment variables directly from the Substratus resources.

E.g.

spec:
  env:
    MY_KEY: value
    SECRETY_SECRET: ${{ secrets.ai.HUGGING_FACE_HUB_TOKEN }}

This also removes the functionality of Substratus of setting PARAM_ environment variables. So this will require updating our images and possibly examples.

samos123 commented 1 year ago

PR for updating the images: https://github.com/substratusai/images/pull/45 . I will also start updating the examples

samos123 commented 1 year ago

Verified on Kind with this model:

apiVersion: substratus.ai/v1
kind: Model
metadata:
  name: llama-2-7b
spec:
  image: substratusai/model-loader-huggingface:pr-45
    #build:
    #  git:
    #    url: https://github.com/substratusai/images
    #    branch: no-param-envs
    #    path: model-loader-huggingface
  env:
    HUGGING_FACE_HUB_TOKEN: ${{ secrets.ai.HUGGING_FACE_HUB_TOKEN }}
  params:
    name: meta-llama/Llama-2-7b-hf