Closed ggbetz closed 5 months ago
Check upon issue creation:
Parameters:
NEXT_MODEL_PATH=databricks/dbrx-instruct NEXT_MODEL_REVISION=main NEXT_MODEL_PRECISION=float16 MAX_LENGTH=2048 GPU_MEMORY_UTILIZATION=0.7 VLLM_SWAP_SPACE=16
Note: Will probably need 6+ A100-80G
ToDos:
thx
Check upon issue creation:
Parameters:
Note: Will probably need 6+ A100-80G
ToDos: