Open ggbetz opened 3 months ago
Check upon issue creation:
Parameters:
NEXT_MODEL_PATH=CohereForAI/c4ai-command-r-v01 NEXT_MODEL_REVISION=main NEXT_MODEL_PRECISION=float16 MAX_LENGTH=2048 GPU_MEMORY_UTILIZATION=0.7 VLLM_SWAP_SPACE=8
Note: Will require 2 A100-80.
ToDos:
Check upon issue creation:
Parameters:
Note: Will require 2 A100-80.
ToDos: