Open arvi18 opened 1 year ago
"hub = { HF_MODEL_ID': 'NumbersStation/nsql-llama-2-7B', SM_NUM_GPUS': json.dumps(1),"
SM_FRAMEWORK_PARAMS': "{precision_mode: 'bf16'}", `NO`
SM_FRAMEWORK_PARAMS': '{"precision": "bfloat16"}', `NO`
SM_FRAMEWORK_PARAMS': '{"precision": "bf16"}', `NO`
'SM_FRAMEWORK_PARAMS': "{'torch_dtype': 'torch.bfloat16'}" `NO`
'SM_FRAMEWORK_PARAMS': "{'torch_dtype': 'bfloat16'}" `NO`
'HF_MODEL_QUANTIZE': 'bitsandbytes'
Error: ValueError: A device map needs to be passed to run convert models into mixed-int8 format. Please run`.from_pretrained` with `device_map='auto'`
#033[2m#033[3mrank#033[0m#033[2m=#033[0m0#033[0m
`NO`
'HF_MODEL_QUANTIZE': 'gptq'
Error: ValueError: gptq quantization is not supported for AutoModel, you can try to quantize it with `text-generation-server quantize ORIGINAL_MODEL_ID NEW_MODEL_ID`
#033[2m#033[3mrank#033[0m#033[2m=#033[0m0#033[0m
`NO`
I am facing problems with deployment on Sagemaker. Instance:
ml.g5.2xlarge
With default config this happensThis could be solved by loading model with
bfloat16
/fp16
instead offloat32
to fit it into the VRAM properly.I couldn't figure out a way to do so. I am using this script to deploy it.