Custom model import - Llama 3 8B sample: Wrong torch_type in PEFT model

aws-samples / amazon-bedrock-samples

This repository contains examples for customers to get started using the Amazon Bedrock Service. This contains examples for all available foundational models

https://aws.amazon.com/bedrock/

MIT No Attribution

358 stars 183 forks source link

Custom model import - Llama 3 8B sample: Wrong torch_type in PEFT model #193

Closed lopezfelipe closed 6 days ago

lopezfelipe commented 3 weeks ago

Hello -- The sample for importing a custom model into Bedrock has a bug when loading the PEFT model.

The current script will finish training but will result in a model that Bedrock cannot load.

# load PEFT model
model = AutoPeftModelForCausalLM.from_pretrained(
    training_args.output_dir,
    low_cpu_mem_usage=True,
    torch_dtype=torch_dtype
)

The issue appears to be fixed when changing the torch_dtype definition to torch.float16

rsgrewal-aws commented 6 days ago

fixed