Open architkulkarni opened 11 months ago
To verify the permissions issue we ran the following from the worker node:
aws s3 sync s3://anyscale-staging-data-cld-kvedzwag2qa8i5bjxuevf5i7/org_7c1Kalm9WcX2bNIjW53GUT/cld_kvedZWag2qA8i5BjxUevf5i7/artifact_storage/archit__kulkarni/ft_llms_with_deepspeed/meta-llama/Llama-2-7b-hf/demo-gsm-7b/ /home/ray/.cache/huggingface/hub/models--meta-llama--Llama-2-7b-chat-hf/snapshots/0000000000000000000000000000000000000000/
fatal error: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
After giving the worker node the correct permissions, manually re-running the s3 sync
command, and re-running aviary run
(where my understanding is that it reuses the cached data in the 000...
folder), the model started producing the correct fine-tuned output.
Reproduction: Add this to the Llama-2-7b model YAML, or probably any model YAML:
Any bucket for which your cluster doesn't have permissions should also reproduce the issue.
When you run
aviary run
for that model YAML, the following messages are printed, which are misleading:In fact, the folder is empty:
From what I understand from @kouroshHakha and @Yard1 (feel free to correct me): The folder is not supposed to be empty. Because it's empty, when you serve the model you're actually serving some other model (the "base model" or "chat model", I don't remember exactly), instead of the model from the specified S3 bucket.