Model compilation happening at every step

philschmid / aws-neuron-samples

MIT License

12 stars 1 forks source link

Model compilation happening at every step #4

Open DhruvaBansal00 opened 1 year ago

DhruvaBansal00 commented 1 year ago

Hello!

I am trying to train a Flan-T5 small model using the Huggingface Seq2SeqTrainer and am noticing that optimum is compiling the model at every step. Is there a good way to avoid this from happening? I tried to follow these steps: https://huggingface.co/docs/optimum-neuron/guides/cache_system but on trying optimum-cli neuron cache create I get an error saying: Optimum CLI tool: error: invalid choice: 'neuron' (choose from 'export', 'env', 'onnxruntime')

DhruvaBansal00 commented 1 year ago

For more context, I am seeing this at every step:

DhruvaBansal00 commented 1 year ago

I made a private repo and tried prepending CUSTOM_CACHE_REPO="dhruvabansal/neuron_model_cache" to the torchrun command as well, but I think it is still compiling at every step: