search

aws-neuron / transformers-neuronx

Apache License 2.0
88 stars 25 forks source link

Backward compatibility with saved llama 2 compiled artifacts #78

Open dacorvo opened 5 months ago

dacorvo commented 5 months ago

After upgrading to transformers-neuronx == 0.9.474, I am not able to reload the compiled artifacts for my llama model I saved using transformers-neuronx == 0.8.268.

FileNotFoundError: Could not find a matching NEFF for your HLO in this directory. Ensure that the model you are trying to load is the same type and has the same parameters as the one you saved or call "save" on this model to reserialize it.

The model is identical and so are my parameters. Could you confirm that this is expected (I assume the LLamaForSampling hlo has been modified between the two versions) ?

Will future releases be backward compatible ? This is important to know when defining a strategy for saving neuron models.

awsilya commented 5 months ago

@dacorvo - unfortunately we do not support this. The model needs to be recompiled. We are aware of this limitation but at the moment I cannot give you an ETA for the fix.