search

neuralmagic / sparseml

Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models
Apache License 2.0
2.01k stars 140 forks source link

[Fix] Allow to create `SparseAutoModelForCausalLM` with `trust_remote_code=True` #2349

Closed dbogunowicz closed 5 days ago

dbogunowicz commented 6 days ago

Feature Description

Now this executes properly:

from sparseml.transformers import SparseAutoModelForCausalLM
model = SparseAutoModelForCausalLM.from_pretrained("microsoft/Phi-3-mini-128k-instruct",trust_remote_code=True)
print(model.__class__.__name__)
>> 'Phi3ForCausalLM'

The hack was to temporarily rename the class so that the from_pretrained method could properly resolve the "remote code load" logic. The model return by the from_pretrained method is unaffected later on, as it will have the class name of the loaded function (as shown above)

This PR needs to land as well, so that the functionality is fully enabled: https://github.com/neuralmagic/compressed-tensors/pull/104