Add additional parameter for model quantization + other features in NeuronXXXModel

[samir-souza]

NeuronXXXModel classes (i.e. NeuronDecoderModel - optimum/neuron/modeling_decoder.py) invoke transformers-neuronx to compile the target model, however these classes don't pass all the supported input parameters and limit key features like model quantization.

Ask: Accept all the supported parameters like "neuron_config" and "context_length_estimate" (and others)

For instance, when using transformers-neuronx we need to pass neuron_config to quantize the model to int8.

neuron_config = NeuronConfig(
    quant=QuantizationConfig(quant_dtype='s8', dequant_dtype='bf16'),
)
kwargs = {
    "batch_size": batch_size,
    "amp": dtype,
    "tp_degree": tp_degree,
    "n_positions": n_positions,
    "unroll": None,
    "neuron_config": neuron_config,
    'context_length_estimate': [32, 128, 256, 400, 512, 850]
}
model = LlamaForSampling.from_pretrained(os.path.join(model_dir, "llama2-split"), **kwargs)

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[yahavb]

we would like to support int8 weight storage like in https://awsdocs-neuron.readthedocs-hosted.com/en/latest/libraries/transformers-neuronx/transformers-neuronx-developer-guide.html#int8-weight-storage-support to run the model on smaller instances.

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!

[HuggingFaceDocBuilderDev]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Thank you!