search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.99k stars 173 forks source link

[Add] sequence length to text gen pipelines #1518

Closed rahul-tuli closed 9 months ago

rahul-tuli commented 9 months ago

This PR exposes sequence_length as a Pipeline level property

Test Code:

from deepsparse import Pipeline

pipeline = Pipeline.create(
            task="text-generation",
            model_path="zoo:mpt-7b-mpt_pretrain-base_quantized",
        )

assert hasattr(pipeline, "sequence_length")

Was failing before, but passes after this PR