neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.94k stars 169 forks source link

[BugFixes] `deepsparse.eval` #1517

Closed rahul-tuli closed 6 months ago

rahul-tuli commented 6 months ago

This PR fixes a few bugs found while updating the eval framework:

1) sequence_length is no longer exposed at the pipeline level

from deepsparse import Pipeline

pipeline = Pipeline.create(
            task="text-generation",
            model_path="zoo:mpt-7b-mpt_pretrain-base_quantized",
        )

assert hasattr(pipeline, "sequence_length")
Traceback (most recent call last):
  File "/home/rahul/projects/deepsparse/local.py", line 12, in <module>
    assert hasattr(pipeline, "sequence_length")
AssertionError
>>> 

Fix: workaround relies on pipeline.ops["single_engine"].sequence_length

2) deepsparse.eval requires a specific commit from lm-eval-harness for now, updating the install message to reflect that