search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.94k stars 169 forks source link

[Move] Common Evaluation Modules to `Sparsezoo`` #1520

Open rahul-tuli opened 6 months ago

rahul-tuli commented 6 months ago

Evaluator Move

This PR moves a few modules from deepsparse.evaluation to sparsezoo.evaluation

Motivation and Context

The moved modules provide a common interface for evaluating models. This interface can be leveraged and used by both SparseML and DeepSparse to evaluate models.

Since SparseZoo is a common dependency for both repositories, it makes sense to move these modules to SparseZoo.

How Has This Been Tested?

The tests for moved modules in deepsparse.evaluation are moved to sparsezoo.evaluation and other deepsparse specific deepsparse.evaluation tests were updated to use the moved modules.

Both deepsparse and sparsezoo tests were run to ensure that the move did not break anything.

Also ran the following eval command:

$ deepsparse.eval --target zoo:mpt-7b-mpt_pretrain-base_quantized \                                                             (move-evaluator-to-sparsezoo|●1✚2…4)
                --dataset hellaswag \
                --dataset gsm8k \
                --integration lm-evaluation-harness \
                --limit 2

Note: This diff relies on sparsezoo diff https://github.com/neuralmagic/sparsezoo/pull/411

The failing tests should pass after sparsezoo side changes are merged!

bfineran commented 5 months ago

base tests failing - LGTM otherwise