intel / neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime
https://intel.github.io/neural-compressor/
Apache License 2.0
2.18k stars 252 forks source link

update itrex ut test #1929

Closed chensuyue closed 2 months ago

chensuyue commented 2 months ago

Type of Change

validation

Description

update itrex ut test

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

github-actions[bot] commented 2 months ago

⛈️ Required checks status: Has failure 🔴

Warning If you do not have the access to re-run the Probot, please contact XuehaoSun for help. If you push a new commit, all of the workflow will be re-triggered.

Groups summary

🔴 Unit Tests basic workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-Basic](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812) | failure | | ❌ | | [UT-Basic (Coverage Compare CollectDatafiles)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=71384379-497b-5787-1f51-cc2e0f831d78) | failure | [download](https://artprodcus3.artifacts.visualstudio.com/Acd5c2212-3bfc-4706-9afe-b292ced6ae69/b7121868-d73a-4794-90c1-23135f974d09/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2xwb3QtaW5jL3Byb2plY3RJZC9iNzEyMTg2OC1kNzNhLTQ3OTQtOTBjMS0yMzEzNWY5NzRkMDkvYnVpbGRJZC8zMjgxMi9hcnRpZmFjdE5hbWUvVVRfY292ZXJhZ2VfcmVwb3J00/content?format=zip) | ❌ | | [UT-Basic (Unit Test FWKs adaptor Test FWKs adaptor)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=e5896b99-a49d-517b-218b-3b918f0c116d) | success | | ✅ | | [UT-Basic (Unit Test FWKs adaptor baseline Test FWKs adaptor baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=f4046902-2b73-5016-25ba-63e28d81f503) | success | | ✅ | | [UT-Basic (Unit Test ITEX Test ITEX)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=7238057d-7331-5929-31ca-3e26743a41d0) | success | | ✅ | | [UT-Basic (Unit Test ITEX baseline Test ITEX baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=baf42d1a-8efd-5d70-9d09-b6bec5e8b55b) | success | | ✅ | | [UT-Basic (Unit Test Pruning Test PyTorch Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=99af828c-75c2-5dc7-0037-29447d6396f5) | success | | ✅ | | [UT-Basic (Unit Test Pruning Test TensorFlow Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=0a0ef4d6-07d3-59ba-0c8d-8c4daa447e58) | success | | ✅ | | [UT-Basic (Unit Test Pruning baseline Test PyTorch Pruning baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=b7474179-6386-5b4d-8836-092e0670eff4) | success | | ✅ | | [UT-Basic (Unit Test Pruning baseline Test TensorFlow Pruning baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=461a3442-9d84-5494-e465-40a939a41758) | success | | ✅ | | [UT-Basic (Unit Test TF newAPI Test TF newAPI)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=24d8ed3a-4c60-5ddd-e9bd-ea3ff52a4e98) | success | | ✅ | | [UT-Basic (Unit Test TF newAPI baseline Test TF newAPI baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=20c67c77-4f92-5283-357b-06b3e2359537) | success | | ✅ | | [UT-Basic (Unit Test User facing API Test User facing API)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=837bf053-a137-5cf0-a3a8-e85b8f58d512) | success | | ✅ | | [UT-Basic (Unit Test User facing API baseline Test User facing API baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=856f095c-c483-5936-5dd8-0c2f82743762) | success | | ✅ | | [UT-Basic (Unit Test other basic case Test other basic case)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=c6aa4c58-99e4-54e9-e3eb-cd322b75c938) | success | | ✅ | | [UT-Basic (Unit Test other cases baseline Test other cases baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32812&view=logs&jobId=b62922ce-9ef8-5e77-08b8-9cffa1d355b8) | success | | ✅ |
UT-Basic coverage report
Base coverage PR coverage Diff
Lines 75.665% 71.868% -3.797%
Branches 64.724% 61.329% -3.395%
Commit FileName Stmts Miss Branch Cover
PR | BASE adaptor/tf_utils/graph_util.py 491 | 491 55 | 53 274 | 274 87% | 87%
PR | BASE compression/hpo/search_algorithms.py 154 | 154 5 | 6 56 | 56 96% | 95%
PR | BASE compression/pruner/patterns/nxm.py 458 | 458 48 | 50 182 | 182 86% | 85%
PR | BASE onnxrt/__init__.py 5 | NA 5 | NA 0 | NA 0% | NA
PR | BASE onnxrt/algorithms/__init__.py 6 | NA 6 | NA 0 | NA 0% | NA
PR | BASE onnxrt/algorithms/layer_wise/__init__.py 2 | NA 2 | NA 0 | NA 0% | NA
PR | BASE onnxrt/algorithms/layer_wise/core.py 103 | NA 103 | NA 40 | NA 0% | NA
PR | BASE onnxrt/algorithms/smoother/__init__.py 2 | NA 2 | NA 0 | NA 0% | NA
PR | BASE onnxrt/algorithms/smoother/calibrator.py 115 | NA 115 | NA 54 | NA 0% | NA
PR | BASE onnxrt/algorithms/smoother/core.py 283 | NA 283 | NA 123 | NA 0% | NA
PR | BASE onnxrt/algorithms/weight_only/__init__.py 0 | NA 0 | NA 0 | NA 100% | NA
PR | BASE onnxrt/algorithms/weight_only/awq.py 189 | NA 189 | NA 78 | NA 0% | NA
PR | BASE onnxrt/algorithms/weight_only/gptq.py 192 | NA 192 | NA 76 | NA 0% | NA
PR | BASE onnxrt/algorithms/weight_only/rtn.py 74 | NA 74 | NA 30 | NA 0% | NA
PR | BASE onnxrt/algorithms/weight_only/utility.py 107 | NA 107 | NA 38 | NA 0% | NA
PR | BASE onnxrt/quantization/__init__.py 5 | NA 5 | NA 0 | NA 0% | NA
PR | BASE onnxrt/quantization/algorithm_entry.py 60 | NA 60 | NA 4 | NA 0% | NA
PR | BASE onnxrt/quantization/autotune.py 47 | NA 47 | NA 10 | NA 0% | NA
PR | BASE onnxrt/quantization/calibrate.py 6 | NA 6 | NA 0 | NA 0% | NA
PR | BASE onnxrt/quantization/config.py 255 | NA 255 | NA 78 | NA 0% | NA
PR | BASE onnxrt/quantization/quantize.py 25 | NA 25 | NA 6 | NA 0% | NA
PR | BASE onnxrt/utils/__init__.py 3 | NA 3 | NA 0 | NA 0% | NA
PR | BASE onnxrt/utils/onnx_model.py 564 | NA 564 | NA 344 | NA 0% | NA
PR | BASE onnxrt/utils/utility.py 112 | NA 112 | NA 50 | NA 0% | NA
These checks are required after the changes to `.azure-pipelines/scripts/ut/run_itrex.sh`.
🟢 Unit Tests basic no coverage workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-Basic-No-Coverage](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32813) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test FWKs adaptor Test FWKs adaptor)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32813&view=logs&jobId=e5896b99-a49d-517b-218b-3b918f0c116d) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test Pruning Test PyTorch Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32813&view=logs&jobId=99af828c-75c2-5dc7-0037-29447d6396f5) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test Pruning Test TensorFlow Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32813&view=logs&jobId=0a0ef4d6-07d3-59ba-0c8d-8c4daa447e58) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test User facing API Test User facing API)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32813&view=logs&jobId=837bf053-a137-5cf0-a3a8-e85b8f58d512) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test other basic case Test other basic case)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32813&view=logs&jobId=c6aa4c58-99e4-54e9-e3eb-cd322b75c938) | success | | ✅ | These checks are required after the changes to `.azure-pipelines/scripts/ut/run_itrex.sh`.
🟢 Unit Tests ITREX workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-ITREX](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32814) | success | | ✅ | These checks are required after the changes to `.azure-pipelines/scripts/ut/run_itrex.sh`.

Thank you for your contribution! 💜

Note This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact chensuyue or XuehaoSun for help.