search

intel / neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime
https://intel.github.io/neural-compressor/
Apache License 2.0
2.23k stars 257 forks source link

implement `incbench` command for ease-of-use benchmark #1884

Closed xin3he closed 4 months ago

xin3he commented 4 months ago

Type of Change

feature

Description

General Use Cases

  1. incbench main.py: run 1 instance on NUMA:0.
  2. incbench --num_i 2 main.py: run 2 instances on NUMA:0.
  3. incbench --num_c 2 main.py: run multi-instances with 2 cores per instance on NUMA:0.
  4. incbench -C 24-47 main.py: run 1 instance on COREs:24-47.
  5. incbench -C 24-47 --num_c 4 main.py: run multi-instances with 4 COREs per instance on COREs:24-47.

Note:

  • num_i works the same as num_instances
  • num_c works the same as num_cores_per_instance

Expected Behavior & Potential Risk

image

How has this PR been tested?

tested on Ultra7 1003H (1 socket, 6 P-Cores, 8 E-Cores and 2 SoC E-Cores) and cascade lake(2 numa, 24 cores per numa)

Dependency Change?

N/A

github-actions[bot] commented 4 months ago

⛈️ Required checks status: Has failure 🔴

Warning If you do not have the access to re-run the Probot, please contact XuehaoSun for help. If you push a new commit, all of the workflow will be re-triggered.

Groups summary

🟢 Code Scan Tests workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [Code-Scan](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32438) | success | | ✅ | | [Code-Scan (Bandit Code Scan Bandit)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32438&view=logs&jobId=343c57fa-283e-589b-e772-0a0553c93e53) | success | | ✅ | | [Code-Scan (DocStyle Code Scan DocStyle)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32438&view=logs&jobId=c1e234ec-db76-5d40-e8f0-e1ad3ef905a3) | success | | ✅ | | [Code-Scan (Pylint Code Scan Pylint)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32438&view=logs&jobId=454075da-6b11-57a5-edf2-4c5947924fa8) | success | | ✅ | These checks are required after the changes to `neural_compressor/common/base_config.py`, `neural_compressor/common/benchmark.py`, `neural_compressor/common/utils/utility.py`, `neural_compressor/tensorflow/utils/utility.py`, `neural_compressor/torch/algorithms/static_quant/utility.py`, `neural_compressor/torch/utils/utility.py`, `setup.py`.
🟢 Code Scan Neural-Solution Tests workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [Code-Scan-Neural-Solution](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32443) | success | | ✅ | | [Code-Scan-Neural-Solution (Bandit Code Scan Bandit)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32443&view=logs&jobId=343c57fa-283e-589b-e772-0a0553c93e53) | success | | ✅ | | [Code-Scan-Neural-Solution (DocStyle Code Scan DocStyle)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32443&view=logs&jobId=c1e234ec-db76-5d40-e8f0-e1ad3ef905a3) | success | | ✅ | | [Code-Scan-Neural-Solution (Pylint Code Scan Pylint)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32443&view=logs&jobId=454075da-6b11-57a5-edf2-4c5947924fa8) | success | | ✅ | These checks are required after the changes to `setup.py`.
🟢 Model Tests workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [Model-Test](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32441) | success | | ✅ | | [Model-Test (Generate Report GenerateReport)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32441&view=logs&jobId=131b5a5d-c16f-50a4-e704-41ef17f1e502) | success | | ✅ | | [Model-Test (Run ONNX Model resnet50-v1-12)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32441&view=logs&jobId=8036be5b-c44f-5dbf-ab25-6a62f8567a7d) | success | | ✅ | | [Model-Test (Run PyTorch Model resnet18_fx)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32441&view=logs&jobId=e3333e26-4334-5d5e-a7bd-ee7471860e42) | success | | ✅ | | [Model-Test (Run TensorFlow Model resnet50v1.5)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32441&view=logs&jobId=d570adab-99d2-59fe-b05d-1b608d431203) | success | | ✅ | | [Model-Test (Run TensorFlow Model ssd_resnet50_v1)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32441&view=logs&jobId=aff40fc6-1d0e-52aa-b4b9-bb1d69413565) | success | | ✅ | These checks are required after the changes to `setup.py`.
🟢 Model Tests 3x workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [Model-Test-3x](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32449) | success | | ✅ | | [Model-Test-3x (Generate Report GenerateReport)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32449&view=logs&jobId=131b5a5d-c16f-50a4-e704-41ef17f1e502) | success | | ✅ | | [Model-Test-3x (Run PyTorch Model opt_125m_woq_gptq_int4)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32449&view=logs&jobId=e3333e26-4334-5d5e-a7bd-ee7471860e42) | success | | ✅ | | [Model-Test-3x (Run PyTorch Model opt_125m_woq_gptq_int4_dq_bnb)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32449&view=logs&jobId=2ce7611e-1b4a-5a32-2344-8e2c635b00bd) | success | | ✅ | | [Model-Test-3x (Run PyTorch Model opt_125m_woq_gptq_int4_dq_ggml)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32449&view=logs&jobId=919e49eb-e265-548b-eaa7-50d75ec3f230) | success | | ✅ | These checks are required after the changes to `neural_compressor/common/base_config.py`, `neural_compressor/common/benchmark.py`, `neural_compressor/common/utils/utility.py`, `neural_compressor/torch/algorithms/static_quant/utility.py`, `neural_compressor/torch/utils/utility.py`, `setup.py`.
🟢 Unit Tests basic workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-Basic](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440) | success | | ✅ | | [UT-Basic (Coverage Compare CollectDatafiles)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=71384379-497b-5787-1f51-cc2e0f831d78) | success | | ✅ | | [UT-Basic (Unit Test FWKs adaptor Test FWKs adaptor)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=e5896b99-a49d-517b-218b-3b918f0c116d) | success | | ✅ | | [UT-Basic (Unit Test FWKs adaptor baseline Test FWKs adaptor baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=f4046902-2b73-5016-25ba-63e28d81f503) | success | | ✅ | | [UT-Basic (Unit Test ITEX Test ITEX)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=7238057d-7331-5929-31ca-3e26743a41d0) | success | | ✅ | | [UT-Basic (Unit Test ITEX baseline Test ITEX baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=baf42d1a-8efd-5d70-9d09-b6bec5e8b55b) | success | | ✅ | | [UT-Basic (Unit Test Pruning Test PyTorch Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=99af828c-75c2-5dc7-0037-29447d6396f5) | success | | ✅ | | [UT-Basic (Unit Test Pruning Test TensorFlow Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=0a0ef4d6-07d3-59ba-0c8d-8c4daa447e58) | success | | ✅ | | [UT-Basic (Unit Test Pruning baseline Test PyTorch Pruning baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=b7474179-6386-5b4d-8836-092e0670eff4) | success | | ✅ | | [UT-Basic (Unit Test Pruning baseline Test TensorFlow Pruning baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=461a3442-9d84-5494-e465-40a939a41758) | success | | ✅ | | [UT-Basic (Unit Test TF newAPI Test TF newAPI)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=24d8ed3a-4c60-5ddd-e9bd-ea3ff52a4e98) | success | | ✅ | | [UT-Basic (Unit Test TF newAPI baseline Test TF newAPI baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=20c67c77-4f92-5283-357b-06b3e2359537) | success | | ✅ | | [UT-Basic (Unit Test User facing API Test User facing API)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=837bf053-a137-5cf0-a3a8-e85b8f58d512) | success | | ✅ | | [UT-Basic (Unit Test User facing API baseline Test User facing API baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=856f095c-c483-5936-5dd8-0c2f82743762) | success | | ✅ | | [UT-Basic (Unit Test other basic case Test other basic case)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=c6aa4c58-99e4-54e9-e3eb-cd322b75c938) | success | | ✅ | | [UT-Basic (Unit Test other cases baseline Test other cases baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32440&view=logs&jobId=b62922ce-9ef8-5e77-08b8-9cffa1d355b8) | success | | ✅ |
UT-Basic coverage report
Base coverage PR coverage Diff
Lines 76.020% 76.096% 0.076%
Branches 65.111% 65.176% 0.065%
These checks are required after the changes to `setup.py`.
🔴 Unit Tests basic no coverage workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-Basic-No-Coverage](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32442) | failure | | ❌ | | [UT-Basic-No-Coverage (Unit Test FWKs adaptor Test FWKs adaptor)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32442&view=logs&jobId=e5896b99-a49d-517b-218b-3b918f0c116d) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test Pruning Test PyTorch Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32442&view=logs&jobId=99af828c-75c2-5dc7-0037-29447d6396f5) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test Pruning Test TensorFlow Pruning)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32442&view=logs&jobId=0a0ef4d6-07d3-59ba-0c8d-8c4daa447e58) | success | | ✅ | | [UT-Basic-No-Coverage (Unit Test User facing API Test User facing API)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32442&view=logs&jobId=837bf053-a137-5cf0-a3a8-e85b8f58d512) | failure | [download](https://artprodcus3.artifacts.visualstudio.com/Acd5c2212-3bfc-4706-9afe-b292ced6ae69/b7121868-d73a-4794-90c1-23135f974d09/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2xwb3QtaW5jL3Byb2plY3RJZC9iNzEyMTg2OC1kNzNhLTQ3OTQtOTBjMS0yMzEzNWY5NzRkMDkvYnVpbGRJZC8zMjQ0Mi9hcnRpZmFjdE5hbWUvMV91dF9hcGlfcmVwb3J00/content?format=file&subPath=%2Fut_api.log) | ❌ | | [UT-Basic-No-Coverage (Unit Test other basic case Test other basic case)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32442&view=logs&jobId=c6aa4c58-99e4-54e9-e3eb-cd322b75c938) | success | | ✅ | These checks are required after the changes to `setup.py`.
🟢 Unit Tests ITREX workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-ITREX](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32446) | success | | ✅ | These checks are required after the changes to `setup.py`.
🟢 Unit Tests Neural-Solution workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-Neural-Solution](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32444) | success | | ✅ | These checks are required after the changes to `setup.py`.
🟢 Unit Tests Neural-Coder workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-Coder](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32439) | success | | ✅ | These checks are required after the changes to `setup.py`.
🟢 Unit Tests 3x-TensorFlow workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-3x-TensorFlow](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32447) | success | | ✅ | | [UT-3x-TensorFlow (Coverage Compare CollectDatafiles)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32447&view=logs&jobId=71384379-497b-5787-1f51-cc2e0f831d78) | success | | ✅ | | [UT-3x-TensorFlow (Unit Test 3x TensorFlow Unit Test 3x TensorFlow)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32447&view=logs&jobId=45daf969-8bc5-5011-3358-6610c803470a) | success | | ✅ | | [UT-3x-TensorFlow (Unit Test 3x TensorFlow baseline Unit Test 3x TensorFlow baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32447&view=logs&jobId=acbfe8aa-f0a9-5aa1-b544-bdcc7f0e36c4) | success | | ✅ | These checks are required after the changes to `neural_compressor/common/base_config.py`, `neural_compressor/common/benchmark.py`, `neural_compressor/common/utils/utility.py`, `neural_compressor/tensorflow/utils/utility.py`, `setup.py`.
🟢 Unit Tests 3x-PyTorch workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-3x-Torch](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32445) | success | | ✅ | | [UT-3x-Torch (Coverage Compare CollectDatafiles)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32445&view=logs&jobId=71384379-497b-5787-1f51-cc2e0f831d78) | success | | ✅ | | [UT-3x-Torch (Unit Test 3x Torch Unit Test 3x Torch)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32445&view=logs&jobId=ce119872-54c8-5686-93fc-b763560515d2) | success | | ✅ | | [UT-3x-Torch (Unit Test 3x Torch baseline Unit Test 3x Torch baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32445&view=logs&jobId=337dfcaa-e49a-58ed-d835-0606580c9539) | success | | ✅ | These checks are required after the changes to `neural_compressor/common/base_config.py`, `neural_compressor/common/benchmark.py`, `neural_compressor/common/utils/utility.py`, `neural_compressor/torch/algorithms/static_quant/utility.py`, `neural_compressor/torch/utils/utility.py`, `test/3x/common/test_benchmark.py`, `test/3x/common/test_utility.py`, `setup.py`.
🟢 Unit Tests 3x-ONNXRT workflow | Check ID | Status | Error details | | | -------- | ------ | ---- | --- | | [UT-3x-ONNXRT](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32448) | success | | ✅ | | [UT-3x-ONNXRT (Coverage Compare CollectDatafiles)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32448&view=logs&jobId=71384379-497b-5787-1f51-cc2e0f831d78) | success | | ✅ | | [UT-3x-ONNXRT (Unit Test 3x ONNXRT Unit Test 3x ONNXRT)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32448&view=logs&jobId=3db173ea-5f83-5cb0-6129-41ce50acfd9a) | success | | ✅ | | [UT-3x-ONNXRT (Unit Test 3x ONNXRT baseline Unit Test 3x ONNXRT baseline)](https://dev.azure.com/lpot-inc/b7121868-d73a-4794-90c1-23135f974d09/_build/results?buildId=32448&view=logs&jobId=407ef47f-0b06-5cf4-d454-c19419fc5d45) | success | | ✅ | These checks are required after the changes to `neural_compressor/common/base_config.py`, `neural_compressor/common/benchmark.py`, `neural_compressor/common/utils/utility.py`, `setup.py`, `requirements_ort.txt`.

Thank you for your contribution! 💜

Note This comment is automatically generated and will be updates every 180 seconds within the next 6 hours. If you have any other questions, contact chensuyue or XuehaoSun for help.

xin3he commented 4 months ago

demo: Multiple Instance Benchmark Summary image

xin3he commented 4 months ago

ext test: https://inteltf-jenk.sh.intel.com/job/intel-lpot-validation-top-pr-extension-3x-API/65/

xin3he commented 4 months ago

The final demo: image

xin3he commented 4 months ago

ext test: https://inteltf-jenk.sh.intel.com/job/intel-lpot-validation-top-pr-extension-3x-API/66/