[Feature Request]: Consistent support for large (>1024) number of cores (i.e. dynamically sized cpu_set)

hpcpony commented 2 months ago

Request Description

2024.3 does not consistently handle large numbers of core w.r.t. cpu_set. thread_affinity.cpp does use dynamically sized cpu sets, but other source files (lin_system_conf.cpp, system_conf.cpp) do not.

Feature Use Case

Would support use on systems that have large (>1024) cores.

Issue submission checklist

[X] The feature request or improvement must be related to OpenVINO

wangleis commented 2 months ago

@hpcpony OpenVINO is the library of one application process.

May I know what's the use case and platform information in your requirement?

hpcpony commented 2 months ago

I think pretty much any use case and any (linux) platform that has more than 1024 cores/threads will suffice, but if you need to see an example:

$ cat prop.py

import openvino as ov
core = ov.Core()
print(core.get_property("CPU","SUPPORTED_PROPERTIES"))

$ python prop.py
Traceback (most recent call last):
  File "/home/hpcpony/openvino/prop.py", line 4, in <module>
    print(core.get_property("CPU","SUPPORTED_PROPERTIES"))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Exception from src/inference/src/cpp/core.cpp:214:
Exception from src/inference/src/dev/core_impl.cpp:708:
Failed to create plugin /opt/openvino-2024.2.0/lib/openvino-2024.2.0/libopenvino_intel_cpu_plugin.so for device CPU
Please, check your environment
Exception from src/plugins/intel_cpu/src/plugin.cpp:633:
Exception from src/inference/src/os/lin/lin_system_conf.cpp:276:
CPU affinity check failed. No CPU is eligible to run inference.

Platform was an HPE Superdome Flex that I had a chance to try out.

If it matters, this was OpenVINO 2024.2 but I just looked at the source on GitHub and at least one of the places (src/os/lin/lin_system_conf.cpp (check_valid_cpu)) is still non-dynamic.

wangleis commented 2 months ago

@hpcpony Could you share the logs of lscpu and lscpu -e on your platform?

sunxiaoxia2022 commented 1 month ago

@hpcpony Could you try this PR26303 on your machine, thank you!

sunxiaoxia2022 commented 1 month ago

@hpcpony PR26303 has been updated, please take a look, thank you！

hpcpony commented 1 month ago

I'm not sure exactly what copy of your fixes I got (my git skills are pretty limited), but I was able to build a CPU only version of OpenVino

$ git log
commit 51e39eea83b997b2293a3f43b6f5a9e5c826edc1 (HEAD -> xiaoxia/cpu_set_1024, origin/xiaoxia/cpu_set_1024)
Author: sunxiaoxia2022 <xiaoxia.sun@intel.com>
Date:   Tue Sep 3 00:06:52 2024 +0800

    change name

commit e3834af754bc672342c58c77da0fac2bf2c10e20
Author: sunxiaoxia2022 <xiaoxia.sun@intel.com>
Date:   Tue Sep 3 00:04:56 2024 +0800

    remove get_affinity

commit 89d9e51f7414bb9609209ea50e1243ac70b41489
Author: sunxiaoxia2022 <xiaoxia.sun@intel.com>
Date:   Fri Aug 30 03:37:44 2024 +0800

    support machines with more than 1024 cores
  .
  .
  .

Then running the little program that failed before now seems to work.

$ python prop.py
{'SUPPORTED_PROPERTIES': 'RO', 'AVAILABLE_DEVICES': 'RO', 'RANGE_FOR_ASYNC_INFER_REQUESTS': 'RO', 'RANGE_FOR_STREAMS': 'RO', 'EXECUTION_DEVICES': 'RO', 'FULL_DEVICE_NAME': 'RO', 'OPTIMIZATION_CAPABILITIES': 'RO', 'DEVICE_TYPE': 'RO', 'DEVICE_ARCHITECTURE': 'RO', 'NUM_STREAMS': 'RW', 'INFERENCE_NUM_THREADS': 'RW', 'PERF_COUNT': 'RW', 'INFERENCE_PRECISION_HINT': 'RW', 'PERFORMANCE_HINT': 'RW', 'EXECUTION_MODE_HINT': 'RW', 'PERFORMANCE_HINT_NUM_REQUESTS': 'RW', 'ENABLE_CPU_PINNING': 'RW', 'SCHEDULING_CORE_TYPE': 'RW', 'MODEL_DISTRIBUTION_POLICY': 'RW', 'ENABLE_HYPER_THREADING': 'RW', 'DEVICE_ID': 'RW', 'CPU_DENORMALS_OPTIMIZATION': 'RW', 'LOG_LEVEL': 'RW', 'CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE': 'RW', 'DYNAMIC_QUANTIZATION_GROUP_SIZE': 'RW', 'KV_CACHE_PRECISION': 'RW', 'AFFINITY': 'RW'}

Thanks.

openvinotoolkit / openvino