unsupported instruction `vpdpbusd'

[AndrewSmithWalter]

while building mediapipe with xnnpack, error occurs.

(base) sstc@sstc-B450MH:~/0506/mediapipe$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/pose_tracking:pose_tracking_cpu --verbose_failures WARNING: /home/sstc/0506/mediapipe/mediapipe/framework/BUILD:69:24: in cc_library rule //mediapipe/framework:calculator_cc_proto: target '//mediapipe/framework:calculator_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use. WARNING: /home/sstc/0506/mediapipe/mediapipe/framework/tool/BUILD:200:24: in cc_library rule //mediapipe/framework/tool:field_data_cc_proto: target '//mediapipe/framework/tool:field_data_cc_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': Only for backward compatibility. Do not use. INFO: Analyzed target //mediapipe/examples/desktop/pose_tracking:pose_tracking_cpu (0 packages loaded, 0 targets configured). INFO: Found 1 target... ERROR: /home/sstc/.cache/bazel/_bazel_sstc/c163b43974ce0aca38520dfe717ef796/external/XNNPACK/BUILD.bazel:1881:19: Compiling src/amalgam/gen/avxvnni.c failed: (Exit 1): gcc failed: error executing command (from target @XNNPACK//:avxvnni_prod_microkernels) (cd /home/sstc/.cache/bazel/_bazel_sstc/c163b43974ce0aca38520dfe717ef796/sandbox/linux-sandbox/1919/execroot/mediapipe && \ exec env - \ LD_LIBRARY_PATH=/usr/local/cuda/lib64 \ PATH=/home/sstc/.cache/bazelisk/downloads/bazelbuild/bazel-6.1.1-linux-x86_64/bin:/home/sstc/ancconda3/envs/ux-seg/bin:/usr/local/cuda/bin:/home/sstc/anaconda3/bin:/home/sstc/anaconda3/condabin:/home/sstc/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \ PWD=/proc/self/cwd \ /usr/local/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections -MD -MF bazel-out/k8-opt/bin/external/XNNPACK/_objs/avxvnni_prod_microkernels/avxvnni.d '-frandom-seed=bazel-out/k8-opt/bin/external/XNNPACK/_objs/avxvnni_prod_microkernels/avxvnni.o' '-DBAZEL_CURRENT_REPOSITORY="XNNPACK"' -iquote external/XNNPACK -iquote bazel-out/k8-opt/bin/external/XNNPACK -isystem external/XNNPACK/include -isystem bazel-out/k8-opt/bin/external/XNNPACK/include -isystem external/XNNPACK/src -isystem bazel-out/k8-opt/bin/external/XNNPACK/src -w -Iinclude -Isrc -mavx2 -mavxvnni -mf16c -mfma '-std=c99' -O2 -fno-canonical-system-headers -Wno-builtin-macro-redefined '-DDATE="redacted"' '-DTIMESTAMP="redacted"' '-DTIME="redacted"' -c external/XNNPACK/src/amalgam/gen/avxvnni.c -o bazel-out/k8-opt/bin/external/XNNPACK/_objs/avxvnni_prod_microkernels/avxvnni.o)

Configuration: b473be029188149ae7eabfb8f3d83cdcc6b3d8de686d6278c0a5d9ee2474aa90

Execution platform: @local_execution_config_platform//:platform

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging /tmp/ccECwYKl.s: Assembler messages: /tmp/ccECwYKl.s:88: Error: unsupported instruction vpdpbusd' /tmp/ccECwYKl.s:92: Error: unsupported instructionvpdpbusd' /tmp/ccECwYKl.s:95: Error: unsupported instruction `vpdpbusd'

[fbarchard]

Thats an odd one. Your compiler must be new enough to accept -mavxvvni, and it seems it got past the compile and failed during link? Thats a new one... its using a cuda linker that doesnt understand avx-vnni? For xnnpack part of the build you can disable avxvnni with a bazel flag.. give that a try --define=xnn_enable_avxvnni=false

[alankelly]

This looks similar to https://github.com/google/XNNPACK/issues/5892

Did you compile your compiler yourself?

"From the Pytorch thread, the user is using CentOS 7 which is now quite old. gcc 10.2 was locally built but the assembler shipped with CentOS is much older. as version > 2.29 is required for vnni instructions.

Closing this as the problem does not come from XNNPack and there is nothing that can be done from the XNNPack side including better error handling."

[fbarchard]

Issue #5892 is local compilers often only support cpus for the host they are on.

I this case the cuda linker appears to not support vnni. If cuda were built on a server, it might support server cpu configurations. In CMake there is likely a way to test compiler support - so far I test compiler versions for MSVC, GCC and CLANG for a minimum version number, but that doesnt account for new compilers that have ISAs limited to the host they are on. I havent seen a way for bazel to do it. But for anything that doesnt work, the --define=xnnenable should have flags to disable it

[alankelly]

Can we close this?

[arwild1]

Just found this issue... RHEL9 ships with a gcc compiler that supports -mavxvnni but also ships with an assembler (gas from binutils-2.35) which does not. binutils gained VNNI support in 2.36. Redhat has a knowledgebase article for those with accounts: https://access.redhat.com/solutions/7049696 This would also impact all RHEL9 based distros (Rocky9, etc.)

Their recommendation is to install their "gcc-12" packages which would also installs/uses a separate binutils release (... well that or disable vnni support in your build).

I ran into this using my own compiler build (of gcc-12), but I was still relying on their default assembler/binutils. Also, in my particular case, I thought I was building targeting broadwell (so no avx512 at all), but it appears my architectural preferences aren't making their way from tensorflow through to XNNPACK's build. ...but that would be al altogether different issue.

Ultimately checking whether the compiler supports the instruction is (arguably) an incomplete check. Maybe you should actual build/link a small test application? That would fail in this case too.