arm_neon.h missing from clang

[patrickchin]

including arm_neon.h always shows up as an error:

My local clang on my Mac M1 includes the arm_neon.h file, and I can compile my project locally.

$ ls /Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/arm_neon.h
/Library/Developer/CommandLineTools/usr/lib/clang/15.0.0/include/arm_neon.h

However the YCM version of clang does not:

ls /Users/patchin/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/clang/lib/clang/17.0.1/include/arm_*
/Users/patchin/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/clang/lib/clang/17.0.1/include/arm_acle.h
/Users/patchin/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/clang/lib/clang/17.0.1/include/arm_neon_sve_bridge.h
/Users/patchin/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/clang/lib/clang/17.0.1/include/arm_cmse.h

It seems these headers need to be built but are not: https://github.com/llvm/llvm-project/blob/main/clang/lib/Headers/CMakeLists.txt#L379

the file arm_neon_sve_bridge.h also includes the arm_neon.h file which doesn't exist

My :YcmDebugInfo

:YcmD
YcmDebugInfo  YcmDiags
Printing YouCompleteMe debug information...
-- Resolve completions: Up front
-- Client logfile: /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/ycm_y4ldp0y0.log
-- Server Python interpreter: /opt/homebrew/opt/python@3.11/bin/python3.11
-- Server Python version: 3.11.6
-- Server has Clang support compiled in: True
-- Clang version: clang version 17.0.1 (https://github.com/ycm-core/llvm f0e018b8aadf1be25e3af0daa7a87d505de67957)
-- No extra configuration file found
-- C-family completer debug information:
--   Clangd running
--   Clangd process ID: 36225
--   Clangd executable: ['/Users/patchin/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/clangd/output/bin/clangd', '-header-insertion-decorators=0', '-resource-dir=/Users/patchin/.vim/plugged/YouCompleteMe/third_party/ycmd/third_party/clang/lib
/clang/17.0.1', '-limit-results=500']
--   Clangd logfiles:
--     /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/clangd_stderrpujipwgw.log
--   Clangd Server State: Initialized
--   Clangd Project Directory: /Users/patchin/Workspace/media-framework
--   Clangd Open Workspaces: {'/Users/patchin/Workspace/media-framework'}
--   Clangd Settings: {}
--   Clangd Compilation Command: False
-- Server running at: http://127.0.0.1:55943
-- Server process ID: 36218
-- Server logfiles:
--   /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/ycmd_55943_stdout_cfj40g1z.log
--   /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/ycmd_55943_stderr_fw7dxnu6.log
-- Semantic highlighting supported: True
-- Virtual text supported: True
-- Popup windows supported: True

[bstaletic]

Hi! Unfortunately, I don't have a mac in order to test this, so please bear with me.

It seems these headers need to be built but are not: https://github.com/llvm/llvm-project/blob/main/clang/lib/Headers/CMakeLists.txt#L379

Hmm... we're packaging LLVM things using this script: https://github.com/ycm-core/llvm/blob/master/package_llvm.py

And we do say -DLLVM_TARGETS_TO_BUILD=all so it should be working? Yet, clearly arm_neon.h is missing... if you can help me figure out how to repackage LLVM, we could fix this in a day.

Another thing to try: Install clangd through your system package manager and set g:ycm_clangd_binary_path. That should pick up your system's "resource directory" instead of relying on what ycmd is bundling.

EDIT: If you're looking at our package_llvm.py script, keep in mind that we're actually cross-compiling the entire LLVM toolchain. Historically, this is because, on linux, we do not want to link against libtinfo. For various reasons. Cross-compiling for Apple ARM was done because back then upstream did not provide binaries for M1. In the end, it was more convenient to automate the whole thing, than to rely on me manually doing this repackaging and uploading.

[bstaletic]

Another thing to check:

1. Put let g:ycm_log_level='debug' in your vimrc.
2. Upload the clangd stderr log, accessible from :YcmToggleLogs.

I'm mostly interested in what clangd will say on the line starting with Driver produced command, but there might be other things in there as well.

[patrickchin]

My clangd logfile: clangd_stderrstijhzxy.log

I'll try running package_llvm.py on my mac to see what it produces. I'm halfway through it atm and the header file exists:

patchin@PATCHIN-M-TF7F ~/Workspace/llvm (master) $ ls
README.md          arm64-apple-darwin package_llvm.py    requirements.txt   toolchain_files    tox.ini            upload_clang.py
patchin@PATCHIN-M-TF7F ~/Workspace/llvm (master) $ find . -name "arm_neon.h"
./arm64-apple-darwin/llvm_build/tools/clang/lib/Headers/arm_neon.h
./arm64-apple-darwin/llvm_build/lib/clang/17/include/arm_neon.h

Another thing to try: Install clangd through your system package manager and set g:ycm_clangd_binary_path. That should pick up your system's "resource directory" instead of relying on what ycmd is bundling.

Would this require building YCM with --system-libclang?

[patrickchin]

Install clangd through your system package manager and set g:ycm_clangd_binary_path. That should pick up your system's "resource directory" instead of relying on what ycmd is bundling.

this works nicely actually

Printing YouCompleteMe debug information...
-- Resolve completions: Up front
-- Client logfile: /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/ycm_qyaub49i.log
-- Server Python interpreter: /opt/homebrew/opt/python@3.11/bin/python3.11
-- Server Python version: 3.11.6
-- Server has Clang support compiled in: True
-- Clang version: clang version 17.0.1 (https://github.com/ycm-core/llvm f0e018b8aadf1be25e3af0daa7a87d505de67957)
-- No extra configuration file found
-- C-family completer debug information:
--   Clangd running
--   Clangd process ID: 9675
--   Clangd executable: ['/opt/homebrew/opt/llvm/bin/clangd', '-header-insertion-decorators=0', '-limit-results=500', '-log=verbose']
--   Clangd logfiles:
--     /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/clangd_stderrt2jx2wxl.log
--   Clangd Server State: Initialized
--   Clangd Project Directory: /Users/patchin
--   Clangd Open Workspaces: {'/Users/patchin'}
--   Clangd Settings: {}
--   Clangd Compilation Command: False
-- Server running at: http://127.0.0.1:53708
-- Server process ID: 9589
-- Server logfiles:
--   /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/ycmd_53708_stdout_y6_m7p7p.log
--   /var/folders/nc/vzmrvhj95ldb1p36bmvgshg40000gn/T/ycmd_53708_stderr_qjxzqly7.log
-- Semantic highlighting supported: True
-- Virtual text supported: True
-- Popup windows supported: True

Clangd logs from changing g:ycm_clangd_binary_path

I[03:03:33.503] Homebrew clangd version 17.0.6
I[03:03:33.504] Features: mac+xpc
I[03:03:33.504] PID: 11590
I[03:03:33.504] Working directory: /Users/patchin
I[03:03:33.504] argv[0]: /opt/homebrew/opt/llvm/bin/clangd
I[03:03:33.504] argv[1]: -header-insertion-decorators=0
I[03:03:33.504] argv[2]: -limit-results=500
I[03:03:33.504] argv[3]: -log=verbose
V[03:03:33.504] User config file is /Users/patchin/Library/Preferences/clangd/config.yaml
I[03:03:33.504] Starting LSP over stdin/stdout
V[03:03:33.504] <<< {"id":1,"jsonrpc":"2.0","method":"initialize","params":{"capabilities":{"textDocument":{"codeAction":{"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["","quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}}},"completion":{"completionItem":{"documentationFormat":["plaintext","markdown"],"resolveSupport":{"properties":["documentation","detail"]}},"completionItemKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25]}},"documentSymbol":{"hierarchicalDocumentSymbolSupport":false,"labelSupport":false,"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"hover":{"contentFormat":["plaintext","markdown"]},"inlay_hint":{},"semanticTokens":{"augmentSyntaxTokens":true,"formats":["relative"],"requests":{"full":{"delta":false},"range":true},"tokenModifiers":[],"tokenTypes":["namespace","type","class","enum","interface","struct","typeParameter","parameter","variable","property","enumMember","event","function","method","member","macro","keyword","modifier","comment","string","number","regexp","operator"]},"signatureHelp":{"signatureInformation":{"documentationFormat":["plaintext","markdown"],"parameterInformation":{"labelOffsetSupport":true}}},"synchronization":{"didSave":true}},"workspace":{"applyEdit":true,"didChangeWatchedFiles":{"dynamicRegistration":true},"symbol":{"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"workspaceEdit":{"documentChanges":true},"workspaceFolders":true}},"initializationOptions":{},"processId":11580,"rootPath":"/Users/patchin","rootUri":"file:///Users/patchin","workspaceFolders":[]}}

I[03:03:33.505] <-- initialize(1)
V[03:03:33.506] Invoking /usr/bin/xcrun to find clang installation
V[03:03:33.578] Invoking /usr/bin/xcrun to find clang installation
I[03:03:33.638] --> reply:initialize(1) 133 ms
V[03:03:33.638] >>> {"id":1,"jsonrpc":"2.0","result":{"capabilities":{"astProvider":true,"callHierarchyProvider":true,"clangdInlayHintsProvider":true,"codeActionProvider":{"codeActionKinds":["quickfix","refactor","info"]},"compilationDatabase":{"automaticReload":true},"completionProvider":{"resolveProvider":false,"triggerCharacters":[".","<",">",":","\"","/","*"]},"declarationProvider":true,"definitionProvider":true,"documentFormattingProvider":true,"documentHighlightProvider":true,"documentLinkProvider":{"resolveProvider":false},"documentOnTypeFormattingProvider":{"firstTriggerCharacter":"\n","moreTriggerCharacter":[]},"documentRangeFormattingProvider":true,"documentSymbolProvider":true,"executeCommandProvider":{"commands":["clangd.applyFix","clangd.applyTweak"]},"foldingRangeProvider":true,"hoverProvider":true,"implementationProvider":true,"inactiveRegionsProvider":true,"inlayHintProvider":true,"memoryUsageProvider":true,"referencesProvider":true,"renameProvider":true,"selectionRangeProvider":true,"semanticTokensProvider":{"full":{"delta":true},"legend":{"tokenModifiers":["declaration","definition","deprecated","deduced","readonly","static","abstract","virtual","dependentName","defaultLibrary","usedAsMutableReference","usedAsMutablePointer","constructorOrDestructor","userDefined","functionScope","classScope","fileScope","globalScope"],"tokenTypes":["variable","variable","parameter","function","method","function","property","variable","class","interface","enum","enumMember","type","type","unknown","namespace","typeParameter","concept","type","macro","modifier","operator","bracket","label","comment"]},"range":false},"signatureHelpProvider":{"triggerCharacters":["(",")","{","}","<",">",","]},"standardTypeHierarchyProvider":true,"textDocumentSync":{"change":2,"openClose":true,"save":true},"typeDefinitionProvider":true,"typeHierarchyProvider":true,"workspaceSymbolProvider":true},"serverInfo":{"name":"clangd","version":"Homebrew clangd version 17.0.6 mac+xpc arm64-apple-darwin23.4.0"}}}

V[03:03:33.640] <<< {"jsonrpc":"2.0","method":"initialized","params":{}}

I[03:03:33.640] <-- initialized
V[03:03:33.641] <<< {"jsonrpc":"2.0","method":"workspace/didChangeConfiguration","params":{"settings":{}}}

I[03:03:33.641] <-- workspace/didChangeConfiguration
V[03:03:33.642] <<< {"jsonrpc":"2.0","method":"textDocument/didOpen","params":{"textDocument":{"languageId":"cpp","text":"#include \n#include \n\nint main()\n{\n    printf(\"asldkfjas\");\n    return 0;\n}\n","uri":"file:///Users/patchin/a.cpp","version":1}}}

I[03:03:33.642] <-- textDocument/didOpen
I[03:03:33.643] Failed to find compilation database for /Users/patchin/a.cpp
I[03:03:33.643] ASTWorker building file /Users/patchin/a.cpp version 1 with command clangd fallback
[/Users/patchin]
/Library/Developer/CommandLineTools/usr/bin/clang -resource-dir=/opt/homebrew/Cellar/llvm/17.0.6_1/lib/clang/17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -- /Users/patchin/a.cpp
V[03:03:33.645] Driver produced command: cc1 -cc1 -triple arm64-apple-macosx14.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -fsyntax-only -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name a.cpp -mrelocation-model pic -pic-level 2 -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -funwind-tables=1 -target-sdk-version=14.4 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu apple-m1 -target-feature +v8.5a -target-feature +aes -target-feature +crc -target-feature +dotprod -target-feature +fp-armv8 -target-feature +fp16fml -target-feature +lse -target-feature +ras -target-feature +rcpc -target-feature +rdm -target-feature +sha2 -target-feature +sha3 -target-feature +neon -target-feature +zcm -target-feature +zcz -target-feature +fullfp16 -target-abi darwinpcs -debugger-tuning=lldb -target-linker-version 1022.1 -fcoverage-compilation-dir=/Users/patchin -resource-dir /opt/homebrew/Cellar/llvm/17.0.6_1/lib/clang/17 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /opt/homebrew/Cellar/llvm/17.0.6_1/lib/clang/17/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -fdeprecated-macro -fdebug-compilation-dir=/Users/patchin -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fmax-type-align=16 -no-round-trip-args -D__GCC_HAVE_DWARF2_CFI_ASM=1 -x c++ /Users/patchin/a.cpp
V[03:03:33.645] Building first preamble for /Users/patchin/a.cpp version 1
I[03:03:33.715] Built preamble of size 2641180 for file /Users/patchin/a.cpp version 1 in 0.07 seconds
I[03:03:33.716] Indexing c++17 standard library in the context of /Users/patchin/a.cpp
V[03:03:33.720] indexed file AST for /Users/patchin/a.cpp version 1:
  symbol slab: 1 symbols, 4448 bytes
  ref slab: 2 symbols, 2 refs, 4272 bytes
  relations slab: 0 relations, 24 bytes
V[03:03:33.720] Build dynamic index for main-file symbols with estimated memory usage of 11576 bytes
I[03:03:33.720] --> textDocument/publishDiagnostics
V[03:03:33.720] >>> {"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"diagnostics":[],"uri":"file:///Users/patchin/a.cpp","version":1}}

V[03:03:33.737] indexed preamble AST for /Users/patchin/a.cpp version 1:
  symbol slab: 6513 symbols, 1711840 bytes
  ref slab: 0 symbols, 0 refs, 128 bytes
  relations slab: 0 relations, 24 bytes
V[03:03:33.755] Build dynamic index for header symbols with estimated memory usage of 4588884 bytes
I[03:03:35.103] Indexed c++17 standard library: 10861 symbols, 2302 filtered
V[03:03:35.144] Build dynamic index for header symbols with estimated memory usage of 9940636 bytes

[bstaletic]

I envy your compile times!

Would this require building YCM with --system-libclang?

No, that's a legacy thing for people using libclang-powered completer.

I'll try running package_llvm.py on my mac to see what it produces. I'm halfway through it atm and the header file exists:

I would love to figure out what went wrong in the first place. arm_neon.h is missing from the linux archives as well...

Maybe it's the update_clang_headers.h that messes things up? Its job is to download the archive produced by package_llvm.py and extract the relevant headers. Could you try running that script as well?

Normally, it downloads the archive as uploaded to ycm-core/llvm, but you can just point it at the archive package_llvm.py has created. If you need help, feel free to drop by to our gitter channel.

[patrickchin]

it's in the install dir

patchin@PATCHIN-M-TF7F ~/Workspace/llvm (master) $ find arm64-apple-darwin/llvm_install -name arm_neon.h
arm64-apple-darwin/llvm_install/lib/clang/17/include/arm_neon.h

but missing from the archive wrong archive

patchin@PATCHIN-M-TF7F ~/Workspace/llvm (master) $ tar -tf arm64-apple-darwin/llvm-project-17.0.1.src.tar.xz | grep arm_neon
llvm-project-17.0.1.src/clang/include/clang/Basic/arm_neon.td
llvm-project-17.0.1.src/clang/include/clang/Basic/arm_neon_incl.td
llvm-project-17.0.1.src/clang/lib/Headers/arm_neon_sve_bridge.h
llvm-project-17.0.1.src/clang/test/CodeGen/arm_neon_intrinsics.c
llvm-project-17.0.1.src/clang/utils/convert_arm_neon.py

yes it's in the archive

patchin@PATCHIN-M-TF7F ~/Workspace/llvm (master) $ tar -tf arm64-apple-darwin/clang+llvm-17.0.1-arm64-apple-darwin.tar.xz | grep arm_neon
clang+llvm-17.0.1-arm64-apple-darwin/include/clang/Basic/arm_neon.inc
clang+llvm-17.0.1-arm64-apple-darwin/lib/clang/17/include/arm_neon_sve_bridge.h
clang+llvm-17.0.1-arm64-apple-darwin/lib/clang/17/include/arm_neon.h

after running ./package_llvm.py --target-architecture arm64 17.0.1 --gh-user a --gh-token a

[bstaletic]

if it's missing from the archive already, this wouldn't be the issue right?

You're absolutely right and your results make me confused.