search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.51k stars 1.49k forks source link

General issue - File not indexed while create xnu database #11866

Open pwn0rz opened 1 year ago

pwn0rz commented 1 year ago

File not indexed while create xnu database

Hi, I am trying to create a CodeQL database for the latest macOS XNU kernel using this script. However I found that only a few files are indexed in the src.zip. For example

unzip -l src.zip | grep installfile

returns nothing though this file did get compiled.

I have searched log/build-tracer.log and here are some info that might help.

[E 12:54:54 14540] Processed command line: /opt/homebrew/Caskroom/codeql/2.11.6/codeql/cpp/tools/osx64/extractor --trapfolder /Users/macuser/Developer/tools/xnu-build/xnu-codeql/trap/cpp --src_archive /Users/macuser/Developer/tools/xnu-build/xnu-codeql/src --mimic_config /Users/macuser/Developer/tools/xnu-build/xnu-codeql/working/compiler_mimic_cache/13227e657457 --executable_filename /Users/macuser/Developer/tools/xnu-build/build/xnu-codeql.obj/SETUP/installfile/installfile.o -w --error_limit 1000 --disable_system_macros --variadic_macros --gcc --gnu_version 40000 --clang --isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk -I/Users/macuser/Developer/tools/xnu-build/xnu/SETUP/installfile/ -I. --blocks -D__GCC_HAVE_DWARF2_CFI_ASM=1 -- installfile.c /Users/macuser/Developer/tools/xnu-build/xnu/SETUP/installfile/installfile.c
[E 12:54:54 14540] Starting compilation TRAP /Users/macuser/Developer/tools/xnu-build/xnu-codeql/trap/cpp/compilations/12/53291551_0.trap.br
[T 12:54:54 14874] Lua: === Intercepted call to /applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang ===
[T 12:54:54 14874] Executing the following tracer actions:
[T 12:54:54 14874] Tracer actions:
[T 12:54:54 14874] pre_invocations(0)
[T 12:54:54 14874] post_invocations(1)
[T 12:54:54 14874] invocation: /opt/homebrew/Caskroom/codeql/2.11.6/codeql/cpp/tools/osx64/extractor, args: [--mimic, /applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang, -mrelax-all, -main-file-name, -mrelocation-model, -mframe-pointer=all, -fno-strict-return, -fno-rounding-math, -funwind-tables=2, -target-sdk-version=13.1, -fvisibility-inlines-hidden-static-local-var, -target-cpu, -target-linker-version, -isysroot, /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk, -O0, -fdebug-compilation-dir=/Users/macuser/Developer/tools/xnu-build/build/xnu-codeql.obj/SETUP/kextsymboltool, -ferror-limit, -fstack-check, -mdarwin-stkchk-strong-link, -fblocks, -fencode-extended-block-signature, -fregister-global-dtors-with-atexit, -fgnuc-version=4.2.1, -fmax-type-align=16, -fcommon, -fno-odr-hash-protocols, -x, c, -o, /Users/macuser/Developer/tools/xnu-build/xnu-codeql/working/tmp//semmle_16_14564_738433.o, -c, /Users/macuser/Developer/tools/xnu-build/xnu-codeql/working/tmp//semmle_15_14564_738376.c]
[T 12:54:54 14874] trace_languages(1): [cpp]
[T 12:54:54 14876] Initializing tracer.
[T 12:54:54 14876] Initialising tags...
[T 12:54:54 14879] Initializing tracer.
[T 12:54:54 14879] Initialising tags...
[T 12:54:54 14885] Attempting to switch stdout/stderr to 10...
installfile.c:
[T 12:54:54 14876] ID set to 0000000000003A1C_0000000000000001 (parent 00000000000038D8_0000000000000001)
[T 12:54:54 14876] ==== Candidate to intercept: /applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang (canonical: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang) ====
[T 12:54:54 14876] Lua: === Intercepted call to /applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang ===
Catastrophic error: cannot open source file "installfile.c"

[T 12:54:54 14876] Executing the following tracer actions:
[T 12:54:54 14876] Tracer actions:
[T 12:54:54 14876] pre_invocations(0)
[T 12:54:54 14876] post_invocations(1)
[T 12:54:54 14876] invocation: /opt/homebrew/Caskroom/codeql/2.11.6/codeql/cpp/tools/osx64/extractor, args: [--mimic, /applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang, -mrelax-all, -main-file-name, -mrelocation-model, -mframe-pointer=all, -fno-strict-return, -fno-rounding-math, -funwind-tables=2, -target-sdk-version=13.1, -fvisibility-inlines-hidden-static-local-var, -target-cpu, -target-linker-version, -isysroot, /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.1.sdk, -O0, -fdebug-compilation-dir=/Users/macuser/Developer/tools/xnu-build/build/xnu-codeql.obj/SETUP/setsegname, -ferror-limit, -fstack-check, -mdarwin-stkchk-strong-link, -fblocks, -fencode-extended-block-signature, -fregister-global-dtors-with-atexit, -fgnuc-version=4.2.1, -fmax-type-align=16, -fcommon, -fno-odr-hash-protocols, -x, c, -o, /Users/macuser/Developer/tools/xnu-build/xnu-codeql/working/tmp//semmle_16_14552_739499.o, -c, /Users/macuser/Developer/tools/xnu-build/xnu-codeql/working/tmp//semmle_15_14552_739458.c]
[T 12:54:54 14876] trace_languages(1): [cpp]
[E 12:54:54 14540] Warning[extractor-c++]: In construct_message: Catastrophic error: cannot open source file "installfile.c"

1 catastrophic error detected in the compilation of "installfile.c".
Compilation terminated.

I have checked the following things:

I am puzzled by the first argument after -- , context:

extractor ... -- installfile.c /Users/macuser/Developer/tools/xnu-build/xnu/SETUP/installfile/installfile.c

I am not sure whether that caused the problem.

jketema commented 1 year ago

Hi @pwn0rz

Thanks for your report. May I ask how you're building XNU? We have seen problems before where building XNU in a certain way with CodeQL would lead to problems like this. See also here: https://github.com/github/codeql-cli-binaries/issues/145

pwn0rz commented 1 year ago

Hi @pwn0rz

Thanks for your report. May I ask how you're building XNU? We have seen problems before where building XNU in a certain way with CodeQL would lead to problems like this. See also here: github/codeql-cli-binaries#145

Hi @jketema

I'am using this script to build xnu-8792.61.2 : https://github.com/pwn0rz/xnu-build And here are enviroment info

$ sw_vers
ProductName:        macOS
ProductVersion:     13.1
BuildVersion:       22C6

$ uname -v
Darwin Kernel Version 22.2.0: Fri Nov 11 02:03:51 PST 2022; root:xnu-8792.61.2~4/RELEASE_ARM64_T6000

$ xcodebuild -version
Xcode 14.2
Build version 14C18

$ /usr/bin/clang --version
Apple clang version 14.0.0 (clang-1400.0.29.202)
Target: arm64-apple-darwin22.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

# brew install codeql
$ codeql --version
CodeQL command-line toolchain release 2.11.6.
Copyright (C) 2019-2022 GitHub, Inc.
Unpacked in: /opt/homebrew/Caskroom/codeql/2.11.6/codeql
   Analysis results depend critically on separately distributed query and
   extractor modules. To list modules that are visible to the toolchain,
   use 'codeql resolve qlpacks' and 'codeql resolve languages'.
jketema commented 1 year ago

So this seems similar to the problem described in the other issue I linked to.

In ql.py instead of

BUILD_CMD = f'make install SDKROOT=macosx ARCH_CONFIGS=ARM64 KERNEL_CONFIGS=RELEASE MACHINE_CONFIGS=VMAPPLE BUILD_WERROR=0 BUILD_LTO=0 SRCROOT={SRCROOT} OBJROOT={OBJROOT} SYMROOT={SYMROOT} DSTROOT={DSTROOT} FAKEROOT={FAKEROOT_DIR} KDKROOT={KDKROOT}'

could you try:

BUILD_CMD = f'python3 x.py'

and see if that resolves the problem?

pwn0rz commented 1 year ago

could you try:

BUILD_CMD = f'python3 x.py'

and see if that resolves the problem?

Yes, it did work!

jketema commented 1 year ago

Yes, it did work!

That's great to hear. We'll track this internally, but given that there's a workaround there doesn't seem to be immediate urgency to fix this.