search

github / codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security
https://codeql.github.com
MIT License
7.71k stars 1.55k forks source link

Problems encountered by codeql in building chromium QL library 2 #13489

Open cooolie opened 1 year ago

cooolie commented 1 year ago

According to the method mentioned here, i removed \out\debug\obj\components\autofill

codeql:2.13.3.0
error:
"../../third_party/protobuf/src/google/protobuf/arena.h", line 486: error: no instance of overloaded "operator new" matches the argument list
            argument types are: (unsigned int, void *)
        return new (ptr) T(static_cast<Args&&>(args)...);

[T 20:53:09 24244] invocation: D:\codeql\bin\codeql-win64\codeql\cpp\tools\win64\extractor.exe, args: --mimic "d:\chromium\src\third_party\llvm-build\release+asserts\bin\clang-cl.exe" /c ../../components/autofill/content/browser/content_autofill_router.cc /Foobj/components/autofill/content/browser/browser/content_autofill_router.obj /nologo /showIncludes:user "-imsvcC:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\include" "-imsvcC:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.35.32215\ATLMFC\include" "-imsvcC:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Auxiliary\VS\include" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt" -DDCHECK_ALWAYS_ON=1 -DUSE_AURA=1 -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS -D_LIBCPP_ENABLE_ASSERTIONS=1 "-DCR_CLANG_REVISION=\"llvmorg-17-init-12166-g7586aeab-3\"" -DCOMPONENT_BUILD -DCR_LIBCXX_REVISION=055b2e17ae4f0e2c025ad0c7508b01787df17758 -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_NI -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DLIBYUV_DISABLE_NEON -DWEBP_EXTERN=extern -DUSE_EGL -DVK_USE_PLATFORM_WIN32_KHR -D_WTL_NO_AUTOMATIC_NAMESPACE -DGOOGLE_PROTOBUF_NO_RTTI -DGOOGLE_PROTOBUF_NO_STATIC_INITIALIZER -DGOOGLE_PROTOBUF_INTERNAL_DONATE_STEAL_INLINE=0 -DPROTOBUF_USE_DLLS -DABSL_CONSUME_DLL -DBORINGSSL_SHARED_LIBRARY -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DUSE_CHROMIUM_ICU=1 -DU_ENABLE_TRACING=1 -DU_ENABLE_RESOURCE_TRACING=0 -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -DLEVELDB_PLATFORM_CHROMIUM=1 -DLEVELDB_SHARED_LIBRARY -DSK_ENABLE_SKSL -DSK_UNTIL_CRBUG_1187654_IS_FIXED "-DSK_USER_CONFIG_HEADER=\"../../skia/config/SkUserConfig.h\"" -DSK_WIN_FONTMGR_NO_SIMULATIONS -DSK_DISABLE_LEGACY_SKSURFACE_METHODS -DSK_DISABLE_LEGACY_GRAPHITE_IMAGE_FACTORIES -DSK_DISABLE_LEGACY_GRAPHITE_IMAGE_METHODS -DSK_DISABLE_LEGACY_SKSURFACE_FACTORIES -DSK_DISABLE_LEGACY_SKSURFACE_FLUSH -DSK_DISABLE_LEGACY_SKSURFACE_AS_IMAGE -DSK_DISABLE_LEGACY_SKSURFACE_DISPLAYLIST -DSK_CODEC_DECODES_JPEG -DSK_CODEC_DECODES_PNG -DSK_CODEC_DECODES_WEBP -DSK_ENCODE_JPEG -DSK_ENCODE_PNG -DSK_ENCODE_WEBP -DSK_HAS_WUFFS_LIBRARY -DSKIA_DLL -DSKCMS_API=__declspec(dllexport) -DGR_GL_FUNCTION_TYPE=__stdcall -DSK_GANESH "-DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\"" -DSK_GL -DSK_VULKAN=1 -DSK_GRAPHITE -DSK_DAWN -DWGPU_SHARED_LIBRARY "-DI18N_ADDRESS_VALIDATION_DATA_URL=\"https://chromium-i18n.appspot.com/ssl-aggregate-address/\"" -DCRASHPAD_ZLIB_SOURCE_EXTERNAL -DWEBRTC_ENABLE_SYMBOL_EXPORT -DWEBRTC_ENABLE_AVX2 -DRTC_ENABLE_WIN_WGC -DWEBRTC_NON_STATIC_TRACE_EVENT_HANDLERS=0 -DWEBRTC_CHROMIUM_BUILD -DWEBRTC_WIN -DABSL_ALLOCATOR_NOTHROW=1 -DLOGGING_INSIDE_WEBRTC -DV8_USE_EXTERNAL_STARTUP_DATA -DUSING_V8_SHARED -DV8_ENABLE_CHECKS -DV8_COMPRESS_POINTERS -DV8_COMPRESS_POINTERS_IN_SHARED_CAGE -DV8_31BIT_SMIS_ON_64BIT_ARCH -DV8_ENABLE_SANDBOX -DV8_DEPRECATION_WARNINGS -DCPPGC_CAGED_HEAP -DCPPGC_YOUNG_GENERATION -DCPPGC_POINTER_COMPRESSION -DCPPGC_SLIM_WRITE_BARRIER -I../.. -Igen -I../../buildtools/third_party/libc++ -I../../third_party/perfetto/include -Igen/third_party/perfetto/build_config -Igen/third_party/perfetto -I../../third_party/libyuv/include -I../../third_party/jsoncpp/source/include -Igen/third_party/dawn/include -I../../third_party/dawn/include -I../../third_party/libwebp/src/src -Igen/third_party/private_membership/src -Igen/third_party/shell-encryption/src -Igen/components/pol[E 20:53:09 20676] Warning[extractor-c++]: In index_template: [2001c5d5868] Unexpected template kind 9.
icy/proto -I../../third_party/khronos -I../../gpu -I../../third_party/vulkan-deps/vulkan-headers/src/include -I../../third_party/wtl/include -I../../third_party/protobuf/src -Igen/protoc_out -I../../third_party/abseil-cpp -I../../third_party/boringssl/src/include -I../../third_party/ipcz/include -I../../third_party/ced/src -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -I../../third_party/leveldatabase -I../../third_party/leveldatabase/src -I../../third_party/leveldatabase/src/include -I../../net/third_party/quiche/overrides -I../../net/third_party/quiche/src/quiche/common/platform/default -I../../net/third_party/quiche/src -Igen/net/third_party/quiche/src -I../../third_party/skia -Igen/third_party/skia -I../../third_party/wuffs/src/release/c -I../../third_party/vulkan/include -I../../third_party/libaddressinput/src/cpp/include -I../../third_party/crashpad/crashpad -I../../third_party/crashpad/crashpad/compat/win -I../../third_party/zlib -I../../third_party/webrtc_overrides -I../../third_party/webrtc -Igen/third_party/webrtc -I../../third_party/libwebm/source -I../../third_party/mesa_headers -I../../third_party/libaom/source/libaom -I../../v8/include -Igen/v8/include /W4 -Wimplicit-fallthrough -Wextra-semi -Wunreachable-code-aggressive -Wthread-safety /WX -Wno-missing-field-initializers -Wno-unused-parameter -Wno-psabi -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-nonportable-include-path -Wenum-compare-conditional -Wno-ignored-pragma-optimize -Wno-deprecated-builtins -Wno-bitfield-constant-conversion -Wno-deprecated-this-capture -Wshadow -fno-delete-null-pointer-checks -fno-ident -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 /clang:-ffp-contract=off -fcomplete-member-pointers /Gy /FS /bigobj /utf-8 /Zc:twoPhase -ffile-reproducible /Zc:sizedDealloc- /D__WRL_ENABLE_FUNCTION_STATICS__ -fmsc-version=1934 /Zc:dllexportInlines- -m64 -msse3 /Brepro -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern /Od /Ob0 /GF /Z7 -gno-codeview-command-line -gcodeview-ghash -Xclang -fuse-ctor-homing /guard:cf,nochecks /MDd -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ref-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang check-stack-allocated -Xclang -plugin-arg-find-bad-constructs -Xclang check-raw-ptr-fields -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=testing/platform_test.h -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=sandbox/mac/ -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=chrome/browser/ui/cocoa/touchbar/browser_window_default_touch_bar.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=device/bluetooth/test/mock_bluetooth_central_manager_mac.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=device/bluetooth/test/mock_bluetooth_cbperipheral_mac.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=ui/views/controls/native/native_view_host_mac_unittest.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=ui/views/widget/native_widget_mac_unittest.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=ui/views/controls/native/native_view_host_mac_unittest.mm -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-exclude-path=um/winnt.h -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-redundant-parens -Wno-null-pointer-subtraction -DPROTOBUF_ALLOW_DEPRECATED=1 -Wno-redundant-parens -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare /std:c++20 -Wno-trigraphs /TP /GR- -I../../buildtools/third_party/libc++/trunk/include /Fd"obj/components/autofill/content/browser/browser_cc.pdb"
[T 20:53:09 24244] trace_languages(1): [cpp]
[E 20:53:09 20676] Warning[extractor-c++]: In index_template: [2001c45a540] Unexpected template kind 9.
[E 20:53:09 20676] Warning[extractor-c++]: In index_template: [2001c459970] Unexpected template kind 9.
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_undef.inc
[E 20:53:09 26764] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/stubs/stl_util.h
[E 20:53:09 26764] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_def.inc
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/generated_message_util.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/C_/Program Files (x86)/Windows Kits/10/Include/10.0.22621.0/ucrt/assert.h
[E 20:53:09 12776] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/buildtools/third_party/libc++/trunk/include/__iterator/counted_iterator.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/stubs/once.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_def.inc
[E 20:53:09 12776] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/buildtools/third_party/libc++/trunk/include/__concepts/common_with.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_undef.inc
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/any.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/message_lite.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/metadata_lite.h
[E 20:53:09 11572] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/third_party/protobuf/src/google/protobuf/port_def.inc
[E 20:53:09 12776] Already archived D:\codeql\autofill_0617\src/D_/chromium/src/buildtools/third_party/libc++/trunk/include/__iterator/front_insert_iterator.h
"../../third_party/protobuf/src/google/protobuf/arena.h", line 486: error: no instance of overloaded "operator new" matches the argument list
            argument types are: (unsigned int, void *)
        return new (ptr) T(static_cast<Args&&>(args)...);
               ^
          detected during:
            instantiation of "T *google::protobuf::Arena::InternalHelper<T>::Construct(void *, Args &&...) [with T=autofill::StrikeData, Args=<google::protobuf::Arena *>]" at line 659
            instantiation of "T *google::protobuf::Arena::DoCreateMessage<T,Args...>(Args &&...) [with T=autofill::StrikeData, Args=<>]" at line 584
            instantiation of "T *google::protobuf::Arena::CreateMessageInternal<T>(google::protobuf::Arena *) [with T=autofill::StrikeData]" at line 269 of "gen/components/autofill/core/browser/proto/strike_data.pb.cc"

[E 20:53:09 15972] Warning[extractor-c++]: In construct_message: "../../third_party/protobuf/src/google/protobuf/arena.h", line 486: error: no instance of overloaded "operator new" matches the argument list
            argument types are: (unsigned int, void *)
        return new (ptr) T(static_cast<Args&&>(args)...);
               ^
          detected during:
            instantiation of "T *google::protobuf::Arena::InternalHelper<T>::Construct(void *, Args &&...) [with T=autofill::StrikeData, Args=<google::protobuf::Arena *>]" at line 659
            instantiation of "T *google::protobuf::Arena::DoCreateMessage<T,Args...>(Args &&...) [with T=autofill::StrikeData, Args=<>]" at line 584
            instantiation of "T *google::protobuf::Arena::CreateMessageInternal<T>(google::protobuf::Arena *) [with T=autofill::StrikeData]" at line 269 of "gen/components/autofill/core/browser/proto/strike_data.pb.cc"
cooolie commented 1 year ago

"Just to clarify, the compilation was successful, it's just that many cc files will not appear in the CodeQL database."

D:\chromium\src>codeql database create "D:/codeql/autofill_06172" --language=cpp --command="ninja -j 18 -C out/debug chrome"
Initializing database at D:\codeql\autofill_06172.
Running build command: [ninja, -j, 18, -C, out/debug, chrome]
[2023-06-17 21:09:10] [build-stdout] ninja: Entering directory `out/debug'
[2023-06-17 21:09:10] [build-stdout] [1/14] CXX obj/components/autofill/content/browser/browser/fingerprint.obj
[2023-06-17 21:09:10] [build-stdout] [2/14] CXX obj/components/autofill/content/browser/browser/bad_message.obj
[2023-06-17 21:09:10] [build-stdout] [3/14] CXX obj/components/autofill/content/browser/browser/form_forest.obj
[2023-06-17 21:09:10] [build-stdout] [4/14] CXX obj/components/autofill/content/browser/browser/content_autofill_driver.obj
[2023-06-17 21:09:10] [build-stdout] [5/14] CXX obj/components/autofill/content/browser/browser/content_autofill_router.obj
[2023-06-17 21:09:25] [build-stdout] [6/14] CXX obj/components/autofill/content/browser/browser/content_autofill_client.obj
[2023-06-17 21:09:25] [build-stdout] [7/14] CXX obj/components/autofill/content/browser/browser/autofill_log_router_factory.obj
[2023-06-17 21:09:25] [build-stdout] [8/14] CXX obj/components/autofill/content/browser/browser/content_autofill_driver_factory.obj
[2023-06-17 21:09:25] [build-stdout] [9/14] LIB obj/components/autofill/content/browser/browser.lib
[2023-06-17 21:09:25] [build-stdout] [10/14] LINK(DLL) chrome.dll chrome.dll.lib chrome.dll.pdb
Finalizing database at D:\codeql\autofill_06172.
Successfully created database at D:\codeql\autofill_06172.
aibaars commented 1 year ago

The error about "../../third_party/protobuf/src/google/protobuf/arena.h" look strange and may be related. When the CodeQL extractor.exe encounters an error it usually carries on and adds the source file to the database anyway, unless there are too many errors.

You said there were many CC files missing. Could you search for a couple of them in the build-tracer.log to see if they are mentioned at all. If they are not mentioned then they are likely not needed by the build process (they may have already been compiled earlier or are part of a different build target). If the files are mentioned in invocation: D:\codeql\bin\...\extractor.exe ... lines then there should be an explanation usually mentioning a "Catastrophic error".

Attaching the entire build-tracer log to this issue is probably not possible, but if you can attach a snippet with of the log related to the trace of a single CC file that would be great for debugging.

cooolie commented 1 year ago

build-tracer.log

Deleted Obj file list

06/17/2023  09:09 PM           130,359 autofill_log_router_factory.obj
06/17/2023  09:09 PM           687,810 bad_message.obj
06/17/2023  09:09 PM         1,074,288 content_autofill_client.obj
06/17/2023  09:09 PM         2,865,175 content_autofill_driver.obj
06/17/2023  09:09 PM         2,278,732 content_autofill_driver_factory.obj
06/17/2023  09:09 PM         3,592,710 content_autofill_router.obj
06/17/2023  09:09 PM         2,407,846 fingerprint.obj
06/17/2023  09:09 PM         3,270,934 form_forest.obj
aibaars commented 1 year ago

Thanks for providing a build-tracer log. I see a couple of things like :

66 errors and 1 catastrophic error detected in the compilation of "../../components/autofill/content/browser/risk/fingerprint.cc".
Compilation aborted.

I think these could explain why some files are missing. I'll pass it on to the team, hopefully they know what is causing the problem.

aibaars commented 1 year ago

There may be a fix for your problem in the pre-release version of CodeQL 2.13.4. Could you try if that helps? See: https://github.com/github/codeql-action/releases/tag/codeql-bundle-v2.13.4

cooolie commented 1 year ago

build-tracer.log Better than before, but there are still file processing failures, see log

Deleted Obj file list

obj\chrome\browser\ui\ui\manage_passwords_ui_controller.obj
jketema commented 1 year ago

Thanks for the updated log. The log seems to indicate that only one file was compiled, not the 8 from https://github.com/github/codeql/issues/13489#issuecomment-1595769249. Is that correct?

cooolie commented 1 year ago

yes, i only delete one obj file (obj\chrome\browser\ui\ui\manage_passwords_ui_controller.obj)

jketema commented 1 year ago

Can you delete all the relevant object files and re-try and share the log. Currently I cannot judge how severe the problem is.

cooolie commented 1 year ago

build-tracer.zip

Deleted Obj file list
obj\chrome\browser\ui\ui\*.obj

CodeQL C++ extractor: Current location: ../../chrome/browser/ui/passwords/manage_passwords_ui_controller.cc:152317,5
CodeQL C++ extractor: Current physical location: ../../base/compiler_specific.h:104,5
"../../base/compiler_specific.h", line 104: internal error: assertion failed: macro_invocation: unknown special predefined macro (macro.c, line 7359 in macro_invocation)

  #if HAS_CPP_ATTRIBUTE(no_unique_address)
      ^

[E 16:25:08 17016] Warning[extractor-c++]: In construct_message: "../../base/compiler_specific.h", line 104: internal error: assertion failed: macro_invocation: unknown special predefined macro (macro.c, line 7359 in macro_invocation)

  #if HAS_CPP_ATTRIBUTE(no_unique_address)
      ^

3 errors and 1 catastrophic error detected in the compilation of "../../chrome/browser/ui/passwords/manage_passwords_ui_controller.cc".
Compilation aborted.
jketema commented 1 year ago

Thanks for the complete log. I cannot easily reproduce this. I'll add this problem to our internal tracking issue for these kinds of problems, but I cannot promise this will be solved any time soon.

cooolie commented 1 year ago

Reproduce step: Guided by this article https://chromium.googlesource.com/chromium/src/+/HEAD/docs/windows_build_instructions.md

build args.gn

Set build arguments here. See gn help buildargs.

is_component_build = true enable_nacl = false target_cpu = "x64" blink_symbol_level = 2 v8_symbol_level = 2 symbol_level = 2

delete obj in obj\chrome\browser\ui\ui*.obj

codeql database create "D:/codeql/test" --language=cpp --command="ninja -j 18 -C out/debug chrome"

jketema commented 1 year ago

Thanks. I'm fully aware how to build Chromium. This doesn't mean it's easy to create a test case that reproduces the problem.