Chromium build snapshot missing several functions

[y0ny0ns0n]

Description of the issue

I have tried to create codeql snapshot for chromium build several times but it looks like some part of functions are missing in snapshot.

One of the example is third_party/blink/renderer/modules/ml/. Every symbol in this modules are not included in codeql snapshot.

I trying to create codeql snapshot only including symbols in third_party/blink/renderer/modules/ml as below:

$ codeql version
CodeQL command-line toolchain release 2.16.4.
Copyright (C) 2019-2024 GitHub, Inc.
....

$ codeql database create /home/y0ny0ns0n/chromium_ml --language=cpp --command="autoninja -C out/Release chrome"
Initializing database at /home/y0ny0ns0n/chromium_ml.
Running build command: [autoninja, -C, out/Release, chrome]
[2024-03-13 00:39:08] [build-stdout] ninja: Entering directory `out/Release'
[2024-03-13 00:39:20] [build-stdout] [1/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_trace.o
[2024-03-13 00:39:21] [build-stdout] [2/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_operator.o
[2024-03-13 00:39:21] [build-stdout] [3/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_error_mojo.o
[2024-03-13 00:39:21] [build-stdout] [4/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_operand.o
[2024-03-13 00:39:21] [build-stdout] [5/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_activation.o
[2024-03-13 00:39:22] [build-stdout] [6/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_buffer_mojo.o
[2024-03-13 00:39:23] [build-stdout] [7/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_model.o
[2024-03-13 00:39:23] [build-stdout] [8/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_context.o
[2024-03-13 00:39:23] [build-stdout] [9/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_model_loader.o
[2024-03-13 00:39:23] [build-stdout] [10/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_graph_mojo.o
[2024-03-13 00:39:23] [build-stdout] [11/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml.o
[2024-03-13 00:39:23] [build-stdout] [12/50] CXX obj/third_party/blink/renderer/modules/ml/ml/navigator_ml.o
[2024-03-13 00:39:24] [build-stdout] [13/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_buffer.o
[2024-03-13 00:39:24] [build-stdout] [14/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_graph_type_converter.o
[2024-03-13 00:39:24] [build-stdout] [15/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_graph_utils.o
[2024-03-13 00:39:25] [build-stdout] [16/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_graph.o
[2024-03-13 00:39:25] [build-stdout] [17/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_graph_builder.o
[2024-03-13 00:39:26] [build-stdout] [18/50] CXX obj/third_party/blink/renderer/modules/ml/ml/ml_graph_xnnpack.o
[2024-03-13 00:39:26] [build-stdout] [19/50] STAMP obj/third_party/blink/renderer/modules/ml/ml.stamp
[2024-03-13 00:39:26] [build-stdout] [20/50] SOLINK ./libblink_modules.so
[2024-03-13 00:39:27] [build-stdout] [21/23] ACTION //tools/v8_context_snapshot:generate_v8_context_snapshot(//build/toolchain/linux:clang_x64)
[2024-03-13 00:39:27] [build-stdout] [22/23] STAMP obj/tools/v8_context_snapshot/generate_v8_context_snapshot.stamp
[2024-03-13 00:39:27] [build-stdout] [23/23] STAMP obj/tools/v8_context_snapshot/v8_context_snapshot.stamp
Finalizing database at /home/y0ny0ns0n/chromium_ml.
Running pre-finalize script /home/y0ny0ns0n/codeql/cpp/tools/pre-finalize.sh in /home/y0ny0ns0n/chromium/src.
Running TRAP import for CodeQL database at /home/y0ny0ns0n/chromium_ml...
Grouping TRAP files by link target
Grouping unlinked TRAP files together
Scanning TRAP files
Assembling TRAP files
Importing TRAP files
Merging relations
Finished writing database (relations: 75.65 KiB; string pool: 2.05 MiB).
TRAP import complete (744ms).
Finished zipping source archive (2.77 MiB).
Successfully created database at /home/y0ny0ns0n/chromium_ml.

But it doesn't worked.

These are log files. database-create-20240313.003859.263.log build-tracer.log

After reading the log, I suspect it could be some kind of code parsing error of the codeql extractor. Because every catastrophic error looks like related to some code parsing functionality like mark_decl_after_first_in_comma_list.

....
CodeQL C++ extractor: Current location: ../../third_party/blink/renderer/modules/ml/ml_trace.cc:193756,66
CodeQL C++ extractor: Current physical location: ../../base/functional/function_ref.h:69,66
"../../base/functional/function_ref.h", line 69: internal error: assertion failed at: "decls.c", line 21498 in mark_decl_after_first_in_comma_list

        std::convertible_to<internal::ExtractReturnType<RunType>, R> &&
                                                                  ^

[E 00:39:20 2270181] Warning[extractor-c++]: In construct_text_message: "../../base/functional/function_ref.h", line 69: internal error: assertion failed at: "decls.c", line 21498 in mark_decl_after_first_in_comma_list

        std::convertible_to<internal::ExtractReturnType<RunType>, R> &&
                                                                  ^

14 errors and 1 catastrophic error detected in the compilation of "../../third_party/blink/renderer/modules/ml/ml_trace.cc".
....
[E 00:39:21 2270038] Warning[extractor-c++]: In /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x746c85c29e40]
construct_text_message/home/y0ny0ns0n/codeql/cpp/tools/linux64/extractor(+0x3b5e09)[0x58943281fe09]
: "../../base/functional/bind_internal.h", line 1237: error: parameter pack "BoundArgs" was referenced but not expanded
    template <bool v = !HasRefCountedTypeAsRawPtr<std::decay_t<BoundArgs>...>>
                                                               ^

[E 00:39:21 2270588] Already archived /home/y0ny0ns0n/chromium_ml/src/home/y0ny0ns0n/chromium/src/third_party/libc++/src/include/__type_traits/is_constructible.h
[E 00:39:21 2270246] Finished compilation TRAP /home/y0ny0ns0n/chromium_ml/trap/cpp/compilations/20/61996404_0.trap.br
CodeQL C++ extractor: Current location: ../../third_party/blink/renderer/modules/ml/webnn/ml_operator.cc:205256,66
CodeQL C++ extractor: Current physical location: ../../base/functional/function_ref.h:69,66
"../../base/functional/function_ref.h", line 69: internal error: assertion failed at: "decls.c", line 21498 in mark_decl_after_first_in_comma_list

        std::convertible_to<internal::ExtractReturnType<RunType>, R> &&
                                                                  ^

[E 00:39:21 2270246] Warning[extractor-c++]: In construct_text_message: "../../base/functional/function_ref.h", line 69: internal error: assertion failed at: "decls.c", line 21498 in mark_decl_after_first_in_comma_list

        std::convertible_to<internal::ExtractReturnType<RunType>, R> &&
                                                                  ^

14 errors and 1 catastrophic error detected in the compilation of "../../third_party/blink/renderer/modules/ml/webnn/ml_operator.cc".
Compilation aborted.
....

I also tested with other modules in Chromium and it looks like most of the missing modules have the same catastrophic error as above.

[jketema]

Hi @y0ny0ns0n,

Thanks for the report. This bug in our tooling. Unfortunately, due to the size of Chromium, resolving this issue is not a priority for us, and for now I have just recorded the issue in our internal tracking issue for Chromium.

If you would like to have this resolved relatively quickly, then it would help us if you're able to come up with and share a small reproducing test case that does not depend on any chromium code.

[jketema]

Reported to our frontend provider.

[jketema]

Fixed in CodeQL 2.18.2, which should be released within the next few weeks.