Missing Source Files from Database for Chromium

[creative16]

When trying to build the database for Chromium using 2.8.5, it seems that none of the source files were successfully extracted into the database.

A quick look into build-tracer.log revealed many instances of the following lines:

1 catastrophic error detected in this compilation.
Compilation terminated.
...
Command-line error: the version of Microsoft being emulated must be at least 1911 to use "--ms_c++17"

I was using Chromium checkout 5cd0b13fb350589c6fbaec173754641e39390c25 (tag 101.0.4951.32), building on Windows 10 Build 19044.1586 x64. I used the following commands to create the database:

> gn gen out\Win --args="is_component_build = false is_debug = false enable_nacl = false symbol_level = 0 blink_symbol_level = 0 v8_symbol_level = 0"
> codeql database create C:\Database\chromium_db --language=cpp --command="ninja =C out\Win chrome -j 6"

Any assistance/advice would be appreciated. Thank you!

[aibaars]

Thanks for the report. The problem seems to be that the CodeQL C++ extractor received an incompatible combination of command line arguments combining a value for --microsoft_version that is too low with -ms_c++17. Could you have a look at the log file before the error message. There should be a large command line including those values, what is the value supplied to --microsoft_version ?

Could you double check that your version of the C++ compiler is new enough? CodeQL complains that it requires 19.11+ for C++17 which I believe is Visual Studio 2017 version 15.3 or greater.

[creative16]

Hello, thanks for the prompt reply.

I'm using Visual Studio 2019 Version 16.11.12. I did a search for --microsoft_version within build-tracer.log, but the search returned no results.

[aibaars]

@creative16 Could you look for a line in the build-tracer.log that contains the -ms_c++17 flag? Ideally one that is right before (well as near as possible to) the message Command-line error: the version of Microsoft being emulated must be at least 1911 to use "--ms_c++17"? This line should be a very large command line that tells us how the CodeQL "extractor" was invoked. Make sure to redact/remove any private information, such as path names, that you don't want to share.

[creative16]

Hello @aibaars,

the line immediately preceding the Command-line error message for one of the instances is given below:

Processed command line: <redact>codeql\cpp\tools\win64/extractor.exe --trapfolder <redact>TestDB\trap\cpp --src_archive <redact>\TestDB\src --mimic_config <redact>\TestDB\working/compiler_mimic_cache/a5833d708376 --object_filename obj/base/third_party/double_conversion/double_conversion/string-to-double.obj -w --error_limit 1000 --disable_system_macros --variadic_macros --clang --ms_extensions --clang-cl --c++ --clang_version 150000 -D_HAS_CHAR16_T_LANGUAGE_SUPPORT=1 -D_INTEGRAL_MAX_BITS=64 -D_MSC_BUILD=1 -D_MSC_FULL_VER=191600000 -D_MSVC_EXECUTION_CHARACTER_SET=65001 -D_MSVC_LANG=201703L -D_MT=1 -D_M_AMD64=100 -D_M_X64=100 -D_RVALUE_REFERENCES_SUPPORTED=1 -D_RVALUE_REFERENCES_V2_SUPPORTED=1 -D_WIN64=1 -D__ATOMIC_ACQUIRE=2 -D__ATOMIC_ACQ_REL=4 -D__ATOMIC_CONSUME=1 -D__ATOMIC_RELAXED=0 -D__ATOMIC_RELEASE=3 -D__ATOMIC_SEQ_CST=5 -D__BIGGEST_ALIGNMENT__=16 -D__BITINT_MAXWIDTH__=128 -D__BOOL_WIDTH__=8 -D__BYTE_ORDER__=__ORDER_LITTLE_ENDIAN__ "-D__CHAR16_TYPE__=unsigned short" "-D__CHAR32_TYPE__=unsigned int" -D__CHAR_BIT__=8 -D__CLANG_ATOMIC_BOOL_LOCK_FREE=2 -D__CLANG_ATOMIC_CHAR16_T_LOCK_FREE=2 -D__CLANG_ATOMIC_CHAR32_T_LOCK_FREE=2 -D__CLANG_ATOMIC_CHAR_LOCK_FREE=2 -D__CLANG_ATOMIC_INT_LOCK_FREE=2 -D__CLANG_ATOMIC_LLONG_LOCK_FREE=2 -D__CLANG_ATOMIC_LONG_LOCK_FREE=2 -D__CLANG_ATOMIC_POINTER_LOCK_FREE=2 -D__CLANG_ATOMIC_SHORT_LOCK_FREE=2 -D__CLANG_ATOMIC_WCHAR_T_LOCK_FREE=2 -D__CONSTANT_CFSTRINGS__=1 -D__DBL_DECIMAL_DIG__=17 -D__DBL_DENORM_MIN__=4.9406564584124654e-324 -D__DBL_DIG__=15 -D__DBL_EPSILON__=2.2204460492503131e-16 -D__DBL_HAS_DENORM__=1 -D__DBL_HAS_INFINITY__=1 -D__DBL_HAS_QUIET_NAN__=1 -D__DBL_MANT_DIG__=53 -D__DBL_MAX_10_EXP__=308 -D__DBL_MAX_EXP__=1024 -D__DBL_MAX__=1.7976931348623157e+308 -D__DBL_MIN_10_EXP__=(-307) -D__DBL_MIN_EXP__=(-1021) -D__DBL_MIN__=2.2250738585072014e-308 -D__DECIMAL_DIG__=__LDBL_DECIMAL_DIG__ -D__DEPRECATED=1 -D__FINITE_MATH_ONLY__=0 -D__FLT_DECIMAL_DIG__=9 -D__FLT_DENORM_MIN__=1.40129846e-45F -D__FLT_DIG__=6 -D__FLT_EPSILON__=1.19209290e-7F -D__FLT_HAS_DENORM__=1 -D__FLT_HAS_INFINITY__=1 -D__FLT_HAS_QUIET_NAN__=1 -D__FLT_MANT_DIG__=24 -D__FLT_MAX_10_EXP__=38 -D__FLT_MAX_EXP__=128 -D__FLT_MAX__=3.40282347e+38F -D__FLT_MIN_10_EXP__=(-37) -D__FLT_MIN_EXP__=(-125) -D__FLT_MIN__=1.17549435e-38F -D__FLT_RADIX__=2 -D__FXSR__=1 -D__GCC_ASM_FLAG_OUTPUTS__=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_4=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8=1 -D__INT16_C_SUFFIX__= "-D__INT16_FMTd__=\"hd\"" "-D__INT16_FMTi__=\"hi\"" -D__INT16_MAX__=32767 -D__INT16_TYPE__=short -D__INT32_C_SUFFIX__= "-D__INT32_FMTd__=\"d\"" "-D__INT32_FMTi__=\"i\"" -D__INT32_MAX__=2147483647 -D__INT32_TYPE__=int -D__INT64_C_SUFFIX__=LL "-D__INT64_FMTd__=\"lld\"" "-D__INT64_FMTi__=\"lli\"" -D__INT64_MAX__=9223372036854775807LL "-D__INT64_TYPE__=long long int" -D__INT8_C_SUFFIX__= "-D__INT8_FMTd__=\"hhd\"" "-D__INT8_FMTi__=\"hhi\"" -D__INT8_MAX__=127 "-D__INT8_TYPE__=signed char" -D__INTMAX_C_SUFFIX__=LL "-D__INTMAX_FMTd__=\"lld\"" "-D__INTMAX_FMTi__=\"lli\"" -D__INTMAX_MAX__=9223372036854775807LL "-D__INTMAX_TYPE__=long long int" -D__INTMAX_WIDTH__=64 "-D__INTPTR_FMTd__=\"lld\"" "-D__INTPTR_FMTi__=\"lli\"" -D__INTPTR_MAX__=9223372036854775807LL "-D__INTPTR_TYPE__=long long int" -D__INTPTR_WIDTH__=64 "-D__INT_FAST16_FMTd__=\"hd\"" "-D__INT_FAST16_FMTi__=\"hi\"" -D__INT_FAST16_MAX__=32767 -D__INT_FAST16_TYPE__=short -D__INT_FAST16_WIDTH__=16 "-D__INT_FAST32_FMTd__=\"d\"" "-D__INT_FAST32_FMTi__=\"i\"" -D__INT_FAST32_MAX__=2147483647 -D__INT_FAST32_TYPE__=int -D__INT_FAST32_WIDTH__=32 "-D__INT_FAST64_FMTd__=\"lld\"" "-D__INT_FAST64_FMTi__=\"lli\"" -D__INT_FAST64_MAX__=9223372036854775807LL "-D__INT_FAST64_TYPE__=long long int" -D__INT_FAST64_WIDTH__=64 "-D__INT_FAST8_FMTd__=\"hhd\"" "-D__INT_FAST8_FMTi__=\"hhi\"" -D__INT_FAST8_MAX__=127 "-D__INT_FAST8_TYPE__=signed char" -D__INT_FAST8_WIDTH__=8 "-D__INT_LEAST16_FMTd__=\"hd\"" "-D__INT_LEAST16_FMTi__=\"hi\"" -D__INT_LEAST16_MAX__=32767 -D__INT_LEAST16_TYPE__=short -D__INT_LEAST16_WIDTH__=16 "-D__INT_LEAST32_FMTd__=\"d\"" "-D__INT_LEAST32_FMTi__=\"i\"" -D__INT_LEAST32_MAX__=2147483647 -D__INT_LEAST32_TYPE__=int -D__INT_LEAST32_WIDTH__=32 "-D__INT_LEAST64_FMTd__=\"lld\"" "-D__INT_LEAST64_FMTi__=\"lli\"" -D__INT_LEAST64_MAX__=9223372036854775807LL "-D__INT_LEAST64_TYPE__=long long int" -D__INT_LEAST64_WIDTH__=64 "-D__INT_LEAST8_FMTd__=\"hhd\"" "-D__INT_LEAST8_FMTi__=\"hhi\"" -D__INT_LEAST8_MAX__=127 "-D__INT_LEAST8_TYPE__=signed char" -D__INT_LEAST8_WIDTH__=8 -D__INT_MAX__=2147483647 -D__INT_WIDTH__=32 -D__LDBL_DECIMAL_DIG__=17 -D__LDBL_DENORM_MIN__=4.9406564584124654e-324L -D__LDBL_DIG__=15 -D__LDBL_EPSILON__=2.2204460492503131e-16L -D__LDBL_HAS_DENORM__=1 -D__LDBL_HAS_INFINITY__=1 -D__LDBL_HAS_QUIET_NAN__=1 -D__LDBL_MANT_DIG__=53 -D__LDBL_MAX_10_EXP__=308 -D__LDBL_MAX_EXP__=1024 -D__LDBL_MAX__=1.7976931348623157e+308L -D__LDBL_MIN_10_EXP__=(-307) -D__LDBL_MIN_EXP__=(-1021) -D__LDBL_MIN__=2.2250738585072014e-308L -D__LITTLE_ENDIAN__=1 -D__LLONG_WIDTH__=64 -D__LONG_LONG_MAX__=9223372036854775807LL -D__LONG_MAX__=2147483647L -D__LONG_WIDTH__=32 -D__MMX__=1 -D__NO_INLINE__=1 -D__NO_MATH_INLINES=1 -D__OBJC_BOOL_IS_BOOL=0 -D__OPENCL_MEMORY_SCOPE_ALL_SVM_DEVICES=3 -D__OPENCL_MEMORY_SCOPE_DEVICE=2 -D__OPENCL_MEMORY_SCOPE_SUB_GROUP=4 -D__OPENCL_MEMORY_SCOPE_WORK_GROUP=1 -D__OPENCL_MEMORY_SCOPE_WORK_ITEM=0 -D__ORDER_BIG_ENDIAN__=4321 -D__ORDER_LITTLE_ENDIAN__=1234 -D__ORDER_PDP_ENDIAN__=3412 -D__PIC__=2 -D__POINTER_WIDTH__=64 -D__PRAGMA_REDEFINE_EXTNAME=1 "-D__PTRDIFF_FMTd__=\"lld\"" "-D__PTRDIFF_FMTi__=\"lli\"" -D__PTRDIFF_MAX__=9223372036854775807LL "-D__PTRDIFF_TYPE__=long long int" -D__PTRDIFF_WIDTH__=64 -D__REGISTER_PREFIX__= -D__SCHAR_MAX__=127 -D__SEG_FS=1 -D__SEG_GS=1 -D__SHRT_MAX__=32767 -D__SHRT_WIDTH__=16 -D__SIG_ATOMIC_MAX__=2147483647 -D__SIG_ATOMIC_WIDTH__=32 -D__SIZEOF_DOUBLE__=8 -D__SIZEOF_FLOAT__=4 -D__SIZEOF_INT128__=16 -D__SIZEOF_INT__=4 -D__SIZEOF_LONG_DOUBLE__=8 -D__SIZEOF_LONG_LONG__=8 -D__SIZEOF_LONG__=4 -D__SIZEOF_POINTER__=8 -D__SIZEOF_PTRDIFF_T__=8 -D__SIZEOF_SHORT__=2 -D__SIZEOF_SIZE_T__=8 -D__SIZEOF_WCHAR_T__=2 -D__SIZEOF_WINT_T__=2 "-D__SIZE_FMTX__=\"llX\"" "-D__SIZE_FMTo__=\"llo\"" "-D__SIZE_FMTu__=\"llu\"" "-D__SIZE_FMTx__=\"llx\"" -D__SIZE_MAX__=18446744073709551615ULL "-D__SIZE_TYPE__=long long unsigned int" -D__SIZE_WIDTH__=64 -D__SSE2_MATH__=1 -D__SSE2__=1 -D__SSE_MATH__=1 -D__SSE__=1 -D__SSP_STRONG__=2 -D__STDCPP_DEFAULT_NEW_ALIGNMENT__=16ULL -D__STDCPP_THREADS__=1 -D__STDC_NO_THREADS__=1 -D__STDC_UTF_16__=1 -D__STDC_UTF_32__=1 -D__UINT16_C_SUFFIX__= "-D__UINT16_FMTX__=\"hX\"" "-D__UINT16_FMTo__=\"ho\"" "-D__UINT16_FMTu__=\"hu\"" "-D__UINT16_FMTx__=\"hx\"" -D__UINT16_MAX__=65535 "-D__UINT16_TYPE__=unsigned short" -D__UINT32_C_SUFFIX__=U "-D__UINT32_FMTX__=\"X\"" "-D__UINT32_FMTo__=\"o\"" "-D__UINT32_FMTu__=\"u\"" "-D__UINT32_FMTx__=\"x\"" -D__UINT32_MAX__=4294967295U "-D__UINT32_TYPE__=unsigned int" -D__UINT64_C_SUFFIX__=ULL "-D__UINT64_FMTX__=\"llX\"" "-D__UINT64_FMTo__=\"llo\"" "-D__UINT64_FMTu__=\"llu\"" "-D__UINT64_FMTx__=\"llx\"" -D__UINT64_MAX__=18446744073709551615ULL "-D__UINT64_TYPE__=long long unsigned int" -D__UINT8_C_SUFFIX__= "-D__UINT8_FMTX__=\"hhX\"" "-D__UINT8_FMTo__=\"hho\"" "-D__UINT8_FMTu__=\"hhu\"" "-D__UINT8_FMTx__=\"hhx\"" -D__UINT8_MAX__=255 "-D__UINT8_TYPE__=unsigned char" -D__UINTMAX_C_SUFFIX__=ULL "-D__UINTMAX_FMTX__=\"llX\"" "-D__UINTMAX_FMTo__=\"llo\"" "-D__UINTMAX_FMTu__=\"llu\"" "-D__UINTMAX_FMTx__=\"llx\"" -D__UINTMAX_MAX__=18446744073709551615ULL "-D__UINTMAX_TYPE__=long long unsigned int" -D__UINTMAX_WIDTH__=64 "-D__UINTPTR_FMTX__=\"llX\"" "-D__UINTPTR_FMTo__=\"llo\"" "-D__UINTPTR_FMTu__=\"llu\"" "-D__UINTPTR_FMTx__=\"llx\"" -D__UINTPTR_MAX__=18446744073709551615ULL "-D__UINTPTR_TYPE__=long long unsigned int" -D__UINTPTR_WIDTH__=64 "-D__UINT_FAST16_FMTX__=\"hX\"" "-D__UINT_FAST16_FMTo__=\"ho\"" "-D__UINT_FAST16_FMTu__=\"hu\"" "-D__UINT_FAST16_FMTx__=\"hx\"" -D__UINT_FAST16_MAX__=65535 "-D__UINT_FAST16_TYPE__=unsigned short" "-D__UINT_FAST32_FMTX__=\"X\"" "-D__UINT_FAST32_FMTo__=\"o\"" "-D__UINT_FAST32_FMTu__=\"u\"" "-D__UINT_FAST32_FMTx__=\"x\"" -D__UINT_FAST32_MAX__=4294967295U "-D__UINT_FAST32_TYPE__=unsigned int" "-D__UINT_FAST64_FMTX__=\"llX\"" "-D__UINT_FAST64_FMTo__=\"llo\"" "-D__UINT_FAST64_FMTu__=\"llu\"" "-D__UINT_FAST64_FMTx__=\"llx\"" -D__UINT_FAST64_MAX__=18446744073709551615ULL "-D__UINT_FAST64_TYPE__=long long unsigned int" "-D__UINT_FAST8_FMTX__=\"hhX\"" "-D__UINT_FAST8_FMTo__=\"hho\"" "-D__UINT_FAST8_FMTu__=\"hhu\"" "-D__UINT_FAST8_FMTx__=\"hhx\"" -D__UINT_FAST8_MAX__=255 "-D__UINT_FAST8_TYPE__=unsigned char" "-D__UINT_LEAST16_FMTX__=\"hX\"" "-D__UINT_LEAST16_FMTo__=\"ho\"" "-D__UINT_LEAST16_FMTu__=\"hu\"" "-D__UINT_LEAST16_FMTx__=\"hx\"" -D__UINT_LEAST16_MAX__=65535 "-D__UINT_LEAST16_TYPE__=unsigned short" "-D__UINT_LEAST32_FMTX__=\"X\"" "-D__UINT_LEAST32_FMTo__=\"o\"" "-D__UINT_LEAST32_FMTu__=\"u\"" "-D__UINT_LEAST32_FMTx__=\"x\"" -D__UINT_LEAST32_MAX__=4294967295U "-D__UINT_LEAST32_TYPE__=unsigned int" "-D__UINT_LEAST64_FMTX__=\"llX\"" "-D__UINT_LEAST64_FMTo__=\"llo\"" "-D__UINT_LEAST64_FMTu__=\"llu\"" "-D__UINT_LEAST64_FMTx__=\"llx\"" -D__UINT_LEAST64_MAX__=18446744073709551615ULL "-D__UINT_LEAST64_TYPE__=long long unsigned int" "-D__UINT_LEAST8_FMTX__=\"hhX\"" "-D__UINT_LEAST8_FMTo__=\"hho\"" "-D__UINT_LEAST8_FMTu__=\"hhu\"" "-D__UINT_LEAST8_FMTx__=\"hhx\"" -D__UINT_LEAST8_MAX__=255 "-D__UINT_LEAST8_TYPE__=unsigned char" -D__USER_LABEL_PREFIX__= "-D__VERSION__=\"Clang 15.0.0 (https://github.com/llvm/llvm-project/ 8133778d3c8d481891143e21e0c5081a02a9889d)\"" -D__WCHAR_MAX__=65535 "-D__WCHAR_TYPE__=unsigned short" -D__WCHAR_UNSIGNED__=1 -D__WCHAR_WIDTH__=16 -D__WINT_MAX__=65535 "-D__WINT_TYPE__=unsigned short" -D__WINT_UNSIGNED__=1 -D__WINT_WIDTH__=16 -D__amd64=1 -D__amd64__=1 -D__clang__=1 "-D__clang_literal_encoding__=\"UTF-8\"" -D__clang_major__=15 -D__clang_minor__=0 -D__clang_patchlevel__=0 "-D__clang_version__=\"15.0.0 (https://github.com/llvm/llvm-project/ 8133778d3c8d481891143e21e0c5081a02a9889d)\"" "-D__clang_wide_literal_encoding__=\"UTF-16\"" -D__code_model_small__=1 -D__cpp_aggregate_bases=201603L -D__cpp_aggregate_nsdmi=201304L -D__cpp_alias_templates=200704L -D__cpp_attributes=200809L -D__cpp_binary_literals=201304L -D__cpp_capture_star_this=201603L -D__cpp_constexpr=201603L -D__cpp_constexpr_in_decltype=201711L -D__cpp_decltype=200707L -D__cpp_decltype_auto=201304L -D__cpp_deduction_guides=201703L -D__cpp_delegating_constructors=200604L -D__cpp_digit_separators=201309L -D__cpp_enumerator_attributes=201411L -D__cpp_fold_expressions=201603L -D__cpp_generic_lambdas=201304L -D__cpp_guaranteed_copy_elision=201606L -D__cpp_hex_float=201603L -D__cpp_if_constexpr=201606L -D__cpp_impl_destroying_delete=201806L -D__cpp_inheriting_constructors=201511L -D__cpp_init_captures=201304L -D__cpp_initializer_lists=200806L -D__cpp_inline_variables=201606L -D__cpp_lambdas=200907L -D__cpp_namespace_attributes=201411L -D__cpp_nested_namespace_definitions=201411L -D__cpp_noexcept_function_type=201510L -D__cpp_nontype_template_args=201411L -D__cpp_nontype_template_parameter_auto=201606L -D__cpp_nsdmi=200809L -D__cpp_range_based_for=201603L -D__cpp_raw_strings=200710L -D__cpp_ref_qualifiers=200710L -D__cpp_return_type_deduction=201304L -D__cpp_rtti=199711L -D__cpp_rvalue_references=200610L -D__cpp_static_assert=201411L -D__cpp_structured_bindings=201606L -D__cpp_template_auto=201606L -D__cpp_threadsafe_static_init=200806L -D__cpp_unicode_characters=200704L -D__cpp_unicode_literals=200710L -D__cpp_user_defined_literals=200809L -D__cpp_variable_templates=201304L -D__cpp_variadic_templates=200704L -D__cpp_variadic_using=201611L -D__k8=1 -D__k8__=1 -D__llvm__=1 -D__pic__=2 -D__seg_fs=__attribute__((address_space(257))) -D__seg_gs=__attribute__((address_space(256))) -D__tune_k8__=1 -D__x86_64=1 -D__x86_64__=1 -DDCHECK_ALWAYS_ON=1 -DUSE_AURA=1 -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS "-DCR_CLANG_REVISION=\"llvmorg-15-init-3677-g8133778d-4\"" -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -D_LIBCPP_NO_AUTO_LINK -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_VB -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -I../../buildtools/third_party/libc++ -D__WRL_ENABLE_FUNCTION_STATICS__ -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -I../../buildtools/third_party/libc++/trunk/include --ms_c++17 --no_rtti -I<redact>\src\third_party\llvm-build\Release+Asserts\lib\clang\15.0.0\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\atlmfc\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\cppwinrt" --sys_include "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\ATLMFC\include" --sys_include "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" --sys_include "C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -- ../../base/third_party/double_conversion/double-conversion/string-to-double.cc

The line contains the --ms_c++17 flag, but nothing about --microsoft_version. As I mentioned, I did a search for the term --microsoft_version on the entire log file and the search came up empty.

[aibaars]

The line contains the --ms_c++17 flag, but nothing about --microsoft_version. As I mentioned, I did a search for the term --microsoft_version on the entire log file and the search came up empty.

I did believe you ;-) I just wanted to find out whether I made a mistake with the name of the --microsoft_version or whether there was no such flag at all on the command line.

I see there is a line -D_MSC_FULL_VER=191600000 . Looking at our code we inject the --microsoft_version flag based on the slightly differently named -D_MSC_VER variable.

As a workaround you could try to add -D_MSC_VER=1916 to the compiler flags. Not sure how hard that is. On linux using the environment variables CPPFLAGS or CXXFLAGS usually works. I don't know about if those work with Microsoft C++ though.

I'll pass this on to the team so they can investigate and fix this properly.

[creative16]

Thanks for looking into this! I'll try to figure out how to add that flag in the meantime.

[aibaars]

Another short-term solution could be to adjust the codeql/cpp/tools/win64/compiler-tracing.spec file and add

prepend --microsoft_version
prepend 1916

to the cl/clang-cl block:

**/cl.exe:
**/clang-cl.exe:
  invoke ${config_dir}/extractor.exe
  order compiler, extractor
  prepend --mimic
  prepend "${compiler}"
  prepend --microsoft_version
  prepend 1916

[creative16]

Hello @aibaars,

I've attempted your suggestion on modifying compiler-tracing.spec, but it doesn't seem to work. Now the log file has the following lines:

Command: <redacted>\codeql\cpp\tools\win64/extractor.exe --mimic <redacted>\src\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe --microsoft_version 1916 /c ../../base/third_party/double_conversion/double-conversion/fast-dtoa.cc /Foobj/base/third_party/double_conversion/double_conversion/fast-dtoa.obj /nologo /showIncludes:user "-imsvcC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-imsvcC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" "-imsvcC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -DDCHECK_ALWAYS_ON=1 -DUSE_AURA=1 -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS "-DCR_CLANG_REVISION=\"llvmorg-15-init-3677-g8133778d-4\"" -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -D_LIBCPP_NO_AUTO_LINK -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_VB -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -I../../buildtools/third_party/libc++ /W4 -Wimplicit-fallthrough -Wunreachable-code-aggressive -Wthread-safety -Wextra-semi /WX -Wno-missing-field-initializers -Wno-unused-parameter -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-nonportable-include-path -Wno-null-pointer-subtraction -Wenum-compare-conditional -Wno-psabi -Wno-ignored-pragma-optimize -Wno-unqualified-std-cast-call -Wmax-tokens -Wshadow -fno-delete-null-pointer-checks -fno-ident -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 /clang:-ffp-contract=off -fcomplete-member-pointers /Gy /FS /bigobj /utf-8 /Zc:twoPhase /Zc:sizedDealloc- /D__WRL_ENABLE_FUNCTION_STATICS__ -fmsc-version=1916 -m64 -msse3 /Brepro -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern /O1 /Ob2 /Oy- /Zc:inline /Gw /Oi /guard:cf /MT -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang use-classify-type -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-unused-const-variable -Wno-unused-function /std:c++17 -Wno-trigraphs /Zc:alignedNew- /TP /GR- -I../../buildtools/third_party/libc++/trunk/include /Fdobj/base/third_party/double_conversion/double_conversion_cc.pdb
Warning: Unrecognised command line argument --microsoft_version
Warning: Unrecognised command line argument /showIncludes:user
Warning: Unrecognised command line argument -fno-delete-null-pointer-checks
Warning: Unrecognised command line argument -fno-ident
Warning: Unrecognised command line argument -fcolor-diagnostics
Warning: Unrecognised command line argument -fmerge-all-constants
Warning: Unrecognised command line argument -fcrash-diagnostics-dir=../../tools/clang/crashreports
Warning: Unrecognised command line argument -mllvm
Warning: Unrecognised command line argument -instcombine-lower-dbg-declare=0
Warning: Unrecognised command line argument /clang:-ffp-contract=off
Warning: Unrecognised command line argument -fcomplete-member-pointers
Warning: Unrecognised command line argument /utf-8
Warning: Unrecognised command line argument -m64
Warning: Unrecognised command line argument -msse3
Warning: Unrecognised command line argument -ffile-compilation-dir=.
Warning: Unrecognised command line argument -no-canonical-prefixes
Warning: Unrecognised command line argument -ftrivial-auto-var-init=pattern
Warning: Unrecognised command line argument /guard:cf
Processed command line: <redacted>\codeql\cpp\tools\win64/extractor.exe --trapfolder <redacted>\WinTest\trap\cpp --src_archive <redacted>\WinTest\src --mimic_config <redacted>\WinTest\working/compiler_mimic_cache/a5833d708376 --object_filename obj/base/third_party/double_conversion/double_conversion/fast-dtoa.obj -w --error_limit 1000 --disable_system_macros --variadic_macros --clang --ms_extensions --clang-cl --c++ --clang_version 150000 -D_HAS_CHAR16_T_LANGUAGE_SUPPORT=1 -D_INTEGRAL_MAX_BITS=64 -D_MSC_BUILD=1 -D_MSC_FULL_VER=191600000 -D_MSVC_EXECUTION_CHARACTER_SET=65001 -D_MSVC_LANG=201703L -D_MT=1 -D_M_AMD64=100 -D_M_X64=100 -D_RVALUE_REFERENCES_SUPPORTED=1 -D_RVALUE_REFERENCES_V2_SUPPORTED=1 -D_WIN64=1 -D__ATOMIC_ACQUIRE=2 -D__ATOMIC_ACQ_REL=4 -D__ATOMIC_CONSUME=1 -D__ATOMIC_RELAXED=0 -D__ATOMIC_RELEASE=3 -D__ATOMIC_SEQ_CST=5 -D__BIGGEST_ALIGNMENT__=16 -D__BITINT_MAXWIDTH__=128 -D__BOOL_WIDTH__=8 -D__BYTE_ORDER__=__ORDER_LITTLE_ENDIAN__ "-D__CHAR16_TYPE__=unsigned short" "-D__CHAR32_TYPE__=unsigned int" -D__CHAR_BIT__=8 -D__CLANG_ATOMIC_BOOL_LOCK_FREE=2 -D__CLANG_ATOMIC_CHAR16_T_LOCK_FREE=2 -D__CLANG_ATOMIC_CHAR32_T_LOCK_FREE=2 -D__CLANG_ATOMIC_CHAR_LOCK_FREE=2 -D__CLANG_ATOMIC_INT_LOCK_FREE=2 -D__CLANG_ATOMIC_LLONG_LOCK_FREE=2 -D__CLANG_ATOMIC_LONG_LOCK_FREE=2 -D__CLANG_ATOMIC_POINTER_LOCK_FREE=2 -D__CLANG_ATOMIC_SHORT_LOCK_FREE=2 -D__CLANG_ATOMIC_WCHAR_T_LOCK_FREE=2 -D__CONSTANT_CFSTRINGS__=1 -D__DBL_DECIMAL_DIG__=17 -D__DBL_DENORM_MIN__=4.9406564584124654e-324 -D__DBL_DIG__=15 -D__DBL_EPSILON__=2.2204460492503131e-16 -D__DBL_HAS_DENORM__=1 -D__DBL_HAS_INFINITY__=1 -D__DBL_HAS_QUIET_NAN__=1 -D__DBL_MANT_DIG__=53 -D__DBL_MAX_10_EXP__=308 -D__DBL_MAX_EXP__=1024 -D__DBL_MAX__=1.7976931348623157e+308 -D__DBL_MIN_10_EXP__=(-307) -D__DBL_MIN_EXP__=(-1021) -D__DBL_MIN__=2.2250738585072014e-308 -D__DECIMAL_DIG__=__LDBL_DECIMAL_DIG__ -D__DEPRECATED=1 -D__FINITE_MATH_ONLY__=0 -D__FLT_DECIMAL_DIG__=9 -D__FLT_DENORM_MIN__=1.40129846e-45F -D__FLT_DIG__=6 -D__FLT_EPSILON__=1.19209290e-7F -D__FLT_HAS_DENORM__=1 -D__FLT_HAS_INFINITY__=1 -D__FLT_HAS_QUIET_NAN__=1 -D__FLT_MANT_DIG__=24 -D__FLT_MAX_10_EXP__=38 -D__FLT_MAX_EXP__=128 -D__FLT_MAX__=3.40282347e+38F -D__FLT_MIN_10_EXP__=(-37) -D__FLT_MIN_EXP__=(-125) -D__FLT_MIN__=1.17549435e-38F -D__FLT_RADIX__=2 -D__FXSR__=1 -D__GCC_ASM_FLAG_OUTPUTS__=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_4=1 -D__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8=1 -D__INT16_C_SUFFIX__= "-D__INT16_FMTd__=\"hd\"" "-D__INT16_FMTi__=\"hi\"" -D__INT16_MAX__=32767 -D__INT16_TYPE__=short -D__INT32_C_SUFFIX__= "-D__INT32_FMTd__=\"d\"" "-D__INT32_FMTi__=\"i\"" -D__INT32_MAX__=2147483647 -D__INT32_TYPE__=int -D__INT64_C_SUFFIX__=LL "-D__INT64_FMTd__=\"lld\"" "-D__INT64_FMTi__=\"lli\"" -D__INT64_MAX__=9223372036854775807LL "-D__INT64_TYPE__=long long int" -D__INT8_C_SUFFIX__= "-D__INT8_FMTd__=\"hhd\"" "-D__INT8_FMTi__=\"hhi\"" -D__INT8_MAX__=127 "-D__INT8_TYPE__=signed char" -D__INTMAX_C_SUFFIX__=LL "-D__INTMAX_FMTd__=\"lld\"" "-D__INTMAX_FMTi__=\"lli\"" -D__INTMAX_MAX__=9223372036854775807LL "-D__INTMAX_TYPE__=long long int" -D__INTMAX_WIDTH__=64 "-D__INTPTR_FMTd__=\"lld\"" "-D__INTPTR_FMTi__=\"lli\"" -D__INTPTR_MAX__=9223372036854775807LL "-D__INTPTR_TYPE__=long long int" -D__INTPTR_WIDTH__=64 "-D__INT_FAST16_FMTd__=\"hd\"" "-D__INT_FAST16_FMTi__=\"hi\"" -D__INT_FAST16_MAX__=32767 -D__INT_FAST16_TYPE__=short -D__INT_FAST16_WIDTH__=16 "-D__INT_FAST32_FMTd__=\"d\"" "-D__INT_FAST32_FMTi__=\"i\"" -D__INT_FAST32_MAX__=2147483647 -D__INT_FAST32_TYPE__=int -D__INT_FAST32_WIDTH__=32 "-D__INT_FAST64_FMTd__=\"lld\"" "-D__INT_FAST64_FMTi__=\"lli\"" -D__INT_FAST64_MAX__=9223372036854775807LL "-D__INT_FAST64_TYPE__=long long int" -D__INT_FAST64_WIDTH__=64 "-D__INT_FAST8_FMTd__=\"hhd\"" "-D__INT_FAST8_FMTi__=\"hhi\"" -D__INT_FAST8_MAX__=127 "-D__INT_FAST8_TYPE__=signed char" -D__INT_FAST8_WIDTH__=8 "-D__INT_LEAST16_FMTd__=\"hd\"" "-D__INT_LEAST16_FMTi__=\"hi\"" -D__INT_LEAST16_MAX__=32767 -D__INT_LEAST16_TYPE__=short -D__INT_LEAST16_WIDTH__=16 "-D__INT_LEAST32_FMTd__=\"d\"" "-D__INT_LEAST32_FMTi__=\"i\"" -D__INT_LEAST32_MAX__=2147483647 -D__INT_LEAST32_TYPE__=int -D__INT_LEAST32_WIDTH__=32 "-D__INT_LEAST64_FMTd__=\"lld\"" "-D__INT_LEAST64_FMTi__=\"lli\"" -D__INT_LEAST64_MAX__=9223372036854775807LL "-D__INT_LEAST64_TYPE__=long long int" -D__INT_LEAST64_WIDTH__=64 "-D__INT_LEAST8_FMTd__=\"hhd\"" "-D__INT_LEAST8_FMTi__=\"hhi\"" -D__INT_LEAST8_MAX__=127 "-D__INT_LEAST8_TYPE__=signed char" -D__INT_LEAST8_WIDTH__=8 -D__INT_MAX__=2147483647 -D__INT_WIDTH__=32 -D__LDBL_DECIMAL_DIG__=17 -D__LDBL_DENORM_MIN__=4.9406564584124654e-324L -D__LDBL_DIG__=15 -D__LDBL_EPSILON__=2.2204460492503131e-16L -D__LDBL_HAS_DENORM__=1 -D__LDBL_HAS_INFINITY__=1 -D__LDBL_HAS_QUIET_NAN__=1 -D__LDBL_MANT_DIG__=53 -D__LDBL_MAX_10_EXP__=308 -D__LDBL_MAX_EXP__=1024 -D__LDBL_MAX__=1.7976931348623157e+308L -D__LDBL_MIN_10_EXP__=(-307) -D__LDBL_MIN_EXP__=(-1021) -D__LDBL_MIN__=2.2250738585072014e-308L -D__LITTLE_ENDIAN__=1 -D__LLONG_WIDTH__=64 -D__LONG_LONG_MAX__=9223372036854775807LL -D__LONG_MAX__=2147483647L -D__LONG_WIDTH__=32 -D__MMX__=1 -D__NO_INLINE__=1 -D__NO_MATH_INLINES=1 -D__OBJC_BOOL_IS_BOOL=0 -D__OPENCL_MEMORY_SCOPE_ALL_SVM_DEVICES=3 -D__OPENCL_MEMORY_SCOPE_DEVICE=2 -D__OPENCL_MEMORY_SCOPE_SUB_GROUP=4 -D__OPENCL_MEMORY_SCOPE_WORK_GROUP=1 -D__OPENCL_MEMORY_SCOPE_WORK_ITEM=0 -D__ORDER_BIG_ENDIAN__=4321 -D__ORDER_LITTLE_ENDIAN__=1234 -D__ORDER_PDP_ENDIAN__=3412 -D__PIC__=2 -D__POINTER_WIDTH__=64 -D__PRAGMA_REDEFINE_EXTNAME=1 "-D__PTRDIFF_FMTd__=\"lld\"" "-D__PTRDIFF_FMTi__=\"lli\"" -D__PTRDIFF_MAX__=9223372036854775807LL "-D__PTRDIFF_TYPE__=long long int" -D__PTRDIFF_WIDTH__=64 -D__REGISTER_PREFIX__= -D__SCHAR_MAX__=127 -D__SEG_FS=1 -D__SEG_GS=1 -D__SHRT_MAX__=32767 -D__SHRT_WIDTH__=16 -D__SIG_ATOMIC_MAX__=2147483647 -D__SIG_ATOMIC_WIDTH__=32 -D__SIZEOF_DOUBLE__=8 -D__SIZEOF_FLOAT__=4 -D__SIZEOF_INT128__=16 -D__SIZEOF_INT__=4 -D__SIZEOF_LONG_DOUBLE__=8 -D__SIZEOF_LONG_LONG__=8 -D__SIZEOF_LONG__=4 -D__SIZEOF_POINTER__=8 -D__SIZEOF_PTRDIFF_T__=8 -D__SIZEOF_SHORT__=2 -D__SIZEOF_SIZE_T__=8 -D__SIZEOF_WCHAR_T__=2 -D__SIZEOF_WINT_T__=2 "-D__SIZE_FMTX__=\"llX\"" "-D__SIZE_FMTo__=\"llo\"" "-D__SIZE_FMTu__=\"llu\"" "-D__SIZE_FMTx__=\"llx\"" -D__SIZE_MAX__=18446744073709551615ULL "-D__SIZE_TYPE__=long long unsigned int" -D__SIZE_WIDTH__=64 -D__SSE2_MATH__=1 -D__SSE2__=1 -D__SSE_MATH__=1 -D__SSE__=1 -D__SSP_STRONG__=2 -D__STDCPP_DEFAULT_NEW_ALIGNMENT__=16ULL -D__STDCPP_THREADS__=1 -D__STDC_NO_THREADS__=1 -D__STDC_UTF_16__=1 -D__STDC_UTF_32__=1 -D__UINT16_C_SUFFIX__= "-D__UINT16_FMTX__=\"hX\"" "-D__UINT16_FMTo__=\"ho\"" "-D__UINT16_FMTu__=\"hu\"" "-D__UINT16_FMTx__=\"hx\"" -D__UINT16_MAX__=65535 "-D__UINT16_TYPE__=unsigned short" -D__UINT32_C_SUFFIX__=U "-D__UINT32_FMTX__=\"X\"" "-D__UINT32_FMTo__=\"o\"" "-D__UINT32_FMTu__=\"u\"" "-D__UINT32_FMTx__=\"x\"" -D__UINT32_MAX__=4294967295U "-D__UINT32_TYPE__=unsigned int" -D__UINT64_C_SUFFIX__=ULL "-D__UINT64_FMTX__=\"llX\"" "-D__UINT64_FMTo__=\"llo\"" "-D__UINT64_FMTu__=\"llu\"" "-D__UINT64_FMTx__=\"llx\"" -D__UINT64_MAX__=18446744073709551615ULL "-D__UINT64_TYPE__=long long unsigned int" -D__UINT8_C_SUFFIX__= "-D__UINT8_FMTX__=\"hhX\"" "-D__UINT8_FMTo__=\"hho\"" "-D__UINT8_FMTu__=\"hhu\"" "-D__UINT8_FMTx__=\"hhx\"" -D__UINT8_MAX__=255 "-D__UINT8_TYPE__=unsigned char" -D__UINTMAX_C_SUFFIX__=ULL "-D__UINTMAX_FMTX__=\"llX\"" "-D__UINTMAX_FMTo__=\"llo\"" "-D__UINTMAX_FMTu__=\"llu\"" "-D__UINTMAX_FMTx__=\"llx\"" -D__UINTMAX_MAX__=18446744073709551615ULL "-D__UINTMAX_TYPE__=long long unsigned int" -D__UINTMAX_WIDTH__=64 "-D__UINTPTR_FMTX__=\"llX\"" "-D__UINTPTR_FMTo__=\"llo\"" "-D__UINTPTR_FMTu__=\"llu\"" "-D__UINTPTR_FMTx__=\"llx\"" -D__UINTPTR_MAX__=18446744073709551615ULL "-D__UINTPTR_TYPE__=long long unsigned int" -D__UINTPTR_WIDTH__=64 "-D__UINT_FAST16_FMTX__=\"hX\"" "-D__UINT_FAST16_FMTo__=\"ho\"" "-D__UINT_FAST16_FMTu__=\"hu\"" "-D__UINT_FAST16_FMTx__=\"hx\"" -D__UINT_FAST16_MAX__=65535 "-D__UINT_FAST16_TYPE__=unsigned short" "-D__UINT_FAST32_FMTX__=\"X\"" "-D__UINT_FAST32_FMTo__=\"o\"" "-D__UINT_FAST32_FMTu__=\"u\"" "-D__UINT_FAST32_FMTx__=\"x\"" -D__UINT_FAST32_MAX__=4294967295U "-D__UINT_FAST32_TYPE__=unsigned int" "-D__UINT_FAST64_FMTX__=\"llX\"" "-D__UINT_FAST64_FMTo__=\"llo\"" "-D__UINT_FAST64_FMTu__=\"llu\"" "-D__UINT_FAST64_FMTx__=\"llx\"" -D__UINT_FAST64_MAX__=18446744073709551615ULL "-D__UINT_FAST64_TYPE__=long long unsigned int" "-D__UINT_FAST8_FMTX__=\"hhX\"" "-D__UINT_FAST8_FMTo__=\"hho\"" "-D__UINT_FAST8_FMTu__=\"hhu\"" "-D__UINT_FAST8_FMTx__=\"hhx\"" -D__UINT_FAST8_MAX__=255 "-D__UINT_FAST8_TYPE__=unsigned char" "-D__UINT_LEAST16_FMTX__=\"hX\"" "-D__UINT_LEAST16_FMTo__=\"ho\"" "-D__UINT_LEAST16_FMTu__=\"hu\"" "-D__UINT_LEAST16_FMTx__=\"hx\"" -D__UINT_LEAST16_MAX__=65535 "-D__UINT_LEAST16_TYPE__=unsigned short" "-D__UINT_LEAST32_FMTX__=\"X\"" "-D__UINT_LEAST32_FMTo__=\"o\"" "-D__UINT_LEAST32_FMTu__=\"u\"" "-D__UINT_LEAST32_FMTx__=\"x\"" -D__UINT_LEAST32_MAX__=4294967295U "-D__UINT_LEAST32_TYPE__=unsigned int" "-D__UINT_LEAST64_FMTX__=\"llX\"" "-D__UINT_LEAST64_FMTo__=\"llo\"" "-D__UINT_LEAST64_FMTu__=\"llu\"" "-D__UINT_LEAST64_FMTx__=\"llx\"" -D__UINT_LEAST64_MAX__=18446744073709551615ULL "-D__UINT_LEAST64_TYPE__=long long unsigned int" "-D__UINT_LEAST8_FMTX__=\"hhX\"" "-D__UINT_LEAST8_FMTo__=\"hho\"" "-D__UINT_LEAST8_FMTu__=\"hhu\"" "-D__UINT_LEAST8_FMTx__=\"hhx\"" -D__UINT_LEAST8_MAX__=255 "-D__UINT_LEAST8_TYPE__=unsigned char" -D__USER_LABEL_PREFIX__= "-D__VERSION__=\"Clang 15.0.0 (https://github.com/llvm/llvm-project/ 8133778d3c8d481891143e21e0c5081a02a9889d)\"" -D__WCHAR_MAX__=65535 "-D__WCHAR_TYPE__=unsigned short" -D__WCHAR_UNSIGNED__=1 -D__WCHAR_WIDTH__=16 -D__WINT_MAX__=65535 "-D__WINT_TYPE__=unsigned short" -D__WINT_UNSIGNED__=1 -D__WINT_WIDTH__=16 -D__amd64=1 -D__amd64__=1 -D__clang__=1 "-D__clang_literal_encoding__=\"UTF-8\"" -D__clang_major__=15 -D__clang_minor__=0 -D__clang_patchlevel__=0 "-D__clang_version__=\"15.0.0 (https://github.com/llvm/llvm-project/ 8133778d3c8d481891143e21e0c5081a02a9889d)\"" "-D__clang_wide_literal_encoding__=\"UTF-16\"" -D__code_model_small__=1 -D__cpp_aggregate_bases=201603L -D__cpp_aggregate_nsdmi=201304L -D__cpp_alias_templates=200704L -D__cpp_attributes=200809L -D__cpp_binary_literals=201304L -D__cpp_capture_star_this=201603L -D__cpp_constexpr=201603L -D__cpp_constexpr_in_decltype=201711L -D__cpp_decltype=200707L -D__cpp_decltype_auto=201304L -D__cpp_deduction_guides=201703L -D__cpp_delegating_constructors=200604L -D__cpp_digit_separators=201309L -D__cpp_enumerator_attributes=201411L -D__cpp_fold_expressions=201603L -D__cpp_generic_lambdas=201304L -D__cpp_guaranteed_copy_elision=201606L -D__cpp_hex_float=201603L -D__cpp_if_constexpr=201606L -D__cpp_impl_destroying_delete=201806L -D__cpp_inheriting_constructors=201511L -D__cpp_init_captures=201304L -D__cpp_initializer_lists=200806L -D__cpp_inline_variables=201606L -D__cpp_lambdas=200907L -D__cpp_namespace_attributes=201411L -D__cpp_nested_namespace_definitions=201411L -D__cpp_noexcept_function_type=201510L -D__cpp_nontype_template_args=201411L -D__cpp_nontype_template_parameter_auto=201606L -D__cpp_nsdmi=200809L -D__cpp_range_based_for=201603L -D__cpp_raw_strings=200710L -D__cpp_ref_qualifiers=200710L -D__cpp_return_type_deduction=201304L -D__cpp_rtti=199711L -D__cpp_rvalue_references=200610L -D__cpp_static_assert=201411L -D__cpp_structured_bindings=201606L -D__cpp_template_auto=201606L -D__cpp_threadsafe_static_init=200806L -D__cpp_unicode_characters=200704L -D__cpp_unicode_literals=200710L -D__cpp_user_defined_literals=200809L -D__cpp_variable_templates=201304L -D__cpp_variadic_templates=200704L -D__cpp_variadic_using=201611L -D__k8=1 -D__k8__=1 -D__llvm__=1 -D__pic__=2 -D__seg_fs=__attribute__((address_space(257))) -D__seg_gs=__attribute__((address_space(256))) -D__tune_k8__=1 -D__x86_64=1 -D__x86_64__=1 -DDCHECK_ALWAYS_ON=1 -DUSE_AURA=1 -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS "-DCR_CLANG_REVISION=\"llvmorg-15-init-3677-g8133778d-4\"" -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -D_LIBCPP_NO_AUTO_LINK -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_VB -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -I../../buildtools/third_party/libc++ -D__WRL_ENABLE_FUNCTION_STATICS__ -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -I../../buildtools/third_party/libc++/trunk/include --ms_c++17 --no_rtti -I<redacted>\src\third_party\llvm-build\Release+Asserts\lib\clang\15.0.0\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\atlmfc\include" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\cppwinrt" --sys_include "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\ATLMFC\include" --sys_include "C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" --sys_include "C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" --sys_include "C:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -- 1916 ../../base/third_party/double_conversion/double-conversion/fast-dtoa.cc
Command-line error: the version of Microsoft being emulated must be at least 1911 to use "--ms_c++17"

So the observation is that --microsoft_version 1916 is now included in the command, but is reported as unrecognised, and is missing from the processed command.

[creative16]

Hello @aibaars,

I played around with the command line options to extractor.exe, and discovered that, if --microsoft_version is passed before --mimic, extractor.exe will accept the argument and the original error appears to be resolved:

**/cl.exe:
**/clang-cl.exe:
  invoke ${config_dir}/extractor.exe
  order compiler, extractor
  prepend --microsoft_version
  prepend 1916
  prepend --mimic
  prepend "${compiler}"

There is, however, now another issue that causes another catastrophic error. The processed command line and the error message is given below:

Processed command line: <redacted>\codeql\cpp\tools\win64/extractor.exe --microsoft_version 1916 --mimic <redacted>\src\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe /c ../../base/third_party/double_conversion/double-conversion/strtod.cc /Foobj/base/third_party/double_conversion/double_conversion/strtod.obj /nologo /showIncludes:user "-imsvcC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\ATLMFC\include" "-imsvcC:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.29.30133\include" "-imsvcC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-imsvcC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" -DDCHECK_ALWAYS_ON=1 -DUSE_AURA=1 -D_HAS_NODISCARD -D_CRT_NONSTDC_NO_WARNINGS -D_WINSOCK_DEPRECATED_NO_WARNINGS "-DCR_CLANG_REVISION=\"llvmorg-15-init-3677-g8133778d-4\"" -D_LIBCPP_ABI_UNSTABLE -D_LIBCPP_DISABLE_VISIBILITY_ANNOTATIONS -D_LIBCPP_ENABLE_NODISCARD -D_LIBCPP_DEBUG=0 -D_LIBCPP_NO_AUTO_LINK -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=2 -DWIN32 -D_SECURE_ATL -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=NTDDI_WIN10_VB -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -I../.. -Igen -I../../buildtools/third_party/libc++ /W4 -Wimplicit-fallthrough -Wunreachable-code-aggressive -Wthread-safety -Wextra-semi /WX -Wno-missing-field-initializers -Wno-unused-parameter -Wloop-analysis -Wno-unneeded-internal-declaration -Wno-nonportable-include-path -Wno-null-pointer-subtraction -Wenum-compare-conditional -Wno-psabi -Wno-ignored-pragma-optimize -Wno-unqualified-std-cast-call -Wmax-tokens -Wshadow -fno-delete-null-pointer-checks -fno-ident -fcolor-diagnostics -fmerge-all-constants -fcrash-diagnostics-dir=../../tools/clang/crashreports -mllvm -instcombine-lower-dbg-declare=0 /clang:-ffp-contract=off -fcomplete-member-pointers /Gy /FS /bigobj /utf-8 /Zc:twoPhase /Zc:sizedDealloc- /D__WRL_ENABLE_FUNCTION_STATICS__ -fmsc-version=1916 -m64 -msse3 /Brepro -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -ffile-compilation-dir=. -no-canonical-prefixes -ftrivial-auto-var-init=pattern /O1 /Ob2 /Oy- /Zc:inline /Gw /Oi /guard:cf /MT -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang raw-ptr-template-as-trivial-member -Xclang -plugin-arg-find-bad-constructs -Xclang use-classify-type -Wheader-hygiene -Wstring-conversion -Wtautological-overlap-compare -Wno-unused-const-variable -Wno-unused-function /std:c++17 -Wno-trigraphs /Zc:alignedNew- /TP /GR- -I../../buildtools/third_party/libc++/trunk/include /Fdobj/base/third_party/double_conversion/double_conversion_cc.pdb
Starting compilation TRAP ./compilations/60/45974613_0.trap.br
/c:
Catastrophic error: cannot open source file "/c"

Interestingly though, the --ms_c++17 flag seems to have disappeared. In fact, the entire processed command seems quite different from what it was previously.

[pooneh-nb]

I'm following up on this issue here. I have the same issue with creating the codeql of chromium source code on linux. A large number of files are missed. Particularly, there isn't any .cpp file in the zip src folder. My understanding of the codeql documentation is that there should be an equivalent file for each source file. But I have only around 3000 files. The command I run to generate the DB is:

./build/install-build-deps.sh && gclient runhooks && gn gen out/Default && 
codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome -j 20”

[aibaars]

@creative16 Apologies, I think the trick of passing the --microsoft_version using the compiler-tracing.spec doesn't work :-( . I think the --mimic flag needs to be the first option. The compiler interception is a sort of two step process. The first step (with --mimic) receives the original compiler command line. The extractor inspects the arguments on the command line and translates them to its own normalized options and the second step invokes the extractor again with the transformed command line. We need to somehow inject the --microsoft_version flag in the second step but not the first. One thing to try would to supply prepend -D_MSC_VER=1916. If we're "lucky" the first step will pick up that flag and translate it to --microsoft_version 1916 for the second step.

I'll pass this problem on to the team to see if they can look into improving support for clang-cl.

[aibaars]

I'm following up on this issue here. I have the same issue with creating the codeql of chromium source code on linux. A large number of files are missed. Particularly, there isn't any .cpp file in the zip src folder. My understanding of the codeql documentation is that there should be an equivalent file for each source file. But I have only around 3000 files. The command I run to generate the DB is:

./build/install-build-deps.sh && gclient runhooks && gn gen out/Default && 
codeql database create chromium-database --language=cpp --command="autoninja -C out/Default chrome -j 20”

@pooneh-nb The problem in this issue is related to clang-cl for Windows and may have a different cause than yours. Could you have a look in the log/build-tracer.log file for error messages that could explain what's going wrong and post them in a separate issue? Thanks!

[aibaars]

@pooneh-nb There is an issue from another Linux user here https://github.com/github/codeql/issues/8755 .

[pooneh-nb]

@aibaars Thanks. I have the same issue as in #8755

[creative16]

Hello @aibaars,

One thing to try would to supply prepend -D_MSC_VER=1916. If we're "lucky" the first step will pick up that flag and translate it to --microsoft_version 1916 for the second step.

No such luck! The -D_MSC_VER=1916 got passed on to the processed command, but the original error still appears.

I figured as much about the two-step process after taking a closer look at the log file and running the extractor manually and observing its output.

Thanks so much for looking into this! Looking forward to a fix!

[jketema]

Hi @creative16, we believe your issue has the same root-cause as the one causing https://github.com/github/codeql/issues/8755 (as linked to by @pooneh-nb). This issue should be fixed in CodeQL 2.12.2, which will be released in a few weeks time.

I'm closing this issue as completed, but please re-open if the problem still occurs with CodeQL 2.12.2 or later.