Issue handling -Xclang custom plugin and flags

[YaronStavitzky]

We are using sccache with S3 for the past few weeks, testing if we can get some benefits. Everything was fine until our most recent release, where we added a clang plugin.

This is what we are passing to clang:

['-fplugin=compilation/build/orion-plugin/orion-clang-plugin.so', '-Xclang', '-plugin-arg-orion', '-Xclang', '-fadd-orion-trace-types', '-Xclang', '-plugin-arg-orion', '-Xclang', '-Werror', '-Xclang', '-plugin-arg-orion', '-Xclang', '-Wno-trace-arg-cast-ignored', '-Xclang', '-plugin-arg-orion', '-Xclang', '-Wno-explicit-constructor', '-Xclang', '-plugin-arg-orion', '-Xclang', '-Wno-mismatch-argument-size-trace', '-Xclang', '-plugin-arg-orion', '-Xclang', '-fthirdparty-regex=^/\\|src/plasma/third_party']

Example output from sccache log:

[2023-02-12T14:41:46Z DEBUG sccache::server] handle_client: compile
[2023-02-12T14:41:46Z DEBUG sccache::server] check_compiler: Supported compiler
[2023-02-12T14:41:46Z DEBUG sccache::server] parse_arguments: CannotCache(Can't handle UnknownFlag arguments with -Xclang, -fadd-orion-trace-types): 
["-o", "src/estore/snaps/giving_tree/metrics/group-0.cu-grouping.o", "-c", "-fplugin=compilation/build/orion-plugin/orion-clang-plugin.so", "-Xclang", "-plugin-arg-orion", 
"-Xclang", "-fadd-orion-trace-types", "-Xclang", "-plugin-arg-orion", "-Xclang", "-Werror", "-Xclang", "-plugin-arg-orion", "-Xclang", "-Wno-trace-arg-cast-ignored", 
"-Xclang", "-plugin-arg-orion", "-Xclang", "-Wno-explicit-constructor", "-Xclang", "-plugin-arg-orion", "-Xclang", "-Wno-mismatch-argument-size-trace", 
"-Xclang", "-plugin-arg-orion", "-Xclang", "-fthirdparty-regex=^/|src/plasma/third_party", "-std=c++11", "-Wno-invalid-offsetof", "-DORION_WITH_ARROW", 
"-flto=full", "-Weverything", "-Wno-flexible-array-extensions", "-Wno-c++98-compat-pedantic", "-Wno-c99-extensions", "-Wno-exit-time-destructors", 
"-Wno-global-constructors", "-Wno-zero-length-array", "-Wno-vla-extension", "-Wno-weak-vtables", "-Wno-float-equal", "-Wno-format-nonliteral", 
"-Wno-covered-switch-default", "-Wno-gnu-zero-variadic-macro-arguments", "-Wno-documentation-unknown-command", "-Wno-documentation", "-Wno-non-virtual-dtor", 
"-Wno-old-style-cast", "-Wno-cast-qual", "-Wno-reserved-id-macro", "-Wno-zero-as-null-pointer-constant", "-Wno-format-pedantic", "-Wno-extra-semi-stmt", 
"-Wno-unused-template", "-Wno-address-of-packed-member", "-Wno-atomic-implicit-seq-cst", "-Wno-alloca", "-Wno-reorder-init-list", "-Wno-undefined-func-template", 
"-Wno-unused-lambda-capture", "-Wno-unused-local-typedef", "-Wno-inconsistent-missing-override", "-Wno-return-std-move", "-Wno-inconsistent-missing-destructor-override",
 "-Wno-shadow-field", "-Wno-double-promotion", "-Wno-implicit-int-float-conversion", "-Wno-header-hygiene", "-Werror", "-g", "-O2", "-fno-omit-frame-pointer", 
 "-Wno-disabled-macro-expansion", "-Wno-vla", "-Wno-packed", "-Wno-unused-function", "-Wno-padded", "-Wno-cast-align", "-D__STDC_FORMAT_MACROS", "-DZSTD_STATIC_LINKING_ONLY", 
 "-Isrc", "-Isrc/plasma/third_party/GSL/include", "-Isrc/plasma/third_party/libpg_query/include", "-I/usr/include/libxml2", "-I/usr/include/jerasure", 
 "src/estore/snaps/giving_tree/metrics/group-0.cu-grouping.cpp"]

sccache -s output:

Compile requests                   1177
Compile requests executed            75
Cache hits                           75
Cache hits (C/C++)                   75
Cache misses                          0
Cache timeouts                        0
Cache read errors                     0
Forced recaches                       0
Cache write errors                    0
Compilation failures                  0
Cache errors                          0
Non-cacheable compilations            0
Non-cacheable calls                 955
Non-compilation calls               147
Unsupported compiler calls            0
Average cache write               0.000 s
Average compiler                  0.000 s
Average cache read hit            0.000 s
Failed distributed compilations       0

Non-cacheable reasons:
Can't handle UnknownFlag arguments with -Xclang     955

Is there anything that we can do on our side or do we have to implement something in sccache's code? I would be happy to provide anything to help!

[YaronStavitzky]

We tried both 0.3.3 and 0.4.0 pre6 (x86_64 and aarch64).

[Xuanwo]

How will this flag affect the compile result? We need to decide whether ignore or add this flag into our hash calculations.

[da-x]

@Xuanwo Let's assume you can't know what the plugin does - it is custom code. Some plugins can change the code being compiled and thus affect the output. So I think it is best to just add all those flags into the hash calculation as they are.

[Xuanwo]

@da-x Thanks for the explaination.

So I think it is best to just add all those flags into the hash calculation as they are.

@YaronStavitzky Please take a look over https://github.com/mozilla/sccache/blob/64eeb966aab499b4661e0e805818d1941a551f21/src/compiler/clang.rs#L161 and add -fadd-orion-trace-types as a flag.

[da-x]

@Xuanwo why does this array need to be modified specially for -fadd-orion-trace-type? This flag -fadd-orion-trace-type is actually a parameter for -plugin-arg-orion, which is a parameter to the orion plugin that is brought by -fplugin=compilation/build/orion-plugin/orion-clang-plugin.so.

To implement what I suggested earlier, I think the algorithm in sccache side should be as follows:

• Treat all -Xclang A as giving parameter A to Clang. E.g. with -Xclang A -Xclang B -Xclang C, Clang gets a vector of parameters A B C.
• Under the Clang parameter scope where we handle A B C, regard all -plugin-arg-X T as plugin arguments which should be hashed as-is, so -plugin-arg-X -fadd-orion-trace-types gets hashed. The original vector or arguments -Xclang -plugin-arg-X -Xclang -fadd-orion-trace-types gets hashed therefore.

[YaronStavitzky]

@Xuanwo I added

take_arg!("-fadd-orion-trace-types", OsString, Separated, PassThrough),

at the end of counted_array!(pub static ARGS: [ArgInfo<gcc::ArgData>; _] = [ It did not work, should it be placed in a different position in the array? Actually, shouldn't we also handle the -plugin-arg-orion ?

[YaronStavitzky]

@Xuanwo we fixed it by adding Argument::UnknownFlag(_) if follows_plugin_arg => &mut common_args, to src/compiler/clang.rs:

 None => match arg {
                Argument::Raw(_) if follows_plugin_arg => &mut common_args,
                Argument::Raw(_) => cannot_cache!("Can't handle Raw arguments with -Xclang"),
                Argument::UnknownFlag(_) if follows_plugin_arg => &mut common_args,
                Argument::UnknownFlag(flag) => {
                    cannot_cache!(
                        "Can't handle UnknownFlag arguments with -Xclang",
                        flag.to_str().unwrap_or("").to_string()
                    )
                }
                _ => unreachable!(),
            },

[Xuanwo]

we fixed it by adding Argument::UnknownFlag(_) if follows_plugin_arg => &mut common_args, to src/compiler/clang.rs:

Would you be willing to submit a pull request for this?