Failed to compile a project with LTO + PGO

[zamazan4ik]

In https://github.com/vectordotdev/vector/issues/15631#issuecomment-1694554798 I tried to optimize Vector with LTO + PGO enabled. But during the optimization phase (done with cargo pgo optimize) I get the following error: https://github.com/Kobzol/cargo-pgo/issues/32#issuecomment-1694445787 . I am not the only person with this error - https://github.com/llvm/llvm-project/issues/57501#issuecomment-1694392182 (@kevincox probably can provide more details about they project).

The only way to avoid the bug is disable LTO completely (switching from Fat to Thin mode does not help).

I expected to see this happen: Vector compiles successfully with LTO and PGO enabled.

Instead, this happened: Vector failed to compile with LTO + PGO.

Meta

rustc --version --verbose (from https://github.com/vectordotdev/vector/blob/master/rust-toolchain.toml):

1.71.1

Other links:

• Possibly related bug in LLVM (but I am not sure): https://github.com/llvm/llvm-project/issues/57501
• A discussion in cargo-pgo: https://github.com/Kobzol/cargo-pgo/issues/32

[Kobzol]

Were you able to reproduce this without using cargo pgo, by using the compiler flags manually? (-Cprofile-generate etc.)

[zamazan4ik]

I haven't tried to do it yet but I will (and then report the results here)

[kevincox]

My project was done using the flags directly. See https://gitlab.com/kevincox/ricochetrobots/-/blob/6bbc44ddbf2ab8ae9edcbdca62e3035572f9c1ed/default.nix#L41-57

# Cargo.toml
[profile.release]
codegen-units = 1
lto = true

RUSTFLAGS=-Cprofile-use={profile-data}

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ricochetrobots` (bin "ricochetrobots-debug-precompute")
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ricochetrobots` (bin "solve")
[naersk] cargo returned with exit code 101, exiting

[caizixian]

I'm using the compiler flag directly (via RUSTFLAGS). RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" fails with

error: file `/tmp/pgo-data/merged.profdata` passed to `-C profile-use` does not exist.`

The file is confirmed to exist via ls. The merged.profdata is generated using llvm-profdata shipped with Rust 1.71.1.

RUSTFLAGS="-Clto=false -Cprofile-use=/tmp/pgo-data/merged.profdata -Cllvm-args=-pgo-warn-missing-function" works.

My PGO+LTO setup worked with Rust 1.66.1, but doesn't work with 1.71.1.

[lqd]

This "file does not exist" error seems like a different one from the OP ("LLVM ERROR: Broken module found, compilation aborted!"), and it would be good to have a small self-contained example for your new error -- and possibly a dedicated issue since it also looks different from #63609. It feels possible to investigate in mmtk-openjdk directly, but to land a fix with a non-regression test, a small reproducer (MCVE) is going to be important.

[zamazan4ik]

I got the same issue during the ruff optimization with PGO with cargo-pgo. When I run cargo pgo optimize bench in https://github.com/astral-sh/ruff/tree/main/crates/ruff_benchmark on my Mac machine (Macbook M1 Pro, macOS 13.4 Ventura), I get the following error:

<skipped lines>
Compiling ruff_benchmark v0.0.0 (/Users/zamazan4ik/open_source/ruff/crates/ruff_benchmark)
warning: 19 warnings emitted========>  ] 287/299: ruff_benchmark, libcst, similar, ruff_notebook, ruff_python_parser, pyproject-toml, ruff_python_formatter, criterion

warning: 468 warnings emitted========> ] 290/299: libcst, ruff_notebook, ruff_python_parser, ruff_python_formatter, criterion

warning: 996 warnings emitted========> ] 292/299: libcst, ruff_python_parser, ruff_python_formatter

   Compiling ruff v0.0.287 (/Users/zamazan4ik/open_source/ruff/crates/ruff)
warning: 75 warnings emitted=========> ] 294/299: ruff, ruff_python_parser

warning: 5188 warnings emitted=======> ] 295/299: ruff

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "formatter"); 45 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "parser"); 42 warnings emitted
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ruff_benchmark` (bench "linter"); 47 warnings emitted
PGO optimized build has failed.
Cargo finished with an error (101)

Note that Ruff also enables LTO as a part of its Release profile. When I switch off LTO, cargo pgo optimize bench builds Ruff successfully.

Additional details - cargo pgo info:

cargo pgo info
[rustc version]: 1.72.0 is recent enough
[llvm-profdata]: found at /Users/zamazan4ik/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/llvm-profdata
[llvm-bolt]: could not be found (Build LLVM with BOLT and add its `bin` directory to PATH.)
[merge-fdata]: could not be found (Build LLVM with BOLT and add its `bin` directory to PATH.)
Some requirements were not satisfied

Tested the same scenario on my Linux machine (Fedora 38, x86-64) for the same scenario with the same Rust version and got the same error.

[zamazan4ik]

Another affected project - https://github.com/foundry-rs/foundry

Got the same error during cargo pgo optimize bench -- --workspace on my Linux machine with Rust 1.72:

warning: 381 warnings emitted

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `anvil-server` (lib test)
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `anvil-rpc` (lib test)
warning: 4095 warnings emitted=====> ] 1501/1535: foundry-abi(test), ethers-solc, foundry-abi, foundry-macros(test), anvil, foun...

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `foundry-macros` (lib test)
PGO optimized build has failed.
Cargo finished with an error (101)

[Kobzol]

~The ruff benchmark worked for me with Rust 1.72 on Linux~, but I haven't investigated further. Edit: it is also broken for me.

[caizixian]

This "file does not exist" error seems like a different one from the OP ("LLVM ERROR: Broken module found, compilation aborted!"), and it would be good to have a small self-contained example for your new error -- and possibly a dedicated issue since it also looks different from #63609. It feels possible to investigate in mmtk-openjdk directly, but to land a fix with a non-regression test, a small reproducer (MCVE) is going to be important.

Indeed. I have a workaround for now (cargo clean before build with profile-use). Might be a problem with incremental compilation.

[kevincox]

In my case the compilation was happening clean, so incremental doesn't seem like the problem for me.

[zamazan4ik]

Another example of hitting this error with cargo-pgo is optimizing https://github.com/bootandy/dust on my both machines (Linux machine with quite recent x86-64 Fedora 38 and Macbook M1 with macOS 13.4 Ventura) with the latest stable Rust 1.72 on both machines:

cargo pgo optimize build

<some logs are skipped>
...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `du-dust` (bin "dust")
PGO optimized build has failed.
Cargo finished with an error (101)

Also, I've checked cleaning the build directory before the PGO optimization build - didn't help. Dust also enables LTO in its Release profile.

[Kobzol]

Uhh, so I was able to reproduce this on Linux (with just lto = "thin", no need for fat LTO or 1 CGU) all the way back to 1.60.0 (older rustc versions can't build dust). Did LTO + PGO ever work? :laughing:

[zamazan4ik]

Uhh, so I was able to reproduce this on Linux (with just lto = "thin", no need for fat LTO or 1 CGU) all the way back to 1.60.0 (older rustc versions can't build dust). Did LTO + PGO ever work?

I need to recheck but AFAIK I remember some Rust projects there LTO + PGO worked fine... :)

[Kobzol]

Yeah it's something specific in the project causing it, I'm trying to minimize dust to find it.

[zamazan4ik]

Yeah it's something specific in the project causing it, I'm trying to minimize dust to find it.

@Kobzol Were you able to reproduce the issue? I have reproduced it once again with the latest Fedora 38 and Rust stable 1.73 and Ruff project on the latest commit 4113d65836e1b3baa3aa77cbdf44f83207e9f424 from its main branch.

[zamazan4ik]

Another affected project: https://github.com/googleforgames/quilkin . When lto = "fat" and codegen-units = 1 when optimizing with cargo-pgo I get the following error:

Compiling quilkin v0.8.0-dev (/home/zamazan4ik/open_source/quilkin)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `quilkin` (bin "quilkin")
PGO optimized build has failed.
Cargo finished with an error (101)

cargo-pgo info:

[rustc version]: 1.73.0 is recent enough
[2023-10-20T11:28:40Z WARN  cargo_pgo::pgo::env] llvm-profdata was resolved from PATH. Make sure that its version is compatible with rustc! If not, run `rustup component add llvm-tools-preview`.
[llvm-profdata]: found at /usr/bin/llvm-profdata
[llvm-bolt]: found at /usr/bin/llvm-bolt
[merge-fdata]: found at /usr/bin/merge-fdata

OS: Fedora 38

[zamazan4ik]

Another affected project - https://github.com/ImageOptim/gifski . On Linux with Rust 1.73 I get the same error:

cargo pgo optimize build

<logs are skipped>

Compiling gifski v1.13.0 (/home/zamazan4ik/open_source/gifski)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `gifski` (bin "gifski")
PGO optimized build has failed.
Cargo finished with an error (101)

Additional information:

cargo pgo info

[rustc version]: 1.73.0 is recent enough
[llvm-profdata]: found at /home/zamazan4ik/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata
[llvm-bolt]: found at /usr/bin/llvm-bolt
[merge-fdata]: found at /usr/bin/merge-fdata

[Kobzol]

Thanks for gifski! I managed to minimize this down to this in main.rs:

use clap::Command;

fn main() {
    Command::new("a").about("a");
}

in combination with this in Cargo.toml:

[lib]
crate-type = ["lib", "cdylib"]

and LTO=thin. I'll probably post this as a new issue and try to investigate more.

[zamazan4ik]

Found another project with the same issue - qsv.

For cargo pgo optimize build -- --release --locked -F feature_capable,apply,geocode,luau,to,polars --bin qsv I get this error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `qsv` (bin "qsv"); 363 warnings emitted
PGO optimized build has failed.
Cargo finished with an error (101)

Rust version: 1.74 QSV version: master branch on commit 531acbb072c48cbaca5d58b593243e0f5f0ec8d3

[zamazan4ik]

Found one more project with the same issue - vtracer.

For cargo pgo optimize build I get the following error:

warning: 67 warnings emitted===========> ] 69/71: vtracer

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `vtracer` (bin "vtracer")
PGO optimized build has failed.
Cargo finished with an error (101)

Rust version: 1.74 Vtracer version: master branch, 74f2a04a17d8c246d80c439fb162780160a7c3e9 commit

Vtracer by default does not enable LTO, so for reproducing the issue I made the following changed to the root Cargo.toml file:

[profile.release]
codegen-units = 1
debug         = false
lto           = true
opt-level     = 3
strip         = true

[zamazan4ik]

Another project - ocrs.

After adding codegen-units = 1 and lto = "fat" to the root Cargo.toml file and compiling with cargo pgo optimize build -- --release --bin ocrs (of course, before that I collected PGO profiles), I get the following error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `ocrs-cli` (bin "ocrs")
PGO optimized build has failed.
Cargo finished with an error (101)

Rustc version: 1.75 OS: Fedora 39 (Linux kernel 6.6.9) ocrs version: main branch, 688c21fff2785e328b4564775dc312ddafdd794a commit

[zamazan4ik]

One more - needletail.

After cargo pgo optimize bench I got an error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `needletail` (bench "benchmark")
PGO optimized build has failed.
Cargo finished with an error (101)

Rustc version:

rustc 1.76.0 (07dca489a 2024-02-04)
binary: rustc
commit-hash: 07dca489ac2d933c78d3c5158e3f43beefeb02ce
commit-date: 2024-02-04
host: x86_64-unknown-linux-gnu
release: 1.76.0
LLVM version: 17.0.6

needletail version: master branch, 25e9b931af87d5aed79ecf7a3ff32245b91ce9dc commit

[zamazan4ik]

Another one - uv. Error is the same:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `uv` (bin "uv")
PGO optimized build has failed.
Cargo finished with an error (101)

Rustc: 1.75 uv version: main branch, 12c19ce50689ee089c1374deb8282555418f9cf4 commit

[zamazan4ik]

Another project - namada. The error is the same:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `namada_benchmarks` (bench "host_env")
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `namada_benchmarks` (bench "allowed_vps")
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `namada_benchmarks` (bench "process_wrapper")
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `namada_benchmarks` (bench "native_vps")
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `namada_benchmarks` (bench "allowed_txs")
PGO optimized build has failed.
Cargo finished with an error (101)

Rustc version: 1.76 Namada version: main branch, 2535c9c8bb899f4f3ae0c9e9accaf24f30e8bff4 commit OS: Fedora 39 (Linux kernel 6.7.4)

[zamazan4ik]

Another project - boa.

Error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boa_interner` (lib test)
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boa_icu_provider` (lib test)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boa_gc` (lib test)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `scripts` (bin "regenerate-about" test)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boa_ast` (lib test)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `boa_parser` (lib test)
PGO optimized build has failed.
Cargo finished with an error (101)

Rustc version: 1.76 Boa version: main branch, 8d12ea8181d9265e2efb8596dbdf0a19013d80fe commit OS: Fedora 39 (Linux kernel 6.7.6)

[zamazan4ik]

Project - pylyzer.

After adding the following lines to the Cargo.toml:

[profile.release]
codegen-units = 1
lto = true

I get the following errors during LTO + PGO optimization build:

...
   Compiling pylyzer v0.0.52 (/home/zamazan4ik/open_source/pylyzer)
warning: 1845 warnings emitted=======> ] 143/145: pylyzer

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `pylyzer` (bin "pylyzer")
PGO optimized build has failed.
Cargo finished with an error (101)

Rustc version: 1.77.2 pylyzer version: main branch, 70c23905ae768ab554000abeefab36fe48ab54f4 commit OS: Fedora 39 (Linux kernel 6.8.4)

[joshstoik1]

I am having the same issue with my image compression program flaca, but have narrowed it down to a single (direct) dependency: mozjpeg-sys.

So long as my instrumentation runs avoid triggering those parts of the program, PGO+LTO work just fine.

Otherwise:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!

I don't know if the problem originates with that crate or just the way I'm using it. I tried stripping out all its extra features and even tested an old version without the extern "C-unwind" functionality in case that was to blame, but could never make LLVM happy with it.

:shrug:

For what it's worth, I'm running each step manually (rather than automating via cargo-pgo), using:

Debian GNU/Linux 12 (bookworm)
rustc 1.77.2 (25ef9e3d8 2024-04-09)
clang 17.0.6 (++20231208085813+6009708b4367-1~exp1~20231208085906.81)

RUSTFLAGS="-Ctarget-cpu=x86-64-v3 -Clinker=clang -Clink-arg=-fuse-ld=lld"
CC="clang"
CXX="clang++"

[zamazan4ik]

Project - https://github.com/iggy-rs/iggy

The error:

...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `tools` (bin "data-seeder-tool"); 927 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `cli` (bin "iggy"); 2249 warnings emitted
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `bench` (bin "iggy-bench"); 774 warnings emitted
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `server` (bin "iggy-server")
PGO optimized build has failed.
Cargo finished with an error (101)

OS: Fedora 40 Rustc version: 1.78 Project version: master branch, f32265229a609f3f69887024667d4093f26ae5eb commit

[insilications]

It seems to be happening with sccache

Using: cargo pgo optimize build -- --release --release --no-default-features

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `sccache` (bin "sccache");
PGO optimized build has failed.
Cargo finished with an error (101)

Rust version: 1.81.0-nightly (6b0f4b5ec 2024-06-24) sccache version: main branch d5a1787a8582bd99d3f2fc27f4c3191bd8cc76d9

[zamazan4ik]

Project - https://github.com/infinyon/fluvio-jolt

Using cargo pgo optimize bench -- --workspace --all-features --profile release-lto (before that PGO profiles were collected via cargo pgo optimize bench -- --workspace --all-features --profile release-lto.

Error:

   Compiling jolt v0.0.0 (/home/zamazan4ik/open_source/fluvio-jolt/smartmodule)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `jolt` (lib test)
PGO optimized build has failed.
Cargo finished with an error (101)

OS: Fedora 40 Rustc: 1.80.1 fluvio-jolt version: main branch, 5b85538f553d338997e20bdbb08cd6fd141e9171 commit

[zamazan4ik]

Project - https://github.com/anacrolix/possum

When I enable LTO (add lto = true to the root Cargo.toml file) and using cargo pgo optimize bench I get the following error:

module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `possum-db` (bin "possum"); 901 warnings emitted
warning: build failed, waiting for other jobs to finish...
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
rustc-LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `possum-db` (bench "possum")
PGO optimized build has failed.
Cargo finished with an error (101)

OS: Fedora 40 Rustc: 1.80.1 Possum: main branch, 35d7f9c03581c649b4c4adced8542520a5d1b54 commit