search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
98.63k stars 12.74k forks source link

Massive memory usage when type annotations needed #125651

Open logistic-bot opened 5 months ago

logistic-bot commented 5 months ago

This is a bit of a complicated issue to report, because it happened seemingly at random after introducing a change in my code. This change lead to a compile error, and afterward a single rustc thread caused my computer to freeze with 15gb of ram usage, and 100% of a single cpu core.

An archive of my project is included. This happens when running any of cargo check, clippy, build in the project folder.

Sorry, this is not a minimally-reproducible example, because it happened on a complex project, and I am not sure what part of my code causes this behavior to show.

The behavior described below does not appear if the following lines in src/routes/game.rs are commented. In that case, cargo reports some warnings, then exits normally.

    // (near line 30)
    if let Some(game) = game.first() {
        //let language_ids = game.languages.iter().map(|x| x.id).collect();
        //let other_languages = conn
        //    .run(|c| {
        //        language::table
        //            .filter(language::columns::id.is_distinct_from(language_ids))
        //            .load(c)
        //    })
        //    .await?;
        Ok(Template::render(
            "game_full",
            context! {
                game,
            },
        ))
    } else {
        Err(ApplicationError::NoSuchGame { id })
    }

I expected to see this happen: rustc to show the error and eventually exit

Instead, this happened: cargo shows the following error:

error[E0283]: type annotations needed
    --> src/routes/game.rs:31:13
     |
31   |         let language_ids = game.languages.iter().map(|x| x.id).collect();
     |             ^^^^^^^^^^^^                                       ------- type must be known at this point
     |
     = note: cannot satisfy `_: FromIterator<i32>`
note: required by a bound in `std::iter::Iterator::collect`
    --> /home/khais/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/iter/traits/iterator.rs:1999:19
     |
1999 |     fn collect<B: FromIterator<Self::Item>>(self) -> B
     |                   ^^^^^^^^^^^^^^^^^^^^^^^^ required by this bound in `Iterator::collect`
help: consider giving `language_ids` an explicit type
     |
31   |         let language_ids: Vec<_> = game.languages.iter().map(|x| x.id).collect();
     |                         ++++++++

^C  Building [=======================> ] 380/381: ttrpgs(bin)

and then consumes more and more ram until I am forced to kill the process

ps capture of the offending process:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
khais    2303047 99.6 57.1 31193988 18702996 pts/1 SNl+ 12:32   5:51 /home/khais/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name ttrpgs --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=175 --crate-type bin --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 -C metadata=43694494ee23ddb7 -C extra-filename=-43694494ee23ddb7 --out-dir /home/khais/src/ttrpgs_bug_report/target/debug/deps -C incremental=/home/khais/src/ttrpgs_bug_report/target/debug/incremental -L dependency=/home/khais/src/ttrpgs_bug_report/target/debug/deps --extern chrono=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libchrono-df20a4981cc1ae50.rmeta --extern csv=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libcsv-914302b8947f3c7d.rmeta --extern diesel=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libdiesel-e10c470af87fb562.rmeta --extern dotenvy=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libdotenvy-9821549988c77cf3.rmeta --extern regex=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libregex-e744e5e84862ee01.rmeta --extern rocket=/home/khais/src/ttrpgs_bug_report/target/debug/deps/librocket-f85756e2b94ef3e4.rmeta --extern rocket_db_pools=/home/khais/src/ttrpgs_bug_report/target/debug/deps/librocket_db_pools-0b6644166093e0a7.rmeta --extern rocket_dyn_templates=/home/khais/src/ttrpgs_bug_report/target/debug/deps/librocket_dyn_templates-d688c2b13a9ab27e.rmeta --extern rocket_sync_db_pools=/home/khais/src/ttrpgs_bug_report/target/debug/deps/librocket_sync_db_pools-0fae6578655fff9d.rmeta --extern sea_orm=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libsea_orm-c9115e6c44d3c8a0.rmeta --extern serde=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libserde-2640d9b683a951ea.rmeta --extern smol=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libsmol-abd1ae4bbcdc3952.rmeta --extern thiserror=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libthiserror-3084c89a5d4ce759.rmeta --extern tokio=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libtokio-5825f409e9792f63.rmeta --extern ttrpgs=/home/khais/src/ttrpgs_bug_report/target/debug/deps/libttrpgs-60d690e0592caaf5.rmeta -C linker=clang -C link-arg=-fuse-ld=lld -L native=/usr/lib

Meta

rustc --version --verbose:

rustc 1.78.0 (9b00956e5 2024-04-29)
binary: rustc
commit-hash: 9b00956e56009bab2aa15d7bff10916599e3d6d6
commit-date: 2024-04-29
host: x86_64-unknown-linux-gnu
release: 1.78.0
LLVM version: 18.1.2

This also happens in nightly (not sure how to check the version). rustup +nightly show

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/khais/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu (default)
nightly-x86_64-unknown-linux-gnu

active toolchain
----------------

nightly-x86_64-unknown-linux-gnu (overridden by +toolchain on the command line)
rustc 1.80.0-nightly (84b40fc90 2024-05-27)

System information:

Linux 6.8.5-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Thu, 11 Apr 2024 01:47:18 +0000 x86_64 unknown unknown GNU/Linux
CPU: 13th Gen Intel i5-13400F (16) @ 4.600GHz

ttrpgs_bug_report.tar.gz

Backtrace

This is not applicable here, since compilation does not finish.

I am happy to provide more details if requested. I am not very experienced in compiler internals, and this is the first time I'm submitting a bug report here.

Noratrieb commented 5 months ago

does it also reproduce on the latest nightly?

logistic-bot commented 5 months ago

Sorry for being unclear. Yes, this also happens in latest nightly rustc 1.80.0-nightly (84b40fc90 2024-05-27).

workingjubilee commented 5 months ago

This also happens in nightly (not sure how to check the version).

rustc +nightly --version --verbose

workingjubilee commented 5 months ago

This has a reproducer, it's just the entire .tar.gz of a project with, yk, diesel and sqlx, which means it is not minimal by far.

logistic-bot commented 5 months ago

rustc +nightly --version --verbose

rustc 1.80.0-nightly (da159eb33 2024-05-28)
binary: rustc
commit-hash: da159eb331b27df528185c616b394bb0e1d2a4bd
commit-date: 2024-05-28
host: x86_64-unknown-linux-gnu
release: 1.80.0-nightly
LLVM version: 18.1.6

This version still reproduces the issue.

I'm happy to help to try to find a more minimal reproducer, are there any tips on how to do that?

workingjubilee commented 5 months ago

the problem is that all the automated ways of doing so require a script that you can run to completion, and this issue, well,

logistic-bot commented 5 months ago

Taking a shot in the dark here, but would it be possible to make the script interrupt the compilation process if it ever uses more than some amount of ram?

saethlin commented 5 months ago

and this issue, well,

timeout 1m cargo check or such will do the trick. Or running this in a docker container with a low memory like 4 GB should make the offending rustc process exit due to a SIGKILL promptly.

logistic-bot commented 5 months ago

Update: This still happens with the following versions:

rustc 1.81.0-nightly (8337ba918 2024-06-12)
binary: rustc
commit-hash: 8337ba9189de188e2ed417018af2bf17a57d51ac
commit-date: 2024-06-12
host: x86_64-unknown-linux-gnu
release: 1.81.0-nightly
LLVM version: 18.1.7
rustc 1.79.0 (129f3b996 2024-06-10)
binary: rustc
commit-hash: 129f3b9964af4d4a709d1383930ade12dfe7c081
commit-date: 2024-06-10
host: x86_64-unknown-linux-gnu
release: 1.79.0
LLVM version: 18.1.7

Is there any way I can help debug this issue?

saethlin commented 5 months ago

Is there any way I can help debug this issue?

Yes. Produce a smaller program that reproduces the rampant memory usage.

saethlin commented 5 months ago

I've minimized by hand to this Cargo.toml:

[package]
name = "ttrpgs"
version = "0.1.0"
edition = "2021"

[dependencies]
diesel = { version = "2.1.6", features = ["postgres", "chrono"] }
rocket_sync_db_pools = { version = "0.1.0", features = ["diesel_postgres_pool"] }

With this src/main.rs:

diesel::table! {
    language (id) {
        id -> Int4,
        name -> Varchar,
        created_at -> Timestamp,
        updated_at -> Timestamp,
    }
}

pub struct GameEntry {
    pub id: i32,
}

pub struct GameEntryCard {
    pub game_entry: GameEntry,
    pub languages: Vec<Language>,
}

pub type GameEntryCardDBValue = ();

impl GameEntryCard {
    pub fn from_db_result(value: GameEntryCardDBValue) -> Vec<Self> {
        loop {}
    }
}

pub struct Language {
    pub id: i32,
}

use diesel::PgExpressionMethods;
use diesel::QueryDsl;
use diesel::RunQueryDsl;

use rocket_sync_db_pools::{database, diesel};

#[database("ttrpgs")]
pub struct Connection(diesel::PgConnection);

pub async fn details(game: &GameEntryCard, conn: Connection) {
    let language_ids = game.languages.iter().map(|x| x.id).collect();
    let other_languages = conn
        .run(|c| {
            language::table
                .filter(language::columns::id.is_distinct_from(language_ids))
                .load(c)
        })
        .await.unwrap();
}

fn main() {}

Taking a break for now. This is still very far from minimal because all the type system complexity is in the diesel APIs being used here.

pacak commented 5 months ago

A bit smaller - no macro and all traits are explicitly imported

This Cargo.toml

[package]
name = "ttrpgs"
version = "0.1.0"
edition = "2021"

[dependencies]

diesel = { version = "2.1.6", features = ["32-column-tables", "postgres_backend"], default-features = false }

with this body

use diesel::expression::Expression;
use diesel::expression::ValidGrouping;
use diesel::internal::table_macro::FromClause;
use diesel::internal::table_macro::SelectStatement;
use diesel::internal::table_macro::StaticQueryFragmentInstance;
use diesel::query_builder::AsQuery;
use diesel::sql_types::Int4;
use diesel::PgExpressionMethods;
use diesel::QueryDsl;
use diesel::QuerySource;
use diesel::Table;

pub struct table;

impl QuerySource for table {
    type FromClause = StaticQueryFragmentInstance<()>;
    type DefaultSelection = <Self as Table>::AllColumns;
}

impl AsQuery for table {
    type Query = SelectStatement<FromClause<Self>>;
}
impl Table for table {
    type AllColumns = (id,);
}

pub struct id;
impl Expression for id {
    type SqlType = Int4;
}

impl ValidGrouping<()> for id {}

pub fn run<F, R>(__f: F) -> R {}

pub fn details(languages: &[i32]) {
    let language_ids = languages.iter().map(|x| x + 1).collect();
    run(|c| table.filter(id.is_distinct_from(language_ids)));
}
pacak commented 5 months ago

Managed to reduce down to a single 9Mb rs file with no external dependencies, chewing on that for a bit.

pacak commented 5 months ago

30ish seconds per check, 6 checks in parallel, 3Mb now. Was down to 700kb at one point but messed up the check so went back to slower/more reliable. My :potato: PC needs more memory.

Making progress. Some result tomorrow hopefully.

pacak commented 5 months ago

Night was not very productive due to me messing up some arguments, but it's been running since morning and now is sitting at 900kb as of right now, going down by about 1.5kb every 30 seconds or so.

1000LOC, 40 traits and 24 structs. At this point it can be done by hand I guess but should be done tomorrow either way.

Noratrieb commented 5 months ago

that's really cool! what tools are you using?

pacak commented 5 months ago

that's really cool! what tools are you using?

Interestingness criteria for treereduce-rust is that rustc gets killed by the memory usage.

Tempted to make something like cargo-flatten that would behave like cargo-expand, but change expanded stdlib derives back to #[derive(Clone, Copy, ...)] form they are usually not interesting and expanded versions require to enable a bunch of nightly features.

Under 20kb now, rerunning it again with "rustc uses over 5Gb" criteria, previous attempt lead to something that uses over 500Mb, but succeeds.

pacak commented 5 months ago

Behold!

trait Expression {}

trait ValidGrouping {
    type IsAggregate;
}

trait MixedAggregates<Other> {}

trait NonAggregate {}

impl<T> NonAggregate for T where T: ValidGrouping {}

impl<F, Predicate> FilterDsl<Predicate> for SelectStatement<F> where (): Expression + NonAggregate {}

struct SelectStatement<From> {}

trait FilterDsl<Predicate> {}

impl<A, B> ValidGrouping for (A, B)
where
    (A, B): ValidGrouping,
    (): MixedAggregates<<(A, B) as ValidGrouping>::IsAggregate>,
{
    type IsAggregate = ();
}
pacak commented 5 months ago

From what I can see it gets stuck inside rustc_trait_selection going recursively deep and wide