Build Issue: thread '<unknown>' has overflowed its stack

[AndyGrant]

This happens on my Ubuntu box, and on a WSL Ubuntu image. I have attempted to increase the stack size, by cd'ing into the engine dir and building via cargo rustc --release --bin velvet -- -C link-args=-Wl,-zstack-size=16777216. This did not work.

Analog Hors suggested the following code change to spawn_engine_thread(), which worked for him.

    std::thread::Builder::new().stack_size(20_000_000).spawn(move || {
        let mut engine = Engine::new(rx);
        engine.start_loop();
    }).unwrap();

A portion of the conversation that was had:

[mhonert]

Hi, thanks for the hint!

In the build pipeline I still use Rust 1.58.1, because newer versions are affected by some performance regressions. I did not yet have the time for further investigations, but perhaps the increased stack usage is also caused by some code optimization regression. In 1.58.1 the superfluous stack allocation gets optimized away and the allocation is done directly on the heap.

If you would like to compile the engine locally, I would recommend to use Rust 1.58.1 for now. Using the Rust toolchain installer (rustup), you could run rustup override set 1.58.1 from inside the source folder. That way the old version is only used for that source folder.

[mhonert]

Today I investigated the issue a bit more. Starting with Rust 1.59.0, the unnecessary stack allocation does not get optimized away anymore. As a workaround I boxed each individual field of the NeuralNetParams struct, instead of the whole struct. This seemed to help the compiler/optimizer and now the struct instance is created directly on the heap again and the stack does not overflow anymore.

I could also mitigate some of the performance regressions by explicitly marking certain functions to be always inlined, but unfortunately binaries compiled with 1.63.0 are still around 8% slower than the ones produced by 1.58.1.

[AndyGrant]

Thanks. I will set Velvet to use 1.58.1 for the time being, and I'll check newer versions every so often for CCC. Can always reach out to me via email (andrew@grantnet.us), or on Discord.