cargo shuttle run fails because of a zbus error (OSX)

[xgillard]

Whenever I try to run my project locally, it ends up with a zbus error; I was however able to deploy it successfully. I do believe this issue is cargo shuttle related as the compilation seems to work ok but it only crashes when the service tries to start. If I add a 'main.rs' module where I manually spawn axum, it works fine.

Some more info about my config:

OS: Mac OS Ventura (13.0.1) -- arm64 Cargo: cargo 1.67.0 (8ecd4f20a 2023-01-10)

And here is a log of how it goes:

xgillard@mac-SE22-107 backtest-rs % cargo shuttle run             
Building /Users/xgillard/Documents/REPO/PERSO/backtest-rs
   Compiling version_check v0.9.4
   Compiling proc-macro2 v1.0.50
   Compiling quote v1.0.23
   Compiling unicode-ident v1.0.6
   Compiling syn v1.0.107
   Compiling cfg-if v1.0.0
   Compiling autocfg v1.1.0
   Compiling libc v0.2.139
   Compiling memchr v2.5.0
   Compiling pin-project-lite v0.2.9
   Compiling futures-core v0.3.26
   Compiling once_cell v1.17.0
   Compiling futures-io v0.3.26
   Compiling log v0.4.17
   Compiling bytes v1.3.0
   Compiling typenum v1.16.0
   Compiling pin-utils v0.1.0
   Compiling cc v1.0.79
   Compiling scopeguard v1.1.0
   Compiling serde_derive v1.0.152
   Compiling smallvec v1.10.0
   Compiling itoa v1.0.5
   Compiling crossbeam-utils v0.8.14
   Compiling futures-task v0.3.26
   Compiling serde v1.0.152
   Compiling event-listener v2.5.3
   Compiling percent-encoding v2.2.0
   Compiling fastrand v1.8.0
   Compiling futures-channel v0.3.26
   Compiling core-foundation-sys v0.8.3
   Compiling waker-fn v1.1.0
   Compiling bitflags v1.3.2
   Compiling tinyvec_macros v0.1.0
   Compiling parking v2.0.0
   Compiling futures-util v0.3.26
   Compiling parking_lot_core v0.9.6
   Compiling thiserror v1.0.38
   Compiling unicode-bidi v0.3.10
   Compiling untrusted v0.7.1
   Compiling minimal-lexical v0.2.1
   Compiling lazy_static v1.4.0
   Compiling async-task v4.3.0
   Compiling either v1.8.1
   Compiling signal-hook v0.3.14
   Compiling mime v0.3.16
   Compiling rustversion v1.0.11
   Compiling atomic-waker v1.1.0
   Compiling rustls v0.20.8
   Compiling parking_lot_core v0.8.6
   Compiling pkg-config v0.3.26
   Compiling vcpkg v0.2.15
   Compiling crossbeam-queue v0.3.8
   Compiling paste v1.0.11
   Compiling async-trait v0.1.64
   Compiling native-tls v0.2.11
   Compiling unicode_categories v0.1.1
   Compiling serde_json v1.0.91
   Compiling futures-sink v0.3.26
   Compiling remove_dir_all v0.5.3
   Compiling unicode-segmentation v1.10.1
   Compiling httparse v1.8.0
   Compiling hex v0.4.3
   Compiling base64 v0.21.0
   Compiling byteorder v1.4.3
   Compiling anyhow v1.0.68
   Compiling crc-catalog v1.1.1
   Compiling askama_escape v0.10.3
   Compiling crc32fast v1.3.2
   Compiling crc-catalog v2.2.0
   Compiling time-core v0.1.0
   Compiling doc-comment v0.3.3
   Compiling semver v1.0.16
   Compiling camino v1.1.2
   Compiling dotenvy v0.15.6
   Compiling encoding_rs v0.8.31
   Compiling dotenv v0.15.0
   Compiling instant v0.1.12
   Compiling value-bag v1.0.0-alpha.9
   Compiling slab v0.4.7
   Compiling lock_api v0.4.9
   Compiling generic-array v0.14.6
   Compiling ahash v0.7.6
   Compiling num-traits v0.2.15
   Compiling tokio v1.22.0
   Compiling polling v2.5.2
   Compiling async-io v1.12.0
   Compiling indexmap v1.9.2
   Compiling num-integer v0.1.45
   Compiling async-process v1.6.0
   Compiling unicase v2.6.0
   Compiling proc-macro-error-attr v1.0.4
   Compiling cookie v0.16.2
   Compiling proc-macro-error v1.0.4
   Compiling tracing-core v0.1.30
   Compiling thread_local v1.1.4
   Compiling form_urlencoded v1.1.0
   Compiling tinyvec v1.6.0
   Compiling http v0.2.8
   Compiling ring v0.16.20
   Compiling blake3 v0.3.8
   Compiling sharded-slab v0.1.4
   Compiling itertools v0.10.5
   Compiling heck v0.4.1
   Compiling crc v2.1.0
   Compiling rustls-pemfile v1.0.2
   Compiling libsqlite3-sys v0.24.2
   Compiling crc v3.0.1
   Compiling time-macros v0.2.6
   Compiling unicode-normalization v0.1.22
   Compiling http-body v0.4.5
   Compiling headers-core v0.2.0
   Compiling time v0.3.17
   Compiling socket2 v0.4.7
   Compiling signal-hook-registry v1.4.0
   Compiling getrandom v0.2.8
   Compiling num_cpus v1.15.0
   Compiling cpufeatures v0.2.5
   Compiling tempfile v3.3.0
   Compiling time v0.1.45
   Compiling nom v7.1.3
   Compiling futures-lite v1.12.0
   Compiling concurrent-queue v2.1.0
   Compiling crossbeam-channel v0.5.6
   Compiling iana-time-zone v0.1.53
   Compiling core-foundation v0.9.3
   Compiling security-framework-sys v2.8.0
   Compiling idna v0.3.0
   Compiling stringprep v0.1.2
   Compiling axum-core v0.3.2
   Compiling axum v0.6.4
   Compiling flate2 v1.0.25
   Compiling rustrict v0.5.12
   Compiling spin v0.9.4
   Compiling rand_core v0.6.4
   Compiling parking_lot v0.12.1
   Compiling atoi v0.4.0
   Compiling atoi v1.0.0
   Compiling parking_lot v0.11.2
   Compiling mime_guess v2.0.4
   Compiling async-lock v2.6.0
   Compiling async-channel v1.8.0
   Compiling block-buffer v0.10.3
   Compiling crypto-common v0.1.6
   Compiling digest v0.9.0
   Compiling crypto-mac v0.11.1
   Compiling crypto-mac v0.8.0
   Compiling block-buffer v0.9.0
   Compiling pipe v0.4.0
   Compiling security-framework v2.8.2
   Compiling sqlformat v0.1.8
   Compiling sqlformat v0.2.1
   Compiling url v2.3.1
   Compiling hashbrown v0.12.3
   Compiling hashbrown v0.11.2
   Compiling rand_chacha v0.3.1
   Compiling webpki v0.22.0
   Compiling sct v0.7.0
   Compiling chrono v0.4.23
   Compiling futures-intrusive v0.4.2
   Compiling async-executor v1.5.0
   Compiling futures-executor v0.3.26
   Compiling blocking v1.3.0
   Compiling digest v0.10.6
   Compiling hmac v0.11.0
   Compiling sha2 v0.9.9
   Compiling rand v0.8.5
   Compiling webpki-roots v0.22.6
   Compiling hashlink v0.8.1
   Compiling hashlink v0.7.0
   Compiling sha2 v0.10.6
   Compiling sha1 v0.10.5
   Compiling hmac v0.12.1
   Compiling headers v0.3.8
   Compiling ctor v0.1.26
   Compiling futures-macro v0.3.26
   Compiling thiserror-impl v1.0.38
   Compiling tokio-macros v1.8.2
   Compiling pin-project-internal v1.0.12
   Compiling tracing-attributes v0.1.23
   Compiling strum_macros v0.24.3
   Compiling displaythis-impl v1.0.23
   Compiling shuttle-codegen v0.9.0
   Compiling pin-project v1.0.12
   Compiling displaythis v1.0.23
   Compiling flume v0.10.14
   Compiling mio v0.8.5
   Compiling tracing v0.1.37
   Compiling want v0.3.0
   Compiling kv-log-macro v1.0.7
   Compiling tracing-log v0.1.3
   Compiling strum v0.24.1
   Compiling zip v0.5.13
   Compiling tracing-subscriber v0.3.16
   Compiling async-global-executor v2.3.1
   Compiling axum-core v0.1.2
   Compiling async-std v1.12.0
   Compiling futures v0.3.26
   Compiling tokio-util v0.7.4
   Compiling tokio-rustls v0.23.4
   Compiling tower v0.4.13
   Compiling tokio-stream v0.1.11
   Compiling async-compat v0.2.1
   Compiling serde_urlencoded v0.7.1
   Compiling serde_path_to_error v0.1.9
   Compiling toml v0.5.11
   Compiling bincode v1.3.3
   Compiling cargo-platform v0.1.2
   Compiling uuid v1.3.0
   Compiling sqlx-rt v0.6.2
   Compiling tower-http v0.3.5
   Compiling h2 v0.3.15
   Compiling simple_excel_writer v0.2.0
   Compiling async-native-tls v0.3.3
   Compiling async-session v3.0.0
   Compiling askama_shared v0.12.2
   Compiling cargo_metadata v0.15.3
   Compiling shuttle-common v0.9.0
   Compiling sqlx-core v0.6.2
   Compiling sqlx-rt v0.5.13
   Compiling sqlx-core v0.5.13
   Compiling hyper v0.14.23
   Compiling askama_derive v0.11.2
   Compiling askama v0.11.1
   Compiling askama_axum v0.1.0
   Compiling hyper-rustls v0.23.2
   Compiling sqlx-macros v0.6.2
   Compiling reqwest v0.11.14
   Compiling sqlx-macros v0.5.13
   Compiling yahoo_finance_api v1.3.0
   Compiling sqlx v0.6.2
   Compiling sqlx v0.5.13
   Compiling async-sqlx-session v0.4.0
   Compiling axum-extra v0.4.2
   Compiling shuttle-service v0.9.0
   Compiling axum-sessions v0.4.1
   Compiling backtest-rs-00000000-0000-0000-0000-000000000000 v0.1.0 (/Users/xgillard/Documents/REPO/PERSO/backtest-rs)
    Finished dev [unoptimized + debuginfo] target(s) in 34.31s

Starting backtest-rs on http://127.0.0.1:8000
zsh: bus error  cargo shuttle run

[oddgrd]

Hey @xgillard! Thanks for reporting this. We have had similar reports of memory issues recently, especially from mac users with actix-web projects. When we've had issues like this in the past, the culprit has been our FFI, but we have yet to figure out why it's just happening on mac OS in this case.

I was able to do a local run on a mac (x86_64) when using the pre-built binary from our latest release, building the project with rust 1.65 (cargo +1.65 build or setting the default toolchain to 1.65) and then doing a cargo shuttle run. I have not tried this on an arm64 mac, however.

[iulianbarbu]

I am taking a look at this bug, FYI. What I noticed is that installing cargo-shuttle through cargo install cargo-shuttle installs the 0.9.0 version and trying to run locally a project with actix-web (a simple one, right after running cargo shuttle init) results in the bus error each time. However, if I compile locally the release version of 0.9.0 cargo-shuttle binary and run it against the very same project, the actix-web server runs without the bus error.

@oddgrd , do you have any ideas why this might happen?

[oddgrd]

Thanks for looking into this! Hmm, did you compile locally from the 0.9 tag or the latest main? Or do you mean to say you tried the pre-built binary from the release assets? In any case, which version of rustc did you use?

[iulianbarbu]

Hey @oddgrd ! I compiled the source code from 0.9 tag. I used rustc 1.67.1 on a M1 macbook pro with MacOS Ventura 13.2. Also, I noticed that 0.10.0 was released and it fails the same.

The latest updates on this topic from my side are that: 1) I'm having trouble with debugging this without the debug info - cargo build doesn't generate them or at least I haven't found them yet - for the generated cargo-shuttle Mach-O binary file.

2) I find it a bit strange that the installed binary - the one obtained from crates.io, which downloads the source code and compiles it locally - is different in size than the binary obtained through cargo build --release on the 0.9 tag. I do not think it's a stripping issue because I've tried stripping down both binaries and they are almost halved after the operation, and the delta still remains.

3) I am not used to debugging on an arm64 darwin machine. After trying to reproduce the issue for few more times, I noticed that starting an actix-web HTTP server with the 0.9 version (installed from crates.io) fails intermittently when running everything inside lldb, which makes me think there might be a race condition. However, I would need to look at a backtrace to pin point the place where the bus error is triggered. LE: the bad access stacktrace looks like this, but is more or less obfuscated:

(lldb) bt
* thread #3, name = 'main', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x1031048d0)
  * frame #0: 0x00000001031048d0
    frame #1: 0x0000000100055bd0 cargo-shuttle`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h1936596fa66aec54 + 160
    frame #2: 0x00000001000c6960 cargo-shuttle`_$LT$futures_util..future..future..catch_unwind..CatchUnwind$LT$Fut$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h5a1a5ed4ceee3073 + 36
    frame #3: 0x000000010005ccd0 cargo-shuttle`cargo_shuttle::Shuttle::run::_$u7b$$u7b$closure$u7d$$u7d$::h3bbdeee18d6a05c4 + 4396
    frame #4: 0x00000001000769e0 cargo-shuttle`tokio::runtime::park::CachedParkThread::block_on::h143bc585254942a5 + 512
    frame #5: 0x00000001000c5d10 cargo-shuttle`tokio::runtime::scheduler::multi_thread::MultiThread::block_on::h2dde6dff0ecf86ac + 104
    frame #6: 0x00000001000d2810 cargo-shuttle`cargo_shuttle::main::h95dae9533139c17d + 288
    frame #7: 0x00000001000aa6dc cargo-shuttle`std::sys_common::backtrace::__rust_begin_short_backtrace::h5e548e71a67487db + 12
    frame #8: 0x00000001000c9128 cargo-shuttle`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::hbb9535e5e1976eb9 + 24
    frame #9: 0x0000000100ad8c30 cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::ha1c2447b9b665e13 at function.rs:606:13 [opt]

Meanwhile, do you have any ideas for why the crates.io installed binary is different in size than my local release build? I am not sure what's the difference in running cargo install from crates.io vs cargo build --release on the same tag as the ones present on crates.io.

[iulianbarbu]

I dug a bit more and managed to get some useful information. Below is a more meaningful backtrace. To help a bit, what I found useful can be found in frames 10-13.

(lldb) bt
* thread #1, name = 'main', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=2, address=0x10700a2f0)
  * frame #0: 0x000000010700a2f0
    frame #1: 0x00000001000e0580 cargo-shuttle`shuttle_service::Bootstrapper::bootstrap::_$u7b$$u7b$closure$u7d$$u7d$::hb764c6a56431fb33((null)=ResumeTy @ 0x000000016fde2ef0) at lib.rs:455:27
    frame #2: 0x0000000100008cdc cargo-shuttle`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hbf0b89ecb50007ec(self=Pin<&mut core::panic::unwind_safe::AssertUnwindSafe<shuttle_service::{impl#0}::bootstrap::{async_fn_env#0}>> @ 0x000000016fde2f78, cx=0x000000016fdf40d0) at unwind_safe.rs:296:9
    frame #3: 0x00000001000d8580 cargo-shuttle`_$LT$futures_util..future..future..catch_unwind..CatchUnwind$LT$Fut$GT$$u20$as$u20$core..future..future..Future$GT$::poll::_$u7b$$u7b$closure$u7d$$u7d$::h69ad8721f3c6f557 at catch_unwind.rs:36:42
    frame #4: 0x0000000100008f6c cargo-shuttle`_$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h31f1ae34f14b68e8(self=AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<shuttle_service::{impl#0}::bootstrap::{async_fn_env#0}>>> @ 0x000000016fde2fd0, _args=<unavailable>) at unwind_safe.rs:271:9
    frame #5: 0x00000001000bdd64 cargo-shuttle`std::panicking::try::do_call::h74901a545fee1ecf(data="hF\xdfo\U00000001") at panicking.rs:483:40
    frame #6: 0x00000001000c14f8 cargo-shuttle`__rust_try + 32
    frame #7: 0x00000001000ba564 cargo-shuttle`std::panicking::try::h59d01eac51d130dd(f=AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<shuttle_service::{impl#0}::bootstrap::{async_fn_env#0}>>> @ 0x000000016fde3180) at panicking.rs:447:19
    frame #8: 0x0000000100055bb4 cargo-shuttle`std::panic::catch_unwind::ha960c18ababb2db6(f=AssertUnwindSafe<futures_util::future::future::catch_unwind::{impl#1}::poll::{closure_env#0}<core::panic::unwind_safe::AssertUnwindSafe<shuttle_service::{impl#0}::bootstrap::{async_fn_env#0}>>> @ 0x000000016fde31d0) at panic.rs:137:14
    frame #9: 0x00000001000d84c4 cargo-shuttle`_$LT$futures_util..future..future..catch_unwind..CatchUnwind$LT$Fut$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hc1c633e231e26562(self=Pin<&mut futures_util::future::future::catch_unwind::CatchUnwind<core::panic::unwind_safe::AssertUnwindSafe<shuttle_service::{impl#0}::bootstrap::{async_fn_env#0}>>> @ 0x000000016fde32c0, cx=0x000000016fdf40d0) at catch_unwind.rs:36:9
    frame #10: 0x0000000100093eec cargo-shuttle`shuttle_service::loader::Loader::load::_$u7b$$u7b$closure$u7d$$u7d$::hdb8964ccc5d408bc((null)=ResumeTy @ 0x000000016fde3c10) at loader.rs:79:13
    frame #11: 0x000000010007dce8 cargo-shuttle`cargo_shuttle::Shuttle::local_run::_$u7b$$u7b$closure$u7d$$u7d$::hf474781a53833706((null)=ResumeTy @ 0x000000016fde5d08) at lib.rs:442:67
    frame #12: 0x00000001000758bc cargo-shuttle`cargo_shuttle::Shuttle::run::_$u7b$$u7b$closure$u7d$$u7d$::h83fc8cdfc70b325d((null)=ResumeTy @ 0x000000016fdf10a8) at lib.rs:77:63
    frame #13: 0x0000000100083604 cargo-shuttle`cargo_shuttle::main::_$u7b$$u7b$closure$u7d$$u7d$::ha519b8e4a453dfce((null)=ResumeTy @ 0x000000016fdf3f60) at main.rs:9:52
    frame #14: 0x0000000100044370 cargo-shuttle`tokio::runtime::park::CachedParkThread::block_on::_$u7b$$u7b$closure$u7d$$u7d$::hc0f9571483ced3f7 at park.rs:272:63
    frame #15: 0x0000000100043e64 cargo-shuttle`tokio::runtime::park::CachedParkThread::block_on::h40c86347832f7b42 at coop.rs:102:5
    frame #16: 0x0000000100043dfc cargo-shuttle`tokio::runtime::park::CachedParkThread::block_on::h40c86347832f7b42 [inlined] tokio::runtime::coop::budget::h99a2fce96aa0dfe8(f={closure_env#0}<cargo_shuttle::main::{async_block_env#0}> @ 0x000000016fdf5648) at coop.rs:68:5
    frame #17: 0x0000000100043d80 cargo-shuttle`tokio::runtime::park::CachedParkThread::block_on::h40c86347832f7b42(self=0x000000016fdf56ef, f={async_block_env#0} @ 0x000000016fdf56f0) at park.rs:272:31
    frame #18: 0x00000001000c9b80 cargo-shuttle`tokio::runtime::context::BlockingRegionGuard::block_on::h428d4a9cc4bb21af(self=0x000000016fdf6c50, f={async_block_env#0} @ 0x000000016fdf6c68) at context.rs:255:13
    frame #19: 0x00000001000c269c cargo-shuttle`tokio::runtime::scheduler::multi_thread::MultiThread::block_on::he9e2d355f3c74c46(self=0x000000016fdfd680, handle=0x000000016fdfd668, future={async_block_env#0} @ 0x000000016fdf9710) at mod.rs:66:9
    frame #20: 0x000000010008c1a8 cargo-shuttle`tokio::runtime::runtime::Runtime::block_on::h6bd7ac9891e754bb(self=0x000000016fdfd658, future={async_block_env#0} @ 0x000000016fdfd7b8) at runtime.rs:281:45
    frame #21: 0x00000001000c2544 cargo-shuttle`cargo_shuttle::main::h947536bd3b729aca at main.rs:17:5
    frame #22: 0x00000001000461f4 cargo-shuttle`core::ops::function::FnOnce::call_once::h66f29fa036c63bee((null)=(cargo-shuttle`cargo_shuttle::main::h947536bd3b729aca at main.rs:6), (null)=<unavailable>) at function.rs:507:5
    frame #23: 0x00000001000576d0 cargo-shuttle`std::sys_common::backtrace::__rust_begin_short_backtrace::hfee08ea92bea8ce1(f=(cargo-shuttle`cargo_shuttle::main::h947536bd3b729aca at main.rs:6)) at backtrace.rs:121:18
    frame #24: 0x0000000100043910 cargo-shuttle`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h688bc618c1ea3f44 at rt.rs:166:18
    frame #25: 0x0000000102126764 cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_once::ha1c2447b9b665e13 at function.rs:606:13 [opt]
    frame #26: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::panicking::try::do_call::ha57d6d1e9532dc1f at panicking.rs:483:40 [opt]
    frame #27: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::panicking::try::hca0526f287961ecd at panicking.rs:447:19 [opt]
    frame #28: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::panic::catch_unwind::hdcaa7fa896e0496a at panic.rs:137:14 [opt]
    frame #29: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h142ec071d3766871 at rt.rs:148:48 [opt]
    frame #30: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::panicking::try::do_call::h95f5e55d6f048978 at panicking.rs:483:40 [opt]
    frame #31: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::panicking::try::h0fa00e2f7b4a5c64 at panicking.rs:447:19 [opt]
    frame #32: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c [inlined] std::panic::catch_unwind::h1765f149814d4d3e at panic.rs:137:14 [opt]
    frame #33: 0x000000010212675c cargo-shuttle`std::rt::lang_start_internal::h00a235e820a7f01c at rt.rs:148:20 [opt]
    frame #34: 0x00000001000438dc cargo-shuttle`std::rt::lang_start::hca7ae022b7a49b1a(main=(cargo-shuttle`cargo_shuttle::main::h947536bd3b729aca at main.rs:6), argc=2, argv=0x000000016fdff1e0, sigpipe='\0') at rt.rs:165:17
    frame #35: 0x00000001000c25ec cargo-shuttle`main + 36
    frame #36: 0x00000001b025be50 dyld`start + 2544

The culprit is this block of code:

I am not sure what's the root cause but I'll follow up once I understand better its role.

[oddgrd]

Thanks for digging deep on this @iulianbarbu! :pray:

I am not sure why the binary size is different, no, as far as I know cargo install ... is equivalent to cargo build --release.

Yes, the culprit is indeed there, to load dynamic libraries (user projects) we need to use an unsafe FFI. With rust's safety guarantees, it should not be possible for this kind of error to happen in safe code. I linked this issue earlier, where the issue was misalignment due to different versions of tokio across the FFI. The question is, why is it happening now, and why is it only (or at least the vast majority of cases) on MacOS?

Did you try my suggestion from above, by the way? I was able to run actix (local runs with actix have consistenly segfaulted on mac) locally on an x86_64 mac with that. I download the mac binary release (I used 0.9.0, but 0.10.0 should be the same), which is compiled with rustc 1.65, then I build the project I want to run with 1.65 (cargo +1.65 build), and then I cargo shuttle run. This worked the last time I tested it, but I arrived at it by experimentation, I have limited experience at this level. My theory is that the misalignment stems from using a binary compiled with a different rustc than what the project to run is compiled with. I don't know why it only happens on MacOs. I may also be completely misguided here, so please, feel free to debunk my theory! :smile:

Again, thank you for looking into this, we would love to get this fixed!

[oddgrd]

I did some more digging, and tokio makes no guarantees about its layout staying the same across FFI even if it's the same version. It seems to work consistently in a controlled environment like our deployers, but we may need to refactor the local runs. :shrug:

[oddgrd]

This should be resolved with the release of 0.12 :partying_face: Feel free to re-open if you still experience this.