Closed vinniejames closed 10 months ago
I ran in the same exception with an old Intel-based Mac. As a quick workaround, you could set to "false" the "auto_sys_proxy" property of ClientBuilder inside reqwest::Client.
As far as I understood, the problem is in retrieving system proxy information when calling Client::default()
Mmm interesting, this seems to be a Cargo issue:
https://github.com/rust-lang/cargo/issues/12784 https://github.com/rust-lang/cargo/issues/12670
@vinniejames do you have a stack trace you can paste here?
Interesting indeed, because the previous release build seemed to work properly apparently it has something to do with proxy retrieval while initing reqwest client as previously mentioned.
0 CoreFoundation 0x00000001836d48c0 __exceptionPreprocess + 176
1 libobjc.A.dylib 0x00000001831cdeb4 objc_exception_throw + 60
2 CoreFoundation 0x00000001837863dc -[NSObject(NSObject) __retain_OA] + 0
3 CoreFoundation 0x000000018363e9e4 ___forwarding___ + 1572
4 CoreFoundation 0x000000018363e300 _CF_forwarding_prep_0 + 96
5 Foundation 0x00000001847134f0 -[NSProcessInfo arguments] + 188
6 CoreFoundation 0x000000018374fff8 __getDefaultArguments_block_invoke + 96
7 libdispatch.dylib 0x00000001833d5910 _dispatch_client_callout + 20
8 libdispatch.dylib 0x00000001833d714c _dispatch_once_callout + 32
9 CoreFoundation 0x000000018374f994 _addBackstopValuesForIdentifierAndSource + 640
10 CoreFoundation 0x000000018360a254 __81-[_CFXPreferences(SourceAdditions) withNamedVolatileSourceForIdentifier:perform:]_block_invoke + 144
11 CoreFoundation 0x000000018374f63c -[_CFXPreferences withNamedVolatileSourceForIdentifier:perform:] + 272
12 CoreFoundation 0x00000001836105c4 -[CFPrefsSearchListSource addNamedVolatileSourceForIdentifier:] + 136
13 CoreFoundation 0x000000018378e8bc __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke.155 + 296
14 CoreFoundation 0x000000018378e564 -[_CFXPreferences withSearchLists:] + 84
15 CoreFoundation 0x000000018360bb18 __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 300
16 CoreFoundation 0x000000018378e710 -[_CFXPreferences withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 384
17 CoreFoundation 0x000000018360b440 -[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 156
18 CoreFoundation 0x000000018360b368 _CFPreferencesCopyAppValueWithContainerAndConfiguration + 112
19 SystemConfiguration 0x000000018439b478 SCDynamicStoreCopyProxiesWithOptions + 180
20 legba 0x0000000104a34b98 _ZN20system_configuration13dynamic_store14SCDynamicStore11get_proxies17h949859e68558a1d1E + 16
21 legba 0x00000001049f326c _ZN4core3ops8function6FnOnce9call_once17h02c254c325789900E + 104
22 legba 0x00000001049cb354 _ZN9once_cell3imp17OnceCell$LT$T$GT$10initialize28_$u7b$$u7b$closure$u7d$$u7d$17h603154414031fffcE.llvm.14320163142611912427 + 48
23 legba 0x0000000104b342f4 _ZN9once_cell3imp18initialize_or_wait17hd4f69634fde70e00E + 328
24 legba 0x0000000104cc79a0 _ZN9once_cell3imp17OnceCell$LT$T$GT$10initialize17h20152ddacc86502fE + 56
25 legba 0x0000000104a01760 _ZN7reqwest10async_impl6client13ClientBuilder5build17h2472c52275236442E + 468
26 legba 0x0000000104a02980 _ZN78_$LT$reqwest..async_impl..client..Client$u20$as$u20$core..default..Default$GT$7default17hc16c7a6fc319903dE + 32
27 legba 0x00000001041b74b8 _ZN5legba7plugins4http4HTTP3new17h1907b94a3438d2b2E + 32
28 legba 0x00000001041bd278 _ZN5legba7plugins4http27register___rust_ctor___ctor27register___rust_ctor___ctor17h08ac0017f612bb74E + 32
29 dyld 0x00000001832255c8 ___ZZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateEENK3$_0clEv_block_invoke + 168
30 dyld 0x000000018326a920 ___ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv_block_invoke.209 + 340
31 dyld 0x000000018325dc60 ___ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE_block_invoke + 496
32 dyld 0x00000001832052fc _ZNK5dyld39MachOFile18forEachLoadCommandER11DiagnosticsU13block_pointerFvPK12load_commandRbE + 300
33 dyld 0x000000018325cc98 _ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE + 192
34 dyld 0x000000018326a434 _ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv + 516
35 dyld 0x0000000183221798 _ZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateE + 448
36 dyld 0x0000000183227b14 _ZNK5dyld416JustInTimeLoader15runInitializersERNS_12RuntimeStateE + 36
37 dyld 0x0000000183221b4c _ZNK5dyld46Loader23runInitializersBottomUpERNS_12RuntimeStateERN5dyld35ArrayIPKS0_EE + 220
38 dyld 0x0000000183225654 _ZZNK5dyld46Loader38runInitializersBottomUpPlusUpwardLinksERNS_12RuntimeStateEENK3$_1clEv + 112
39 dyld 0x0000000183221ccc _ZNK5dyld46Loader38runInitializersBottomUpPlusUpwardLinksERNS_12RuntimeStateE + 304
40 dyld 0x0000000183246ad4 _ZN5dyld44APIs25runAllInitializersForMainEv + 464
41 dyld 0x0000000183209f34 _ZN5dyld4L7prepareERNS_4APIsEPKN5dyld313MachOAnalyzerE + 3192
42 dyld 0x0000000183208f44 start + 1948
)
libc++abi: terminating due to uncaught exception of type NSException
until @evilsocket or cargo come up with a proper solution for everyone facing the same issue, here is a quick workaround: Install the latest cURL via Homebrew and then run one of the following commands.
DYLD_INSERT_LIBRARIES=/opt/homebrew/opt/curl/lib/libcurl.dylib ./legba
or just
export DYLD_INSERT_LIBRARIES=/opt/homebrew/opt/curl/lib/libcurl.dylib
./legba
thanks @0xsha, by previous release build you mean macOS, reqwest, curl or legba?
@evilsocket Legba, the last time I built it was a few days ago on a M1 Mac machine. The release version works without the need for inserting DYLD, and it was on the same macOS Sonoma.
default mac cURL version
curl 8.1.2 (x86_64-apple-darwin23.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1
Release-Date: 2023-05-30
Homebrew cURL version
curl 8.4.0 (aarch64-apple-darwin23.0.0) libcurl/8.4.0 (SecureTransport) OpenSSL/3.1.4 zlib/1.2.12 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libssh2/1.11.0 nghttp2/1.58.0 librtmp/2.3 OpenLDAP/2.6.6
Release-Date: 2023-10-11
current reqwest is matched with main branch
reqwest = { version = "0.11.20"
Weird, as far as I can remember reqwest version was not changed https://github.com/evilsocket/legba/blame/main/Cargo.toml#L36
It is indeed strange, but as is it certainly affects the default release build on M1 if there's a plan to distribute a binary release of Legba through Homebrew and other binary distribution channels. probably following up cargo issues might give more clues.
will do my best to understand this better, your feedback has been extremely valuable @0xsha, thank you so much
okay had some time to actually debug it.
the actual fix is here : https://github.com/curl/curl/commit/6ab7e19
for now a better workaround than
DYLD_INSERT_LIBRARIES=/opt/homebrew/opt/curl/lib/libcurl.dylib ./legba
is to first install latest curl with homebrew
brew install curl
add a build.rs
file in root directory.
use std::env;
use std::path::PathBuf;
fn main() {
if env::var("CARGO_CFG_TARGET_OS").unwrap() == "macos"
// && env::var("CARGO_CFG_TARGET_ARCH").unwrap() == "aarch64"
{
let lib_dir = PathBuf::from(env::var("HOMEBREW_PREFIX").unwrap_or("/opt/homebrew".into()))
.join("lib");
let include_dir =
PathBuf::from(env::var("HOMEBREW_PREFIX").unwrap_or("/opt/homebrew".into()))
.join("include");
// Tell cargo to link against libcurl from Homebrew
println!("cargo:rustc-link-search=native={}", lib_dir.display());
println!("cargo:rustc-link-lib=curl");
println!("cargo:include={}", include_dir.display());
}
}
now just build a release as you normally do
cargo build --release
and everything should work as it was !
➜ legba git:(main) ✗ ./target/release/legba --version
legba v0.3.0
I haven't created a pull request yet because I'm waiting to see if the issue can be resolved with the new Cargo update or if @evilsocket has a better idea regarding the build.rs file. However, this is certainly the appropriate solution for distribution if needed.
Enjoy !
@0xsha I just pushed a simpler fix that uses @p1tsi workaround since we don't need to detect the system proxy. But since I'm unable to reproduce the bug locally, I'll need your help. Can you please clone from master and check if the issue is gone?
@evilsocket didn't fix it and it still crashes, as that workaround doesn't seem to be a workaround. Read my last workaround; the core issue is linked to the version of cURL
it was worth trying :/
@0xsha pushed your build.rs, thanks ... honestly I feel a bit dumb because I don't understand at which point curl is involved.
Reqwest doesn't seem to link/reference it anywhere:
says he feels dumb, one of the best infosec tool developers alive! The proposed build.rs
workaround is mainly based on observations of DYLD_INSERT_LIBRARIES
via recent libcurl (hence didn't made a PR), which practically fixes the crash. Furthermore, the crash stack provides some hints about CoreServices, which were precisely addressed in cURL to make it work on MacOS 14, apparently. I tested build.rs
on M1 and M2 machines (which links the lib to the final rustc command), and the release doesn't crash with it. But you are right; Reqwest uses Hyper as its underlying library, which doesn't use curl. However , A more accurate observation might be the direct/indirect dependencies of the entire cargo tree.
therefore, The real root cause still seems a bit of a mystery.
lol i feel dumb more often than you'd imagine XD ... i'll try to dig into it, maybe i can identify what the hell is using libcurl and why
I upgraded to Sonoma and finally was able to reproduce on M1. This is the output of bt from an lldb session:
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
* frame #0: 0x0000000180d4911c libsystem_kernel.dylib`__pthread_kill + 8
frame #1: 0x0000000180d80cc0 libsystem_pthread.dylib`pthread_kill + 288
frame #2: 0x0000000180c90a40 libsystem_c.dylib`abort + 180
frame #3: 0x0000000180d386d8 libc++abi.dylib`abort_message + 132
frame #4: 0x0000000180d287c8 libc++abi.dylib`demangling_terminate_handler() + 348
frame #5: 0x00000001809d3894 libobjc.A.dylib`_objc_terminate() + 144
frame #6: 0x0000000180d37a9c libc++abi.dylib`std::__terminate(void (*)()) + 16
frame #7: 0x0000000180d37a40 libc++abi.dylib`std::terminate() + 56
frame #8: 0x00000001809ed608 libobjc.A.dylib`objc_terminate + 16
frame #9: 0x0000000180bd3924 libdispatch.dylib`_dispatch_client_callout + 40
frame #10: 0x0000000180bd514c libdispatch.dylib`_dispatch_once_callout + 32
frame #11: 0x0000000180f4d974 CoreFoundation`_addBackstopValuesForIdentifierAndSource + 640
frame #12: 0x0000000180e082ac CoreFoundation`__81-[_CFXPreferences(SourceAdditions) withNamedVolatileSourceForIdentifier:perform:]_block_invoke + 144
frame #13: 0x0000000180f4d61c CoreFoundation`-[_CFXPreferences withNamedVolatileSourceForIdentifier:perform:] + 272
frame #14: 0x0000000180e0e61c CoreFoundation`-[CFPrefsSearchListSource addNamedVolatileSourceForIdentifier:] + 136
frame #15: 0x0000000180f8c89c CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke.155 + 296
frame #16: 0x0000000180f8c544 CoreFoundation`-[_CFXPreferences withSearchLists:] + 84
frame #17: 0x0000000180e09b70 CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 300
frame #18: 0x0000000180f8c6f0 CoreFoundation`-[_CFXPreferences withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 384
frame #19: 0x0000000180e09498 CoreFoundation`-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 156
frame #20: 0x0000000180e093c0 CoreFoundation`_CFPreferencesCopyAppValueWithContainerAndConfiguration + 112
frame #21: 0x0000000181b9a478 SystemConfiguration`SCDynamicStoreCopyProxiesWithOptions + 180
frame #22: 0x000000010035a398 legba`system_configuration::dynamic_store::SCDynamicStore::get_proxies::hc7b498b75a64e921(self=0x000000016fdfaa00) at dynamic_store.rs:200:34
frame #23: 0x00000001001e2654 legba`reqwest::proxy::get_from_platform_impl::hdd39565c15c5f7e1 at proxy.rs:946:29
frame #24: 0x00000001001e2a54 legba`reqwest::proxy::get_from_platform::h3a5d242957856cd3 at proxy.rs:973:5
frame #25: 0x00000001001e152c legba`reqwest::proxy::SYS_PROXIES::_$u7b$$u7b$closure$u7d$$u7d$::h0ca7225194f506ed((null)=0x000000016fdfacfe) at proxy.rs:806:43
frame #26: 0x00000001001bcce8 legba`core::ops::function::FnOnce::call_once::h1ea6ef4c42325291((null)={closure_env#0} @ 0x000000016fdfacfe, (null)=<unavailable>) at function.rs:250:5
frame #27: 0x00000001001bd64c legba`core::ops::function::FnOnce::call_once::h5aa13aaf4dbdada7((null)=(legba`core::ops::function::FnOnce::call_once::h1ea6ef4c42325291 at function.rs:250), (null)=<unavailable>) at function.rs:250:5
frame #28: 0x000000010031651c legba`once_cell::sync::Lazy$LT$T$C$F$GT$::force::_$u7b$$u7b$closure$u7d$$u7d$::h220e01153b6cd751 at lib.rs:1312:28
frame #29: 0x0000000100316590 legba`once_cell::sync::OnceCell$LT$T$GT$::get_or_init::_$u7b$$u7b$closure$u7d$$u7d$::h703176d357cd95d6 at lib.rs:1122:57
frame #30: 0x0000000100242bd8 legba`once_cell::imp::OnceCell$LT$T$GT$::initialize::_$u7b$$u7b$closure$u7d$$u7d$::h0737e340da671ee6 at imp_std.rs:72:23
frame #31: 0x00000001005396d0 legba`core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$mut$u20$F$GT$::call_mut::h327fdb529802aa76(self=0x000000016fdfae90, args=<unavailable>) at function.rs:294:13
frame #32: 0x000000010053b004 legba`once_cell::imp::initialize_or_wait::h7723628e960a4eb7(queue=0x0000000100dd0068, init=Option<&mut dyn core::ops::function::FnMut<(), Output=bool>> @ 0x000000016fdfae90) at imp_std.rs:196:20
frame #33: 0x0000000100926338 legba`once_cell::imp::OnceCell$LT$T$GT$::initialize::h29138a46d46cc4c1(self=0x0000000100dd0068, f={closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, once_cell::sync::{impl#11}::force::{closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, fn() -> alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>>> @ 0x000000016fdfaf90) at imp_std.rs:68:9
frame #34: 0x000000010031666c legba`once_cell::sync::OnceCell$LT$T$GT$::get_or_try_init::hfb69e8abb8b774dd(self=0x0000000100dd0068, f={closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, once_cell::sync::{impl#11}::force::{closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, fn() -> alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>>> @ 0x000000016fdfb018) at lib.rs:1163:13
frame #35: 0x0000000100316550 legba`once_cell::sync::OnceCell$LT$T$GT$::get_or_init::h8dc1d521d31532de(self=0x0000000100dd0068, f={closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, fn() -> alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>> @ 0x000000016fdfb070) at lib.rs:1122:19
frame #36: 0x0000000100316490 legba`once_cell::sync::Lazy$LT$T$C$F$GT$::force::h5613712f0a232921(this=0x0000000100dd0068) at lib.rs:1311:13
frame #37: 0x0000000100314ae4 legba`_$LT$once_cell..sync..Lazy$LT$T$C$F$GT$$u20$as$u20$core..ops..deref..Deref$GT$::deref::hdcccd425ad3f6d81(self=0x0000000100dd0068) at lib.rs:1378:13
frame #38: 0x00000001001de0fc legba`reqwest::proxy::Proxy::system::h828757717302a160 at proxy.rs:290:42
frame #39: 0x00000001002e49e4 legba`reqwest::async_impl::client::ClientBuilder::build::h644b7a1f4a102dc8(self=<unavailable>) at client.rs:256:26
frame #40: 0x00000001002e5e20 legba`reqwest::async_impl::client::Client::new::h12013370f53a9976 at client.rs:1667:9
frame #41: 0x00000001002e5df4 legba`_$LT$reqwest..async_impl..client..Client$u20$as$u20$core..default..Default$GT$::default::hc6cf3cede11e1d25 at client.rs:1652:9
frame #42: 0x00000001000a6e5c legba`legba::plugins::http::HTTP::new::h6a2f806d772c6630(strategy=Request) at mod.rs:87:21
frame #43: 0x00000001000a8568 legba`legba::plugins::http::register::h474a7b89cf719366 at mod.rs:26:56
frame #44: 0x00000001000a88c0 legba`legba::plugins::http::register___rust_ctor___ctor::register___rust_ctor___ctor::h6db0dcb3b74de01a at mod.rs:24:1
frame #45: 0x0000000180a21a24 dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
frame #46: 0x0000000180a67328 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340
frame #47: 0x0000000180a5a668 dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 496
frame #48: 0x0000000180a012fc dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 300
frame #49: 0x0000000180a596a0 dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
frame #50: 0x0000000180a66e3c dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
frame #51: 0x0000000180a1db38 dyld`dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 524
frame #52: 0x0000000180a23f70 dyld`dyld4::JustInTimeLoader::runInitializers(dyld4::RuntimeState&) const + 36
frame #53: 0x0000000180a1df24 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
frame #54: 0x0000000180a21ab0 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
frame #55: 0x0000000180a1e0f0 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 380
frame #56: 0x0000000180a434dc dyld`dyld4::APIs::runAllInitializersForMain() + 464
frame #57: 0x0000000180a05fa0 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3192
frame #58: 0x0000000180a04edc dyld`start + 1844
The issue is caused by this call https://github.com/seanmonstar/reqwest/blob/50dbaf391087cfa951accc765126b4f5d017d8a3/src/proxy.rs#L946 which references the system_configuration crate which ultimately ends up calling this https://github.com/mullvad/system-configuration-rs/blob/main/system-configuration-sys/src/dynamic_store_copy_specific.rs#L35
Looks like libcurl is somehow invoked by Sonoma to return the current system proxy settings from SCDynamicStoreCopyProxies ... I wonder how this happens since there're no direct libcurl invokations ... anyways, I've pushed a fix that doesn't require us to link explicitly to libcurl ... @p1tsi was correct, I just forgot to add the fix to the constructor of the HTTP plugin, that called a reqwest::Client::new() :D
thanks all for the help!
Installed on an M2 with
cargo
, every command fails with the same error: