NSInvalidArgumentException with Cargo package on macOS Sonoma

[vinniejames]

Installed on an M2 with cargo, every command fails with the same error:

legba --version
*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFString stringByStandardizingPath]: unrecognized selector sent to instance 0x600002068140'
...

[p1tsi]

I ran in the same exception with an old Intel-based Mac. As a quick workaround, you could set to "false" the "auto_sys_proxy" property of ClientBuilder inside reqwest::Client.

As far as I understood, the problem is in retrieving system proxy information when calling Client::default()

[evilsocket]

Mmm interesting, this seems to be a Cargo issue:

https://github.com/rust-lang/cargo/issues/12784 https://github.com/rust-lang/cargo/issues/12670

@vinniejames do you have a stack trace you can paste here?

[0xsha]

Interesting indeed, because the previous release build seemed to work properly apparently it has something to do with proxy retrieval while initing reqwest client as previously mentioned.

        0   CoreFoundation                      0x00000001836d48c0 __exceptionPreprocess + 176
        1   libobjc.A.dylib                     0x00000001831cdeb4 objc_exception_throw + 60
        2   CoreFoundation                      0x00000001837863dc -[NSObject(NSObject) __retain_OA] + 0
        3   CoreFoundation                      0x000000018363e9e4 ___forwarding___ + 1572
        4   CoreFoundation                      0x000000018363e300 _CF_forwarding_prep_0 + 96
        5   Foundation                          0x00000001847134f0 -[NSProcessInfo arguments] + 188
        6   CoreFoundation                      0x000000018374fff8 __getDefaultArguments_block_invoke + 96
        7   libdispatch.dylib                   0x00000001833d5910 _dispatch_client_callout + 20
        8   libdispatch.dylib                   0x00000001833d714c _dispatch_once_callout + 32
        9   CoreFoundation                      0x000000018374f994 _addBackstopValuesForIdentifierAndSource + 640
        10  CoreFoundation                      0x000000018360a254 __81-[_CFXPreferences(SourceAdditions) withNamedVolatileSourceForIdentifier:perform:]_block_invoke + 144
        11  CoreFoundation                      0x000000018374f63c -[_CFXPreferences withNamedVolatileSourceForIdentifier:perform:] + 272
        12  CoreFoundation                      0x00000001836105c4 -[CFPrefsSearchListSource addNamedVolatileSourceForIdentifier:] + 136
        13  CoreFoundation                      0x000000018378e8bc __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke.155 + 296
        14  CoreFoundation                      0x000000018378e564 -[_CFXPreferences withSearchLists:] + 84
        15  CoreFoundation                      0x000000018360bb18 __108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 300
        16  CoreFoundation                      0x000000018378e710 -[_CFXPreferences withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 384
        17  CoreFoundation                      0x000000018360b440 -[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 156
        18  CoreFoundation                      0x000000018360b368 _CFPreferencesCopyAppValueWithContainerAndConfiguration + 112
        19  SystemConfiguration                 0x000000018439b478 SCDynamicStoreCopyProxiesWithOptions + 180
        20  legba                               0x0000000104a34b98 _ZN20system_configuration13dynamic_store14SCDynamicStore11get_proxies17h949859e68558a1d1E + 16
        21  legba                               0x00000001049f326c _ZN4core3ops8function6FnOnce9call_once17h02c254c325789900E + 104
        22  legba                               0x00000001049cb354 _ZN9once_cell3imp17OnceCell$LT$T$GT$10initialize28_$u7b$$u7b$closure$u7d$$u7d$17h603154414031fffcE.llvm.14320163142611912427 + 48
        23  legba                               0x0000000104b342f4 _ZN9once_cell3imp18initialize_or_wait17hd4f69634fde70e00E + 328
        24  legba                               0x0000000104cc79a0 _ZN9once_cell3imp17OnceCell$LT$T$GT$10initialize17h20152ddacc86502fE + 56
        25  legba                               0x0000000104a01760 _ZN7reqwest10async_impl6client13ClientBuilder5build17h2472c52275236442E + 468
        26  legba                               0x0000000104a02980 _ZN78_$LT$reqwest..async_impl..client..Client$u20$as$u20$core..default..Default$GT$7default17hc16c7a6fc319903dE + 32
        27  legba                               0x00000001041b74b8 _ZN5legba7plugins4http4HTTP3new17h1907b94a3438d2b2E + 32
        28  legba                               0x00000001041bd278 _ZN5legba7plugins4http27register___rust_ctor___ctor27register___rust_ctor___ctor17h08ac0017f612bb74E + 32
        29  dyld                                0x00000001832255c8 ___ZZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateEENK3$_0clEv_block_invoke + 168
        30  dyld                                0x000000018326a920 ___ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv_block_invoke.209 + 340
        31  dyld                                0x000000018325dc60 ___ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE_block_invoke + 496
        32  dyld                                0x00000001832052fc _ZNK5dyld39MachOFile18forEachLoadCommandER11DiagnosticsU13block_pointerFvPK12load_commandRbE + 300
        33  dyld                                0x000000018325cc98 _ZNK5dyld39MachOFile14forEachSectionEU13block_pointerFvRKNS0_11SectionInfoEbRbE + 192
        34  dyld                                0x000000018326a434 _ZNK5dyld313MachOAnalyzer18forEachInitializerER11DiagnosticsRKNS0_15VMAddrConverterEU13block_pointerFvjEPKv + 516
        35  dyld                                0x0000000183221798 _ZNK5dyld46Loader25findAndRunAllInitializersERNS_12RuntimeStateE + 448
        36  dyld                                0x0000000183227b14 _ZNK5dyld416JustInTimeLoader15runInitializersERNS_12RuntimeStateE + 36
        37  dyld                                0x0000000183221b4c _ZNK5dyld46Loader23runInitializersBottomUpERNS_12RuntimeStateERN5dyld35ArrayIPKS0_EE + 220
        38  dyld                                0x0000000183225654 _ZZNK5dyld46Loader38runInitializersBottomUpPlusUpwardLinksERNS_12RuntimeStateEENK3$_1clEv + 112
        39  dyld                                0x0000000183221ccc _ZNK5dyld46Loader38runInitializersBottomUpPlusUpwardLinksERNS_12RuntimeStateE + 304
        40  dyld                                0x0000000183246ad4 _ZN5dyld44APIs25runAllInitializersForMainEv + 464
        41  dyld                                0x0000000183209f34 _ZN5dyld4L7prepareERNS_4APIsEPKN5dyld313MachOAnalyzerE + 3192
        42  dyld                                0x0000000183208f44 start + 1948
)
libc++abi: terminating due to uncaught exception of type NSException

until @evilsocket or cargo come up with a proper solution for everyone facing the same issue, here is a quick workaround: Install the latest cURL via Homebrew and then run one of the following commands.

DYLD_INSERT_LIBRARIES=/opt/homebrew/opt/curl/lib/libcurl.dylib ./legba

or just

export DYLD_INSERT_LIBRARIES=/opt/homebrew/opt/curl/lib/libcurl.dylib

./legba

[evilsocket]

thanks @0xsha, by previous release build you mean macOS, reqwest, curl or legba?

[0xsha]

@evilsocket Legba, the last time I built it was a few days ago on a M1 Mac machine. The release version works without the need for inserting DYLD, and it was on the same macOS Sonoma.

default mac cURL version

curl 8.1.2 (x86_64-apple-darwin23.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.55.1
Release-Date: 2023-05-30

Homebrew cURL version

curl 8.4.0 (aarch64-apple-darwin23.0.0) libcurl/8.4.0 (SecureTransport) OpenSSL/3.1.4 zlib/1.2.12 brotli/1.1.0 zstd/1.5.5 libidn2/2.3.4 libssh2/1.11.0 nghttp2/1.58.0 librtmp/2.3 OpenLDAP/2.6.6
Release-Date: 2023-10-11

current reqwest is matched with main branch

reqwest = { version = "0.11.20"

[evilsocket]

Weird, as far as I can remember reqwest version was not changed https://github.com/evilsocket/legba/blame/main/Cargo.toml#L36

[0xsha]

It is indeed strange, but as is it certainly affects the default release build on M1 if there's a plan to distribute a binary release of Legba through Homebrew and other binary distribution channels. probably following up cargo issues might give more clues.

[evilsocket]

will do my best to understand this better, your feedback has been extremely valuable @0xsha, thank you so much

[0xsha]

okay had some time to actually debug it.

the actual fix is here : https://github.com/curl/curl/commit/6ab7e19

for now a better workaround than

DYLD_INSERT_LIBRARIES=/opt/homebrew/opt/curl/lib/libcurl.dylib ./legba

is to first install latest curl with homebrew

brew install curl 

add a build.rs file in root directory.

use std::env;
use std::path::PathBuf;

fn main() {
    if env::var("CARGO_CFG_TARGET_OS").unwrap() == "macos"
    // && env::var("CARGO_CFG_TARGET_ARCH").unwrap() == "aarch64"
    {
        let lib_dir = PathBuf::from(env::var("HOMEBREW_PREFIX").unwrap_or("/opt/homebrew".into()))
            .join("lib");
        let include_dir =
            PathBuf::from(env::var("HOMEBREW_PREFIX").unwrap_or("/opt/homebrew".into()))
                .join("include");

        // Tell cargo to link against libcurl from Homebrew
        println!("cargo:rustc-link-search=native={}", lib_dir.display());
        println!("cargo:rustc-link-lib=curl");
        println!("cargo:include={}", include_dir.display());
    }
}

now just build a release as you normally do

cargo build --release 

and everything should work as it was !

➜  legba git:(main) ✗ ./target/release/legba --version
legba v0.3.0

I haven't created a pull request yet because I'm waiting to see if the issue can be resolved with the new Cargo update or if @evilsocket has a better idea regarding the build.rs file. However, this is certainly the appropriate solution for distribution if needed.

Enjoy !

[evilsocket]

@0xsha I just pushed a simpler fix that uses @p1tsi workaround since we don't need to detect the system proxy. But since I'm unable to reproduce the bug locally, I'll need your help. Can you please clone from master and check if the issue is gone?

[0xsha]

@evilsocket didn't fix it and it still crashes, as that workaround doesn't seem to be a workaround. Read my last workaround; the core issue is linked to the version of cURL

[evilsocket]

it was worth trying :/

[evilsocket]

@0xsha pushed your build.rs, thanks ... honestly I feel a bit dumb because I don't understand at which point curl is involved.

Reqwest doesn't seem to link/reference it anywhere:

[0xsha]

says he feels dumb, one of the best infosec tool developers alive! The proposed build.rs workaround is mainly based on observations of DYLD_INSERT_LIBRARIES via recent libcurl (hence didn't made a PR), which practically fixes the crash. Furthermore, the crash stack provides some hints about CoreServices, which were precisely addressed in cURL to make it work on MacOS 14, apparently. I tested build.rs on M1 and M2 machines (which links the lib to the final rustc command), and the release doesn't crash with it. But you are right; Reqwest uses Hyper as its underlying library, which doesn't use curl. However , A more accurate observation might be the direct/indirect dependencies of the entire cargo tree. therefore, The real root cause still seems a bit of a mystery.

[evilsocket]

lol i feel dumb more often than you'd imagine XD ... i'll try to dig into it, maybe i can identify what the hell is using libcurl and why

[evilsocket]

I upgraded to Sonoma and finally was able to reproduce on M1. This is the output of bt from an lldb session:

* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x0000000180d4911c libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000180d80cc0 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x0000000180c90a40 libsystem_c.dylib`abort + 180
    frame #3: 0x0000000180d386d8 libc++abi.dylib`abort_message + 132
    frame #4: 0x0000000180d287c8 libc++abi.dylib`demangling_terminate_handler() + 348
    frame #5: 0x00000001809d3894 libobjc.A.dylib`_objc_terminate() + 144
    frame #6: 0x0000000180d37a9c libc++abi.dylib`std::__terminate(void (*)()) + 16
    frame #7: 0x0000000180d37a40 libc++abi.dylib`std::terminate() + 56
    frame #8: 0x00000001809ed608 libobjc.A.dylib`objc_terminate + 16
    frame #9: 0x0000000180bd3924 libdispatch.dylib`_dispatch_client_callout + 40
    frame #10: 0x0000000180bd514c libdispatch.dylib`_dispatch_once_callout + 32

    frame #11: 0x0000000180f4d974 CoreFoundation`_addBackstopValuesForIdentifierAndSource + 640
    frame #12: 0x0000000180e082ac CoreFoundation`__81-[_CFXPreferences(SourceAdditions) withNamedVolatileSourceForIdentifier:perform:]_block_invoke + 144
    frame #13: 0x0000000180f4d61c CoreFoundation`-[_CFXPreferences withNamedVolatileSourceForIdentifier:perform:] + 272
    frame #14: 0x0000000180e0e61c CoreFoundation`-[CFPrefsSearchListSource addNamedVolatileSourceForIdentifier:] + 136
    frame #15: 0x0000000180f8c89c CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke.155 + 296
    frame #16: 0x0000000180f8c544 CoreFoundation`-[_CFXPreferences withSearchLists:] + 84
    frame #17: 0x0000000180e09b70 CoreFoundation`__108-[_CFXPreferences(SearchListAdditions) withSearchListForIdentifier:container:cloudConfigurationURL:perform:]_block_invoke + 300
    frame #18: 0x0000000180f8c6f0 CoreFoundation`-[_CFXPreferences withSearchListForIdentifier:container:cloudConfigurationURL:perform:] + 384
    frame #19: 0x0000000180e09498 CoreFoundation`-[_CFXPreferences copyAppValueForKey:identifier:container:configurationURL:] + 156
    frame #20: 0x0000000180e093c0 CoreFoundation`_CFPreferencesCopyAppValueWithContainerAndConfiguration + 112

    frame #21: 0x0000000181b9a478 SystemConfiguration`SCDynamicStoreCopyProxiesWithOptions + 180

    frame #22: 0x000000010035a398 legba`system_configuration::dynamic_store::SCDynamicStore::get_proxies::hc7b498b75a64e921(self=0x000000016fdfaa00) at dynamic_store.rs:200:34
    frame #23: 0x00000001001e2654 legba`reqwest::proxy::get_from_platform_impl::hdd39565c15c5f7e1 at proxy.rs:946:29
    frame #24: 0x00000001001e2a54 legba`reqwest::proxy::get_from_platform::h3a5d242957856cd3 at proxy.rs:973:5
    frame #25: 0x00000001001e152c legba`reqwest::proxy::SYS_PROXIES::_$u7b$$u7b$closure$u7d$$u7d$::h0ca7225194f506ed((null)=0x000000016fdfacfe) at proxy.rs:806:43
    frame #26: 0x00000001001bcce8 legba`core::ops::function::FnOnce::call_once::h1ea6ef4c42325291((null)={closure_env#0} @ 0x000000016fdfacfe, (null)=<unavailable>) at function.rs:250:5
    frame #27: 0x00000001001bd64c legba`core::ops::function::FnOnce::call_once::h5aa13aaf4dbdada7((null)=(legba`core::ops::function::FnOnce::call_once::h1ea6ef4c42325291 at function.rs:250), (null)=<unavailable>) at function.rs:250:5
    frame #28: 0x000000010031651c legba`once_cell::sync::Lazy$LT$T$C$F$GT$::force::_$u7b$$u7b$closure$u7d$$u7d$::h220e01153b6cd751 at lib.rs:1312:28
    frame #29: 0x0000000100316590 legba`once_cell::sync::OnceCell$LT$T$GT$::get_or_init::_$u7b$$u7b$closure$u7d$$u7d$::h703176d357cd95d6 at lib.rs:1122:57
    frame #30: 0x0000000100242bd8 legba`once_cell::imp::OnceCell$LT$T$GT$::initialize::_$u7b$$u7b$closure$u7d$$u7d$::h0737e340da671ee6 at imp_std.rs:72:23
    frame #31: 0x00000001005396d0 legba`core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$mut$u20$F$GT$::call_mut::h327fdb529802aa76(self=0x000000016fdfae90, args=<unavailable>) at function.rs:294:13
    frame #32: 0x000000010053b004 legba`once_cell::imp::initialize_or_wait::h7723628e960a4eb7(queue=0x0000000100dd0068, init=Option<&mut dyn core::ops::function::FnMut<(), Output=bool>> @ 0x000000016fdfae90) at imp_std.rs:196:20
    frame #33: 0x0000000100926338 legba`once_cell::imp::OnceCell$LT$T$GT$::initialize::h29138a46d46cc4c1(self=0x0000000100dd0068, f={closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, once_cell::sync::{impl#11}::force::{closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, fn() -> alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>>> @ 0x000000016fdfaf90) at imp_std.rs:68:9
    frame #34: 0x000000010031666c legba`once_cell::sync::OnceCell$LT$T$GT$::get_or_try_init::hfb69e8abb8b774dd(self=0x0000000100dd0068, f={closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, once_cell::sync::{impl#11}::force::{closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, fn() -> alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>>> @ 0x000000016fdfb018) at lib.rs:1163:13
    frame #35: 0x0000000100316550 legba`once_cell::sync::OnceCell$LT$T$GT$::get_or_init::h8dc1d521d31532de(self=0x0000000100dd0068, f={closure_env#0}<alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>, fn() -> alloc::sync::Arc<std::collections::hash::map::HashMap<alloc::string::String, reqwest::proxy::ProxyScheme, std::collections::hash::map::RandomState>>> @ 0x000000016fdfb070) at lib.rs:1122:19
    frame #36: 0x0000000100316490 legba`once_cell::sync::Lazy$LT$T$C$F$GT$::force::h5613712f0a232921(this=0x0000000100dd0068) at lib.rs:1311:13
    frame #37: 0x0000000100314ae4 legba`_$LT$once_cell..sync..Lazy$LT$T$C$F$GT$$u20$as$u20$core..ops..deref..Deref$GT$::deref::hdcccd425ad3f6d81(self=0x0000000100dd0068) at lib.rs:1378:13
    frame #38: 0x00000001001de0fc legba`reqwest::proxy::Proxy::system::h828757717302a160 at proxy.rs:290:42
    frame #39: 0x00000001002e49e4 legba`reqwest::async_impl::client::ClientBuilder::build::h644b7a1f4a102dc8(self=<unavailable>) at client.rs:256:26
    frame #40: 0x00000001002e5e20 legba`reqwest::async_impl::client::Client::new::h12013370f53a9976 at client.rs:1667:9
    frame #41: 0x00000001002e5df4 legba`_$LT$reqwest..async_impl..client..Client$u20$as$u20$core..default..Default$GT$::default::hc6cf3cede11e1d25 at client.rs:1652:9
    frame #42: 0x00000001000a6e5c legba`legba::plugins::http::HTTP::new::h6a2f806d772c6630(strategy=Request) at mod.rs:87:21
    frame #43: 0x00000001000a8568 legba`legba::plugins::http::register::h474a7b89cf719366 at mod.rs:26:56
    frame #44: 0x00000001000a88c0 legba`legba::plugins::http::register___rust_ctor___ctor::register___rust_ctor___ctor::h6db0dcb3b74de01a at mod.rs:24:1

    frame #45: 0x0000000180a21a24 dyld`invocation function for block in dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const::$_0::operator()() const + 168
    frame #46: 0x0000000180a67328 dyld`invocation function for block in dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 340
    frame #47: 0x0000000180a5a668 dyld`invocation function for block in dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 496
    frame #48: 0x0000000180a012fc dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 300
    frame #49: 0x0000000180a596a0 dyld`dyld3::MachOFile::forEachSection(void (dyld3::MachOFile::SectionInfo const&, bool, bool&) block_pointer) const + 192
    frame #50: 0x0000000180a66e3c dyld`dyld3::MachOAnalyzer::forEachInitializer(Diagnostics&, dyld3::MachOAnalyzer::VMAddrConverter const&, void (unsigned int) block_pointer, void const*) const + 516
    frame #51: 0x0000000180a1db38 dyld`dyld4::Loader::findAndRunAllInitializers(dyld4::RuntimeState&) const + 524
    frame #52: 0x0000000180a23f70 dyld`dyld4::JustInTimeLoader::runInitializers(dyld4::RuntimeState&) const + 36
    frame #53: 0x0000000180a1df24 dyld`dyld4::Loader::runInitializersBottomUp(dyld4::RuntimeState&, dyld3::Array<dyld4::Loader const*>&) const + 220
    frame #54: 0x0000000180a21ab0 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const::$_1::operator()() const + 112
    frame #55: 0x0000000180a1e0f0 dyld`dyld4::Loader::runInitializersBottomUpPlusUpwardLinks(dyld4::RuntimeState&) const + 380
    frame #56: 0x0000000180a434dc dyld`dyld4::APIs::runAllInitializersForMain() + 464
    frame #57: 0x0000000180a05fa0 dyld`dyld4::prepare(dyld4::APIs&, dyld3::MachOAnalyzer const*) + 3192
    frame #58: 0x0000000180a04edc dyld`start + 1844

The issue is caused by this call https://github.com/seanmonstar/reqwest/blob/50dbaf391087cfa951accc765126b4f5d017d8a3/src/proxy.rs#L946 which references the system_configuration crate which ultimately ends up calling this https://github.com/mullvad/system-configuration-rs/blob/main/system-configuration-sys/src/dynamic_store_copy_specific.rs#L35

Looks like libcurl is somehow invoked by Sonoma to return the current system proxy settings from SCDynamicStoreCopyProxies ... I wonder how this happens since there're no direct libcurl invokations ... anyways, I've pushed a fix that doesn't require us to link explicitly to libcurl ... @p1tsi was correct, I just forgot to add the fix to the constructor of the HTTP plugin, that called a reqwest::Client::new() :D

thanks all for the help!