search

fedimint / fedimint

Federated E-Cash Mint
https://fedimint.org/
MIT License
578 stars 223 forks source link

CLN fail during `just mprocs` #2905

Closed EthnTuttle closed 1 year ago

EthnTuttle commented 1 year ago 
tail: cannot open '/tmp/nix-shell.kJ8666/fm-KXLN/logs/lightningd.log' for reading: No such file or directory
tail: '/tmp/nix-shell.kJ8666/fm-KXLN/logs/lightningd.log' has appeared;  following new file
2023-08-04T02:26:01.376Z INFO    lightningd: Creating configuration directory /tmp/nix-shell.kJ8666/fm-KXLN/cln/regtest
2023-08-04T02:26:02.560Z INFO    lightningd: Creating database
2023-08-04T02:26:02.580Z UNUSUAL hsmd: HSM: created new hsm_secret file

Could not connect to bitcoind using bitcoin-cli. Is bitcoind running?

Make sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.

You can verify that your Bitcoin Core installation is ready for use by running:

    $ bitcoin-cli -regtest -rpcuser=... -rpcpassword=... echo 'hello world'
2023-08-04T02:26:02.603Z **BROKEN** plugin-bcli: \nCould not connect to bitcoind using bitcoin-cli. Is bitcoind running?\n\nMake sure you have bitcoind running and that bitcoin-cli is able to connect to bitcoind.\n\nYou can verify that your Bitcoin Core installation is ready for use by running:\n\n    $ bitcoin-cli -regtest -rpcuser=... -rpcpassword=... echo 'hello world'\n
2023-08-04T02:26:02.603Z INFO    plugin-bcli: Killing plugin: exited before we sent init
The Bitcoin backend died.
thread 'main' panicked at 'Failed to create cln rpc service: Error(Failed to start cln plugin - another instance of lightningd may already be running.

Stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /Users/ethantuttle/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/libunwind.rs:93:5
      backtrace::backtrace::trace_unsynchronized
             at /Users/ethantuttle/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:66:5
   1: backtrace::backtrace::trace
             at /Users/ethantuttle/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.67/src/backtrace/mod.rs:53:14
   2: anyhow::backtrace::capture::Backtrace::create
             at /Users/ethantuttle/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.71/src/backtrace.rs:216:13
   3: anyhow::backtrace::capture::Backtrace::capture
             at /Users/ethantuttle/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.71/src/backtrace.rs:204:17
   4: anyhow::error::<impl anyhow::Error>::msg
             at /Users/ethantuttle/.cargo/registry/src/github.com-1ecc6299db9ec823/anyhow-1.0.71/src/error.rs:83:36
   5: anyhow::__private::format_err
   6: gateway_cln_extension::ClnRpcService::new::{{closure}}
   7: gateway_cln_extension::main::{{closure}}
   8: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
  10: tokio::runtime::park::CachedParkThread::block_on
  11: tokio::runtime::context::BlockingRegionGuard::block_on
  12: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
  13: tokio::runtime::runtime::Runtime::block_on
  14: gateway_cln_extension::main
  15: core::ops::function::FnOnce::call_once
  16: std::sys_common::backtrace::__rust_begin_short_backtrace
  17: std::rt::lang_start::{{closure}}
  18: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/ops/function.rs:606:13
      std::panicking::try::do_call
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal::{{closure}}
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/rt.rs:148:48
      std::panicking::try::do_call
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:483:40
      std::panicking::try
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:447:19
      std::panic::catch_unwind
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panic.rs:137:14
      std::rt::lang_start_internal
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/rt.rs:148:20
  19: std::rt::lang_start
  20: _main)', gateway/ln-gateway/src/bin/cln_extension.rs:56:10
stack backtrace:
   0: rust_begin_unwind
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/panicking.rs:64:14
   2: core::result::unwrap_failed
             at /rustc/fc594f15669680fa70d255faec3ca3fb507c3405/library/core/src/result.rs:1791:5
   3: core::result::Result<T,E>::expect
   4: gateway_cln_extension::main::{{closure}}
   5: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
   6: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
   7: tokio::runtime::park::CachedParkThread::block_on
   8: tokio::runtime::context::BlockingRegionGuard::block_on
   9: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
  10: tokio::runtime::runtime::Runtime::block_on
  11: gateway_cln_extension::main
  12: core::ops::function::FnOnce::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I tried delaying the start of the CLN to rule out race conditions and had the same error. I reverted to a commit prior to the recent Devimint change, same error. I restarted my machine, same error.

EthnTuttle commented 1 year ago

I tried adding --bitcoin-cli to the start function for cln

    pub async fn start(process_mgr: &ProcessManager, cln_dir: &Path) -> Result<ProcessHandle> {
        let gateway_extension_path = cmd!("which", "gateway-cln-extension")
            .out_string()
            .await
            .context("gateway-cln-extension not on path")?;
        let bitcoin_cli_extension_path = cmd!("which", "bitcoin-cli")
        .out_string()
        .await
        .context("bitcoin-cli not on path")?; 
        let cmd = cmd!(
            "lightningd",
            "--dev-fast-gossip",
            "--dev-bitcoind-poll=1",
            format!("--lightning-dir={}", utf8(cln_dir)),
            "--plugin={gateway_extension_path}",
            "--bitcoin-cli={bitcoin_cli_extension_path}"
        );

        process_mgr.spawn_daemon("lightningd", cmd).await
    }
elsirion commented 1 year ago

Which commit are you on? Also, did you try manually running a bitcoin-cli command? What did the bitcoind log say?

EthnTuttle commented 1 year ago

On master branch as of last night. I even nuked the repository and ran just build to make sure I was current. Same error occurred this morning.

└─(07:01:40 on master)──> nix develop                                                                                                                                                                 ──(Fri,Aug04)─┘
warning: ignoring untrusted substituter 'https://fedimint.cachix.org', you are not a trusted user.
Run `man nix.conf` for more information on the `substituters` configuration option.
warning: ignoring untrusted substituter 'https://fedimint.cachix.org', you are not a trusted user.
Run `man nix.conf` for more information on the `substituters` configuration option.
⚠️  ulimit too small. Run 'ulimit -Sn 1024' to avoid problems running tests
💡 Run 'just' for a list of available 'just ...' helper recipes
mbp:fedimint ethantuttle$ which bitcoin-cli
/nix/store/6n48iipig86gb9c12dz8szmhngb3hc8q-bitcoind-25.0/bin/bitcoin-cli
mbp:fedimint ethantuttle$ lsof -i | rg 'bitcoind|fedimintd|lightningd|lnd|cln|gatewayd' | awk '{print $2}' | xargs kill -9^C

mbp:fedimint ethantuttle$ bitcoin-cli -regtest -rpcuser=bitcoin -rpcpassword=bitcoin echo 'hello world'

************************
EXCEPTION: St13runtime_error
Invalid combination of -regtest, -signet, -testnet and -chain. Can use at most one.
bitcoin in AppInitRPC()

mbp:fedimint ethantuttle$ bitcoin-cli -rpcuser=bitcoin -rpcpassword=bitcoin echo 'hello world'
error: timeout on transient error: Could not connect to the server 127.0.0.1:38332 (error code 1 - "EOF reached")

Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
mbp:fedimint ethantuttle$ lsof -i | rg bitcoind
bitcoind  54703 ethantuttle    9u  IPv6 0x384f1dfb25c0df21      0t0  TCP localhost:18443 (LISTEN)
bitcoind  54703 ethantuttle   10u  IPv4 0x384f1e04becebd49      0t0  TCP localhost:18443 (LISTEN)
bitcoind  54703 ethantuttle   22u  IPv4 0x384f1e04bede3fa9      0t0  TCP localhost:38332 (LISTEN)
bitcoind  54703 ethantuttle   25u  IPv4 0x384f1e04bf3e5369      0t0  TCP localhost:38333 (LISTEN)
bitcoind  54703 ethantuttle   36u  IPv6 0x384f1dfb25c0f721      0t0  TCP *:18444 (LISTEN)
bitcoind  54703 ethantuttle   37u  IPv4 0x384f1e04bee03499      0t0  TCP localhost:18445 (LISTEN)
bitcoind  54703 ethantuttle   38u  IPv4 0x384f1e04bedcde79      0t0  TCP *:18444 (LISTEN)
bitcoind  54703 ethantuttle   39u  IPv4 0x384f1e04bef71c19      0t0  TCP localhost:18444->localhost:53932 (ESTABLISHED)
bitcoind  54703 ethantuttle   43u  IPv4 0x384f1e04bf35b499      0t0  TCP localhost:38332->localhost:53941 (ESTABLISHED)
bitcoind  54703 ethantuttle   44u  IPv4 0x384f1e04bede9369      0t0  TCP localhost:18443->localhost:53950 (ESTABLISHED)
bitcoind  54703 ethantuttle   45u  IPv4 0x384f1e04bee15e79      0t0  TCP localhost:38333->localhost:53943 (ESTABLISHED)
mbp:fedimint ethantuttle$
EthnTuttle commented 1 year ago

Just pulled current master and rebuilt. Same error in the mprocs.

EthnTuttle commented 1 year ago 
mbp:fedimint ethantuttle$ bitcoin-cli getblockchaininfo

************************
EXCEPTION: St13runtime_error       
Invalid combination of -regtest, -signet, -testnet and -chain. Can use at most one.       
bitcoin in AppInitRPC()

in the user shell, bitcoin-cli is non-fucntional.

EthnTuttle commented 1 year ago

trying nix-store --gc then nix develop.

EthnTuttle commented 1 year ago

trying nix-store --gc then nix develop.

did not work

elsirion commented 1 year ago 
mbp:fedimint ethantuttle$ bitcoin-cli getblockchaininfo

************************
EXCEPTION: St13runtime_error       
Invalid combination of -regtest, -signet, -testnet and -chain. Can use at most one.       
bitcoin in AppInitRPC()

in the user shell, bitcoin-cli is non-fucntional.

My best guess: your user has a local bitcoin.conf, which specifies a network and bitcoin-cli isn't intelligent enough to notice that the CLI option should override any config that might be there? A fix might be setting the work dir for the CLI as we supposedly do for bitcoind (there should be some bitcoin folder in the temporary directory created by devimint).

justinmoon commented 1 year ago

@EthnTuttle: Is there a bitcoin.conf in ~/Library/Application Support/Bitcoin/?

EthnTuttle commented 1 year ago

This solved it. Thank you.

I’ll dig around for a place to document this. Open to suggestions.

Sent from Proton Mail for iOS

On Mon, Aug 7, 2023 at 11:25, Justin Moon @.***(mailto:On Mon, Aug 7, 2023 at 11:25, Justin Moon < wrote:

@.***(https://github.com/EthnTuttle): Is there a bitcoin.conf in ~/Library/Application Support/Bitcoin/?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

EthnTuttle commented 1 year ago

Reopening and assigning to myself to document somewhere.