Not very graceful exit from CTRL-C

[m3dwards]

I used CTRL-C to stop sim-ln and got a bunch of panics:

2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4566 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4571 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4576 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4581 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4586 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4591 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4596 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4601 panicked
2023-09-11T14:01:59.526Z ERROR [sim_lib] Simulation results producer task exited with error: task 4606 panicked

[carlaKC]

Hm, I suspect that this may be because we need to merge #75 and you were running with CLN activity - will check it out!

[sr-gi]

Can we confirm this @maxwedwards?

[m3dwards]

I can confirm I was running CLN as well as LND. If you need I can try it without a CLN node.

[sr-gi]

Oh no, I mean if you can try it out now that #75 is merged

[m3dwards]

I'm struggling to get it to run now. I keep getting:`

2023-09-15T11:59:44.191Z INFO [sim_cli] Connected to 03842469335b6ebaa457 - Node ID: 03842469335b6ebaa45726c8d7b15e6df00f7802b347abc34216535881d6a82ef4.
2023-09-15T11:59:44.197Z INFO [sim_cli] Connected to BIZARREFEED-v23.08-modded - Node ID: 026ba0f8fd6d39f5dfd59cbfcb58dbe828512d97ee3a2f5c17736497e28c3b8a34.
2023-09-15T11:59:44.198Z INFO [sim_lib] Running the simulation forever.
Error: Lightning Error: GetNodeInfoError("Get node info error status: NotFound, message: \"unable to find node\", details: [], metadata: MetadataMap { headers: {\"content-type\": \"application/grpc\"} }")

Caused by:
    Get node info error Get node info error status: NotFound, message: "unable to find node", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

I've stared at the config file and limited it to just one activity and checked and rechecked the pubkeys and they seem correct to me. What am I missing? My config:

{
  "nodes": [
    {
      "LND": {
        "id": "03842469335b6ebaa45726c8d7b15e6df00f7802b347abc34216535881d6a82ef4",
        "address": "https://localhost:28103",
        "macaroon": "/Users/max/source/sim-ln/auth/carol/admin.macaroon",
        "cert": "/Users/max/source/sim-ln/auth/carol/tls.cert"
      }
    },
    {
      "CLN": {
        "id": "026ba0f8fd6d39f5dfd59cbfcb58dbe828512d97ee3a2f5c17736497e28c3b8a34",
        "address": "https://localhost:28101",
        "ca_cert": "/Users/max/source/sim-ln/auth/alice/ca.pem",
        "client_cert": "/Users/max/source/sim-ln/auth/alice/client.pem",
        "client_key": "/Users/max/source/sim-ln/auth/alice/client-key.pem"
      }
    }
  ],
  "activity": [
    {
      "destination": "026ba0f8fd6d39f5dfd59cbfcb58dbe828512d97ee3a2f5c17736497e28c3b8a34",
      "source": "03842469335b6ebaa45726c8d7b15e6df00f7802b347abc34216535881d6a82ef4",
      "interval_secs": 1,
      "amount_msat": 2000
    }
  ]
}

Output from ScalingLightning:

➜  scaling-lightning git:(main) ./sl pubkey -n alice
026ba0f8fd6d39f5dfd59cbfcb58dbe828512d97ee3a2f5c17736497e28c3b8a34
➜  scaling-lightning git:(main) ./sl pubkey -n carol
03842469335b6ebaa45726c8d7b15e6df00f7802b347abc34216535881d6a82ef4

[carlaKC]

I suspect that this has something to do with gossip sync - when running on the regular settings there's a lot of time and block based rate limiting that goes on that's suuuper annoying for local dev. Building CLN in dev mode with --dev-fast-gossip helps a lot, but (afaik) LND doesn't have a similar setting.

Should be able to confirm that issue with: cl listnodes | grep {pubkey}?

On the sim-ln side, I think that we should expand our wait logic to more gracefully handle this by adding some backoffs / retry loop. Where we won't be able to address this is if it's a block height based local sync problem, but we can at least prompt that.

[m3dwards]

It was my own stupid fault, just hadn't connected nodes properly.

Things are much cleaner now:

2023-09-18T13:07:35.599Z DEBUG [sim_lib] Sending action for 76ec9c2020d5d10d2ad19ff2d4e9df6ae12c1aa3dbb1aa50e6831a2a3bd5d156.
2023-09-18T13:07:35.599Z ERROR [sim_lib] Track payment failed for ce7c414b83ae8339ed4869e5dd6c954ca574b515bf9136c75c92a91cd5deb89a: Track payment error Shutdown before tracking results.
2023-09-18T13:07:36.184Z DEBUG [sim_lib] Send payment: 0219e492854bea514aa6d24cf1056baf3aae2caa27f769c4afe37004fd25c44d7c -> 0289a36b0f5aac93932b3d77998135de1b212582ed475f29572e6a62779eeb4b96: (65fec49fd330b0b34e30c1f08e894f59d6d3bdbfaeffb9cd87c93af52c06f41e).
2023-09-18T13:07:36.184Z DEBUG [sim_lib] Sending action for 65fec49fd330b0b34e30c1f08e894f59d6d3bdbfaeffb9cd87c93af52c06f41e.
2023-09-18T13:07:36.184Z ERROR [sim_lib] Error sending action outcome: SendError { .. }.

Still two errors but not the loads we had before.

[carlaKC]

Should all be addressed by now!