Add circuitbreaker to lnd

pinheadmz commented 3 months ago

Closes: https://github.com/bitcoin-dev-project/warnet/issues/228 Requires (based on) https://github.com/bitcoin-dev-project/warnet/pull/287

Adds one more data element to tanks in the graphml file:

 <data key="ln-cb-image">carlakirkcohen/circuitbreaker:endorsement-experiment</data>

This adds a container running circuit breaker, and connects it to the corresponding lnd container. In K8s, both containers are in the same pod so they can interact via localhost and share a volume at /root/.lnd in both containers. In docker they also share a volume with that same path (this is needed to pass circuitbreaker the credentials generated by lnd).

~I'm not sure how else to test circuit breaker~, but the logs look promising. It keeps retrying to connect until it finally does:

2024-02-27 12:55:38 2024-02-27T17:55:38.313Z    INFO    Circuit Breaker starting        {"version": ""}
2024-02-27 12:55:38 2024-02-27T17:55:38.313Z    INFO    Opening database        {"path": "/root/.circuitbreaker/circuitbreaker.db"}
2024-02-27 12:55:38 2024-02-27T17:55:38.314Z    ERROR   Unexpected exit {"err": "open /root/.lnd/tls.cert: no such file or directory"}
2024-02-27 12:55:38 main.main
2024-02-27 12:55:38     /src/main.go:144
2024-02-27 12:55:38 runtime.main
2024-02-27 12:55:38     /usr/local/go/src/runtime/proc.go:250
2024-02-27 12:55:44 2024-02-27T17:55:44.878Z    INFO    Circuit Breaker starting        {"version": ""}
2024-02-27 12:55:44 2024-02-27T17:55:44.878Z    INFO    Opening database        {"path": "/root/.circuitbreaker/circuitbreaker.db"}
2024-02-27 12:55:44 2024-02-27T17:55:44.884Z    INFO    Press ctrl-c to exit
2024-02-27 12:55:44 2024-02-27T17:55:44.884Z    INFO    Grpc server starting    {"listenAddress": "127.0.0.1:9234"}
2024-02-27 12:55:44 2024-02-27T17:55:44.884Z    INFO    CircuitBreaker started
2024-02-27 12:55:44 2024-02-27T17:55:44.884Z    INFO    HTTP server starting    {"listenAddress": "127.0.0.1:9235"}
2024-02-27 12:55:44 2024-02-27T17:55:44.886Z    INFO    Connected to lnd node   {"pubkey": "0378b895ecaff7d8a140a1626c46b0d006cd0107dcec2aa261749e338bfc3c17fa"}
2024-02-27 12:55:44 2024-02-27T17:55:44.887Z    INFO    Interceptor/notification handlers registered

UPDATE: add rpc exec_run to make a request from the circuit breaker local API and check the forwards.

reference:

wget -q -O - localhost:9235/api/forwarding_history

{
  "forwards":
  [
    {
      "addTimeNs": "1709062043928206045",
      "resolveTimeNs": "1709062044022878253",
      "settled": true,
      "incomingAmount": "11000",
      "outgoingAmount": "10000",
      "incomingPeer": "000000000000000000000000000000000000000000000000000000000000000000",
      "incomingCircuit":
      {
        "shortChannelId": "123145302441984",
        "htlcIndex": 0
      },
      "outgoingPeer": "000000000000000000000000000000000000000000000000000000000000000000",
      "outgoingCircuit":
      {
        "shortChannelId": "123145302376448",
        "htlcIndex": 0
      },
      "incomingEndorsed": false,
      "outgoingEndorsed": false,
      "cltvDelta": 40
    }
  ]
}

pinheadmz commented 3 months ago

TODO: Still gotta figure out how to get pre-populated databases into those new cb containers

pinheadmz commented 3 months ago

rebased on main after #290 so test base healthcheck should wait for circuit breakers, which was causing failures before

bitcoin-dev-project / warnet

Add circuitbreaker to lnd #288