[epic]: Payment and Router separation

paymentLifecyle is the main struct to handle sending payments and can be moved out of routing package, which has the benefits,

indepedent package is easier to analyze/maintain, e.g., we can easily increase logging verbosity for this subservice.
fixed an interface violation which can cause a race condition when sending HTLCs.

Current State

As of today, multiple components are involved when sending payments, as shown below,

flowchart LR
        pay_req-->paymentLifecycle

        subgraph paymentLifecycle
        p[resumePayment]
    end

    subgraph PaymentSession
        RequestRoute-->p
        p-->UpdateAdditionalEdge
        p-->GetAdditionalEdgePolicy
    end

    subgraph MissionControl
      p-->ReportPaymentSuccess
      p-->ReportPaymentFailure
    end

    subgraph PaymentAttemptDispatcher
        p-->SendHTLC
        GetAttemptResult-->p
    end

    subgraph ControlTower
        crud[DB operations]-->p
        FetchPayment
        FailPayment
        more[...]
    end

    subgraph ChannelRouter
        p--->ChannelRouter.applyChannelUpdate
    end

The components are,

ControlTower: handles interaction with DB(CRUD)
MissionControl: records success probablity
PaymentSession(paymentSession): requests routes and updates edges/policies from private channels
PaymentAttemptDispatcher: interacts with Switch to send HTLC and get HTLC attempt results, which is an interface violation.
ChannelRouter.applyChannelUpdate: updates edges using failed HTLCs

Proposed Design

The proposed architecture,

flowchart LR
    pay_req-->paymentLifecycle

    subgraph paymentLifecycle
      p[resumePayment]
      ct
    end

    subgraph ct[ControlTower]
      crud[DB operations]
      FetchPayment
      FailPayment
      more[...]
    end

    subgraph cg[ChannelRouter]
      p-->sa[SendAttempt]
      ps
      mc
      pad
    end

    subgraph ps[PaymentSession]
      RequestRoute
      UpdateAdditionalEdge
      GetAdditionalEdgePolicy
    end

    subgraph mc[MissionControl]
      ReportPaymentSuccess
      ReportPaymentFailure
    end

    subgraph pad[PaymentAttemptDispatcher]
      SendHTLC
      GetHTLCResult
    end

Some highlights,

Payment lifecycle is in its own package (payment) and interacts with the third layer ChannelRouter.
The only interface method used by paymentLifecycle is ChannelRouter.SendAttempt, which takes an attempt request specifying params like routes, onions, amt, etc.
"HTLC" is a concept used in htlcswitch (layer-two), and "HTLC attempt" or "Attempt" (or other better names?) is a concept used in routing (layer-three).

By adding SendAttempt, htlcswitch is no longer exposed to paymentLifecycle. Instead, an HTLC attempt always goes through the graph managed by ChannelRouter first, and it's up to ChannelRouter to decide whether a route can be used or not, which solves the race condition when two HTLC attempts are using the same route without being aware of each other, as manifested in itest flakes.

Plan

[ ] Create a new package and move paymentLifecycle out of routing.
[ ] Add interface method SendAttempt, implement it in routing and use it in paymentLifecycle.

lightningnetwork / lnd