Agree on a sketch of the overall wallet architecture

str4d commented 7 months ago

Rough (and incomplete, hence not put into the top post) sketches from a meeting with Daira-Emma and Kris:

Current light client wallet architecture:

flowchart
    zcashdConsensus

    zcashdConsensus -- Chain tip change --> lightwalletd
    zcashdConsensus -- New blocks --> lightwalletd

    subgraph Indexer
    zcashdDb
    lightwalletd
    end

    zcashdConsensus <-- Stuff --> zcashdDb
    zcashdDb<-- UpdateChainTip --> lightwalletd
    zcashdDb<-- GetUTXOs --> lightwalletd
    zcashdDb<-- GetTransaction --> lightwalletd

    subgraph Mobile wallet app
    Scanner
    Wallet
    end

    Wallet -- Register UFVK --> Scanner
    Wallet -- (LC) [nf], ScanRange --> Scanner
    lightwalletd <-- UpdateChainTip --> Wallet
    lightwalletd <-- GetUTXOs --> Wallet
    lightwalletd <-- GetTransaction --> Wallet
    Scanner <-- CompactBlockRange --> lightwalletd
    Scanner -- ScannedBlock --> Wallet

Sketch of possible desired architecture:

flowchart
    FullValidator
    Indexer
    Scanner
    Wallet

    FullValidator -- Chain tip change --> Indexer
    FullValidator -- New blocks --> Indexer

    Wallet -- Register UFVK --> Scanner
    Wallet -- (LC) [nf], ScanRange --> Scanner
    Scanner <-- (Compact)BlockRange --> Indexer
    Scanner -- ScannedBlock --> Wallet

Wallet
- Trusted with privacy and spend authority.
Scanner
- Trusted with privacy in perpetuity. Has access to user-specific data.
Indexer
- Can observe usage patterns for the duration it is used.
- Trusted with correctness of the chain data, but that trust can be removed with bandwidth increase (Fly Client proofs).
Full Validator
- Trusted with correctness of the chain data.

LC: Light Client

str4d commented 7 months ago

jackgavigan commented 7 months ago

Should the Scanner obtain the blocks from the Indexer, or direct from Zebra's database? (Assuming the scanner is running on the same machine as Zebra.)

I assume that t-address balances and transactions will be obtained by querying the Indexer?

ETA: Also, in the diagram above, is it the Indexer's job to index transparent addresses and transactions (e.g. for the block explorer use case)? I think it's worth being pedantically explicit about the functionality of each component to ensure that we don't have differing understandings/expectations.

jackgavigan commented 6 months ago

Temporary diagram for discussion

   flowchart

   subgraph Zcash_P2P_Network
   Other_nodes
   end
   Other_nodes -- chain state -->   FullValidator
   subgraph Zebra
      FullValidator
      Indexer
      TxBroadcaster
  subgraph State
      NonFinalizedState
      FinalizedState
  end
      FullValidator -- Chain tip change --> NonFinalizedState;
      FullValidator -- New blocks --> NonFinalizedState;
      FullValidator -- Finalized blocks --> FinalizedState
      State -- transparent TX info --> Indexer
   end

   State -- Read Zebra State --> Scanner
   Wallet -- T-addrs query --> Indexer
   Indexer --T-addrs state --> Wallet
   Wallet -- New transaction --> TxBroadcaster
   TxBroadcaster -- New Transaction --> Other_nodes
   Indexer --> BlockExplorer

   subgraph CLI_Wallet
      Scanner
      Wallet -- Register viewing key --> Scanner
      Scanner -- Shielded TX scan results  --> Wallet
   end

mpguerra commented 4 months ago

Are we settled on this design for the wallet architecture? What else is required in order to close out this issue?

pacu commented 4 months ago

Although risky in terms of scope creep, I think that a general purpose approach would be much more helpful in terms of requirement elicitation. Many kind of users, either they are individuals, DAOs, Protocols or other Organizations (like CEXs, Asset Custodians, Banks, etc) need to make use of a Zcash Wallet and currently their needs are not met by our tooling. It's better to think of that beforehand.

This is a layered approach where different application domains can be added horizontally as needed. The CLI can be the first one, but others come to mind.

Principles:

Applications never connect to the Zebra Wallet process directly.
Only the transformation layer connects to the wallet
Applications don't have a contract API with the wallet, they do with the Data Transformation Layer (DTL) logic module of their preference
DTL can be coded independently from wallet as long as interfaces are agreed beforehand.
DTL can stub data for modules to be developed independently from Wallet as well.
DTL modules can be worked independently
Testing with high degree of scope
- unit testing
- layer communication testing (Example: Client Facing API <-> CLI Transformation API)
- end-to-end

flowchart

   subgraph Zcash_P2P_Network
   Other_nodes
   end
   Other_nodes -- chain state -->   FullValidator
   subgraph Zebra
      FullValidator
      Indexer
      TxBroadcaster
  subgraph State
      NonFinalizedState
      FinalizedState
  end
      FullValidator -- Chain tip change --> NonFinalizedState;
      FullValidator -- New blocks --> NonFinalizedState;
      FullValidator -- Finalized blocks --> FinalizedState
      State -- transparent TX info --> Indexer
   end

   State -- Read Zebra State --> Scanner
   Wallet -- T-addrs query --> Indexer
   Indexer --T-addrs state --> Wallet
   Wallet -- New transaction --> TxBroadcaster
   TxBroadcaster -- New Transaction --> Other_nodes
   Indexer --> BlockExplorer

   subgraph Data Transformation Layer
      CLI_API <--> CoreTransformationLogic
      ZcashSnapAPI <--> CoreTransformationLogic
      B2B_API <--> CoreTransformationLogic

      CoreTransformationLogic
   end

   CoreTransformationLogic -- works with native types --> Wallet

   subgraph Wallet CLI
        CLIClientLogic -- mediates formats and lifecycle -->  CLI_API
   end

   subgraph Zcash Metamask Snap
      BackendLogic <--> ZcashSnapAPI
   end
   subgraph Zcash Avax Bridge
      ZavaxDataLayer <--> B2B_API
   end
   subgraph Zcash CEX Infrastructure
      CEX_Backend <--> B2B_API
   end
   subgraph Wallet Backend
      Scanner
      Wallet -- Register viewing key --> Scanner
      Scanner -- Shielded TX scan results  --> Wallet
   end

str4d commented 4 months ago

@pacu that is compatible with the sketch we have in this issue, but you should open a separate issue for discussing it.

zancas commented 3 months ago

Apologies in advance, if this question is naive or out-of-scope.

Should we explicitly consider an API that offers guidance to mobile developers in this discussion?

It's not explicit in the title of this issue that we're focused on CLI-only wallets, and it seems like explicit consideration of other interface cases, even if there's no requirement to implement (or it's already implemented) might be useful in this context.

I am specifically thinking about APIs that encourage non-CLI developers (e.g. mobile) to prefer QR code representations of addresses (or payment requests etc.) over strings.

arya2 commented 2 months ago

Updated temporary diagram based on recent discussions (see note at the end about the block explorer use case):

flowchart

subgraph Zcash_P2P_Network
    Other_nodes
end

Other_nodes -- chain state -->   FullValidator

subgraph Zebra
    FullValidator
    TxBroadcaster
    RpcServer
    subgraph State
        NonFinalizedState
        FinalizedState
    end
    FullValidator -- Chain tip change --> NonFinalizedState;
    FullValidator -- New blocks --> NonFinalizedState;
    FullValidator -- Finalized blocks --> FinalizedState

end

State -- transparent TX and nullifier info --> Indexer

subgraph Zebroid
    Indexer
    IndexerRpcServer
end

ReadStateService --> Scanner
RemoteReadStateService --> Scanner
State --> ReadStateService --> Wallet
IndexerRpcServer --T-addrs state --> RemoteReadStateService --> Wallet
Wallet -- New transaction --> TxBroadcaster
TxBroadcaster -- New Transaction --> Other_nodes
FullValidator --> RpcServer
Indexer --> IndexerRpcServer --> BlockExplorer
RpcServer -- NonFinalizedState via getblock method --> Indexer

subgraph CLI_Wallet
    Scanner["Scanner<br>(zcash_client_backend)"]
    ReadStateService["ReadStateService<br>(zebra-state)"]
    RemoteReadStateService["RemoteReadStateService<br>(zebra-state)"]
    Wallet -- Register viewing key --> Scanner
    Scanner -- Shielded TX scan results  --> Wallet
    WalletRpcServer --> Wallet
end

WalletRpcServer --> Client

The indexer RPC server is shown supporting the block explorer use case here as we've discussed moving some of the RPC methods currently supported in Zebra to the indexer RPC server so that Zebra cannot be called directly with those queries.

conradoplg commented 2 months ago

We just discussed this in the Zcashd Deprecation Collaboration meeting. I'll try to update the figure later but here are the gist of it while it's fresh in my mind.

Current zcash_client_backend-backed lightwallets work like this (where "->" is function calls inside process, and "-->" is across processes):

zcash_client_backed -> block_source --> lightwalletd --> zcashd

The proposal is to first support this:

zcash_client_backed -> block_source --> ("Zebroid": something equivalent to lightwalletd using zebra-state) --> zebrad (and zebra's DB directly)

And then eventually (but not required for zcashd deprecation) we could migrate to:

zcash_client_backed -> (another block source using zebra-state) --> zebrad (and zebra's DB directly)

Please confirm if this is accurate.

idky137 commented 2 months ago

Hey this is our (Zingolabs) current understanding and plan for Zaino (previously Zebroid). It is very similar to Arya's diagram but also includes the "light" wallet architecture. I think from our point of view we plan to carry on using the lightwallet gRPC service for our "light" wallet (Zingo Mobile) but we will also be implementing a remote endpoint for the ReadStateService in Zaino for wallet consumption. My understanding is that this is planned to be consumed by "full node" wallet implementations not "light" wallets though. zcash_wallet_architecture

jackgavigan commented 1 month ago

Annotated diagram based on discussions during the Arborist call on 19 Sept 2024.

Annotated_Zaino_diagram

idky137 commented 1 month ago

I've opened an issue in the Zaino repo (here https://github.com/zingolabs/zaino/issues/69) to finalise the red "gRPC" in the diagram above. I believe there are currently a couple differing ideas regarding this so have put forward an initial option there.

zcash / wallet

Agree on a sketch of the overall wallet architecture #7