search

zcash / zips

Zcash Improvement Proposals
https://zips.z.cash
MIT License
274 stars 156 forks source link

Standardize a protocol for creating shielded transactions offline #693

Open nuttycom opened 1 year ago

nuttycom commented 1 year ago

@str4d commented on Mon Jul 17 2017

With t-addrs, it is possible to run a "split-node" configuration, where the user runs two nodes:

To make a transaction, the user performs the following steps (either via that RPC or their wallet's UI):

We can imagine a similar workflow that would enable users to create shielded transactions in a split-node configuration (actual RPC method names TBD):

We should design and implement the two additional RPC methods that enable this workflow.

Two notes:

@daira commented on Thu Jul 20 2017

If possible, we should make it so that the API doesn't need to change in order to support Sapling's private delegated proving. Also, the preparation RPC should have as near as possible to the same syntax as the Beswick-improved version of z_sendmany.

@str4d commented on Mon Aug 14 2017

Based on https://github.com/zcash/zcash/issues/1997#issuecomment-322050878 I propose the following adjusted workflow:

I also propose that the offline node should create as much of the randomness as possible, since it holds the most trust. Regarding inputs and outputs, we have two possible options:

So, the "proto-transaction" must contain at a minimum:

It would also contain either:

or:

I'm inclined to go with the latter for better privacy and compartmentalisation, but we may need to go with the former depending on how usable the latter is.

@str4d commented on Mon Aug 14 2017

Note that if we go with the latter, we could instead structure the RPC methods so that z_finishtransaction has the exact same syntax as the Beswick-improved version of z_sendmany, but also taking the anchor and Merkle tree paths (file) as input. Internally, the improved z_sendmany could just be a wrapper around z_finishtransaction that passes in the internally-cached paths and internally-chosen anchor.

In fact, the improved z_sendmany will likely end up wrapping the sequence [z_preparetransaction, z_finishtransaction]. What we need to decide is the boundary between those two.

@arielgabizon commented on Mon Nov 20 2017

I wonder if we should leave this for sapling, and do something simpler for the very near future: A watch only option for z-addrs that doesn't require the spending key.

@str4d commented on Mon Nov 20 2017

@arielgabizon See #1997, which this issue depends on.

@daira commented on Mon Nov 20 2017

I'm also in favour of concentrating on #1997 and deferring this until Sapling.

@niutaifan commented on Thu Jan 04 2018

Hello, I'd like to ask how to operate zcash through RPC in Java Web, including the windows and Linux systems, and the port number.

@owenkellogg commented on Thu Feb 07 2019

Support for createrawtransaction with z_addresses as outputs will indeed be very useful. Otherwise payments can't be specific about with unspent transaction outputs they want to spend.

@magmel48 commented on Thu Feb 20 2020

is there any update on such functionality?

@str4d commented on Tue May 12 2020

I spent some time in early 2019 working on a partially-created Zcash transaction format, similar to the PSBT format for Bitcoin but with additional participant modes (to handle proof creation) and privacy considerations. I wrote a demo based on ProtoBuf, which I've now rebased on current master and pushed to https://github.com/str4d/zcash/tree/pczt-wip. There is still a lot of work remaining, but I did manage to use this branch at the time to create a Zcash transaction across several machines.

@daira commented on Sat Aug 29 2020

There is interest in this feature in a forum thread.

@nuttycom commented on Fri May 12 2023

https://github.com/hhanh00/zcash-cold-wallet may be an alternative to this.

magmel48 commented 1 year ago

hello,

@nuttycom, usually it works not as you described. No exchange run their "offline" nodes. Generally, exchanges have two major contours: less secured and more secured. In less secured contour they manage utxo list and build unsigned transaction using some libraries like bitgo, bitcoinjs-lib or something else. In some cases building a transaction can be done via createrawtransaction call, but libraries usually provide more flexibility.

Then this unsigned transaction with metadata (at least about inputs) is sent to more secured contour to get needed signatures to be included in the final transaction also via, again, some library code. This more secured contour usually doesn't have any internet connection or connection to any node since storing any key on any node is not great at least from performance (exchanges can have tens of millions of keys and related addresses) and security aspects. All keys are stored in some fast database or derived using HD wallet xpub/xprv stuff. From my experience bitcoin and bitcoin fork nodes with 800k+ addresses on them become really slow.

Also all non-custodial wallets can't have their even offline nodes with their user private keys since in non-custodial wallets only end users manage their private keys security. Running separate offline node on user machines also bad idea - it will require large machine resources amount.

So, regarding the case there should be another way to create shielded transaction except calling z_shieldtransaction. Ideally, making transaction shielded should be achievable without using any connection to any node with private keys.

Looking forward to your thoughts on this.

nuttycom commented 1 year ago

All keys are stored in some fast database or derived using HD wallet xpub/xprv stuff. From my experience bitcoin and bitcoin fork nodes with 800k+ addresses on them become really slow.

This is interesting; for Zcash unified accounts spending shielded funds, this should not be necessary, since diversified addresses allow the use of a single key to provide spending authority for a very large number of addresses used to receive funds. Unlike when using the transparent pool, you don't need a distinct spending key for each address.

Also all non-custodial wallets can't have their even offline nodes with their user private keys since in non-custodial wallets only end users manage their private keys security. Running separate offline node on user machines also bad idea - it will require large machine resources amount.

Can you provide a bit more detail here? I don't understand exactly what the desired workflow is for this scenario.

So, regarding the case there should be another way to create shielded transaction except calling z_shieldtransaction. Ideally, making transaction shielded should be achievable without using any connection to any node with private keys.

This also is hard for me to understand, and an example of the desired workflow would really help. For shielded transactions, there are essentially 3 phases:

  1. Input selection - this does not require access to private keys, only cached information about the notes being spent.
  2. Proving - this requires access to proving keys, and the requirements are slightly different between Sapling and Orchard.
  3. Signing - this is the stage that requires access to the full spending keys.

Ideally, we'd like infrastructure that we create to serve these use cases to be reusable both for hardware wallets and for exchange-type use cases.

magmel48 commented 1 year ago

I don't know a non-custodial wallet that can create shielded transaction since it requires communication with zcash full node with private keys on this node. Non-custodial wallets can't run these nodes, because they don't have an access to user private keys.

So, if you would like to make the feature widely supported it's better to consider some scenario implementation where shielded transaction can be created without a RPC call to zcash full node with private keys = to nodes without wallet support (https://developer.bitcoin.org/reference/rpc/#wallet-rpcs).

str4d commented 3 months ago

I raised this again today in the context of hardware wallets, and I have a few more thoughts. In the intervening years, we have defined a "proposal" protobuf format, which serves several purposes:

The proposal protobuf format is pretty similar to the demo PCZT protobuf format I was experimenting with in 2019/2020. The proposal format includes more metadata, while the demo PCZT format included note details, but both have e.g. the values of notes being spent and created.

My current idea is that we could define a PCZT format as either:

This could then be passed around in e.g. QR codes to hardware wallets, or taken on a USB drive to an HSM containing a FROST spending key share.

pacu commented 3 months ago

My current idea is that we could define a PCZT format as either:

  • a direct wrapper around the proposal protobuf format (in the same way that the proposal protobuf internally contains ZIP 321 transaction requests); or
  • a separate format (likely with a more canonical encoding) that contains a superset of the information in a proposal (along with note plaintexts etc), such that we can take a PCZT and generate a proposal protobuf encoding from it.

Can we iteratively work on it so that we can start from the proposal (first alternative) and take it from there (towareds ? Or is it not practical because this would actually mean that we would have created versions of this and it will mean more maintenance in the future?

daira commented 3 months ago

I think it's useful to be able to trivially obtain the ZIP 321 request because it is directly a field of the PCZT. Since that's how the proposal format works, we have some experience with that approach and it seems to work fine in that context. I'm not aware of a difference between the two contexts that would make it work any less well. So I don't think of the first option as a lesser alternative that we would have to replace. @str4d, what do you think of as the advantages of the second option?

conradoplg commented 3 months ago

After taking a look at the proposal protobuf format, what I find it weird is that I see the PCZT as something derived from it rather something that contains it.

To be concrete, using an Orchard input as example:

If we have the latter, why would we need the former? It may not hurt but it might be confusing.

And for FROST, we'd need to go a step further:

So it seems we might need multiple formats defining different "phases" of the proposal/plan/partial tx.

nuttycom commented 3 months ago

After taking a look at the proposal protobuf format, what I find it weird is that I see the PCZT as something derived from it rather something that contains it.

To be concrete, using an Orchard input as example:

  • Proposal: if I'm reading it correctly, it is identified as a TXID, index and value
  • Ywallet tx plan: represented as Diversifier, Rho, RandomSeed, witness

If we have the latter, why would we need the former? It may not hurt but it might be confusing.

And for FROST, we'd need to go a step further:

  • FROST plan: represented as the Action itself: nf, rk, cmx, cv_net, etc; also Circuit; essentially an orchard::Bundle<InProgress<Unproven, Unauthorized>, Amount>

So it seems we might need multiple formats defining different "phases" of the proposal/plan/partial tx.

I agree; the Proposal type requires the interpreter to be able to look up transaction inputs, which is not appropriate for the PCZT format. A PCZT would be the result of interpreting a Proposal in the context of the wallet; I think it should result in a different format of data, potentially closer to what ywallet is using but with potentially better type discipline.

conradoplg commented 3 months ago

I agree; the Proposal type requires the interpreter to be able to look up transaction inputs, which is not appropriate for the PCZT format. A PCZT would be the result of interpreting a Proposal in the context of the wallet; I think it should result in a different format of data, potentially closer to what ywallet is using but with potentially better type discipline.

Sounds good!

What I'd like to determine next is: should the PCZT simply be the serialized version of a TransactionData<Unauthorized> or is there some reason it shouldn't?

nuttycom commented 3 months ago

What I'd like to determine next is: should the PCZT simply be the serialized version of a TransactionData<Unauthorized> or is there some reason it shouldn't?

I don't like this option quite much, because it requires serializing absent authorizing data as zeros or something of the sort. Maybe this is okay if we write the parser for TransactionData<Unauthorized> to verify that all those bytes are zeros? I am just generally cautious about zero sentinel values, because I have encountered way too many situations where those null sentinels end up being interpreted as valid data; it's better for those values to be unrepresentable in the data structure if possible.

Part of this hesitancy is from the history of txids that were inherited from Bitcoin, which had txid as just a hash of the serialized form of the transaction instead of a unique value derived from the transaction semantics as it is today. Everything gets mapped down to bytes eventually, but I want to be very cautious about any circumstance in which the byte string itself, absent of structure, might be interpreted as having independent meaning. That may not be the case here, I just want to think about it a little more.

One possible way to approach this would be to define a new prefix for unauthorized transaction data that could be prepended to the ordinary serialized transaction data; that way, a parser that attempted to treat the PCZT data as a normal transaction would fail. That's the kind of safeguard that I think we would need if we decide to use the ordinary transaction serialization; if it's internal to a new format that can't get confused with the normal one, then I think it's probably fine.

nuttycom commented 3 months ago

@conradoplg After some discussion with @str4d a simple encoding of TransactionData<Unauthorized> will not be sufficient, because it's necessary for the PCZT to represent a transaction with each part of the transaction signature data, so that partially-complete transactions can be returned to the transaction builder and aggregated. Also, the proposal contains information that cannot be recovered from the raw transaction data, such as the original UA to which the transaction is being sent.

The format needs to be able to:

So a larger protobuf structure that embeds the proposal is likely the right way forward.

conradoplg commented 3 months ago

Got it. Here's a first attempt at mapping out what we need:

Do you have any preference for the specific format? Protobuf?

str4d commented 3 months ago

Got it. Here's a first attempt at mapping out what we need:

* To display information to the user:

  * Destination addresses (UA if used) and respective amounts
  * Change
  * Fee

This information is a subset of what the Proposal protobuf currently contains; we should ensure that all information in the Proposal protobuf is representable.

* To create the transaction deterministically:

  * All data in `TransactionData<Unauthorized>`

For Sapling-only transactions this will be similar to what my initial PCZT draft (https://github.com/str4d/zcash/blob/pczt-wip/src/pczt.proto) contained. But we probably want to adapt learnings from the Proposal protobuf (both good and bad) in terms of how the multi-pool logic is structured.

* (Partial) Signatures (I feel like this should be a separate format?)

In my initial PCZT draft I had signatures and proofs as explicit fields that were empty until provided. Generalizing to the FROST context, what we'd actually want is some kind of Signature enum that can be either "partial" (containing the group information needed for FROST, along with zero or more signature shares from round 2) or "full" (containing a plain signature in the non-FROST use case like my PCZT draft did, or the post-aggregation FROST signature with FROST information discarded).

Do you have any preference for the specific format? Protobuf?

I think Protobuf would be a good format for an initial draft (for figuring out the data structure), but I doubt it is the format we want to finalize due to the way Protobuf handles optional fields by filling in their default value. This can be fine for byte arrays (as we can interpret the zero-length array as None and require non-zero-length arrays to be the correct length), but for e.g. integer fields we cannot distinguish between None and 0. So we may want to figure out an alternative format that has explicit support for optionals.

str4d commented 1 month ago

Notes from a chat with @daira just now:

str4d commented 4 days ago

While working through the DAG updates for Keystone integration, it occurred to me that the PCZT design needs to ensure that the transparent BIP 32 derivation path needs to support non-hardened child nodes, because transparent signing keys exist below the account level (whereas ZIP 32 derivation paths can be hardened-only and derivation down to the account level is sufficient to obtain the spending key). In particular, the Updater needs to be able to specify the derivation path for ephemeral addresses used within ZIP 320 transaction pairs, to enable Keystone to correctly produce signatures for the second transaction in a pair.