About contributing back a fairly big change

[thomaseizinger]

Hey everyone,

we've vendored this library here and made some bigger changes across the board, most notably:

• Model TxOuts distinctly as explicit or confidential instead of each individual field: https://github.com/comit-network/droplet/blob/8cb45d3029c4edcf499ae8cc21c98e43829a6d59/elements-fun/src/transaction.rs#L79-L90 This model makes expressing nonsensical states like a confidential value with an explicit asset impossible. We've found that quite helpful in trying to wrap our heads around elements. It is quite possible that some edge-cases are not covered but for now all the tests pass.
• Direct integration with our fork of rust-secp256k1 that binds against libsecp256k1-zkp to provide all the cryptographic primitives for blinding / unblinding assets etc. This allows us to not have a dependency on libwally (which doesn't compile to WASM and has kind of quirky APIs). This is not fully complete (see this PR) because we need SigHashCache to land on master here :D

For now, all of this happens within a single cargo workspace so we don't have to bother with releases. However, we are keen to upstream some of these changes. In particular:

1. Could our fork of rust-secp256k1 potentially replace https://github.com/ElementsProject/rust-secp256k1-zkp? I'd assume we would have to go through some bike-shedding of the APIs as we've currently optimized ours for easy of understanding (YMMV) and to prevent misuse as much as possible at the cost of doing extra allocations for example.
2. The already mentioned refactoring of how Transactions are modeled might be of interest. Similarly, it would take some effort to untangle the changes made there to make them upstreamable because we for example didn't adhere to a MSRV of 1.29.

For example, something that would be possible if we can get (1) done is:

• release a new version of secp256k1-zkp to crates.io (it seems that the name secp256k1-zkp is pointing to https://github.com/ElementsProject/rust-secp256k1-zkp)
• depend on it in rust-elements
• upstream changes like ConfidentialTxOut::unblind() which provides a quite straight-forward API to invoke some decently heavy cryptography.

Directly depending on secp256k1-zkp also removes a lot of duplicated knowledge like how to parse and serialize generators, pedersen commitments, proofs, etc. Additionally, it allows us to use all this cryptography from WASM which is why we are going through all of this :)

Thoughts?

[apoelstra]

• I don't think it's nonsensical to have a confidential amount but explicit asset, or vice-versa. These are explicitly supported modes of operation (the former is equivalent to pre-asset CT and may be useful for confidential pegs into CT chains, the latter I agree seems to be nonsensical).
• It's unfortunate that libwally doesn't compile to WASM. Is there an open bug about this? We do target WASM in that library.
• I'd be happy to review your rust-secp-zkp, and even happier to pull it into ElementsProject to replace our current very-out-of-date project. I'll also start an internal discussion to see if it makes sense to invite you to the org, if you'd want.
• ...and also make rust-elements depend on it. This is a longstanding TODO that I was hoping to do this month, but other things are taking longer than expected, as always :)
• ...and pull in your unblind API and others. Thanks so much for doing this work!

[apoelstra]

@thomaseizinger so, I've reviewed the entirety of rust-secp-zkp and it looks great, aside from a few nits (I opened some issues on your repo). If you want to replace our crate with this, I'm happy to do so.

There are a couple options for how to do this... probably the easiest thing would be for you to overwrite the entirety of our rust-secp-zkp with your tree (you can use git-read-tree to do this), add a followup commit which puts our README back (maybe changing Rust 1.22 to Rust 1.29 :)), and then PR that to our repo. This will be a huge noisy diff but it will avoid any history rewriting. @jonasnick and I can review the result, which is pretty straightforward, just re-running the vendor script and looking at the diff, then diffing against upstream rust-secp to see what you changed.

This has the unfortunate effect of losing the "is a fork of rust-bitcoin/rust-secp256k1" that you currently have, but we can just re-add that to the README. In my experience having too many layers of Github forks causes problems anyway because there are bugs in Github that cause it to target PRs to the wrong repo.

Alternate strategies are:

• We delete our repo then you do a transfer of yours into ElementsProject. This seems like we're playing with dangerous tools for no benefit
• I just reset --hard our repo's master branch to your repo's master branch, then force-push this. The result of this would probably be a lot of confusion because our history would be identical to yours but there'd be no evidence of why.

[thomaseizinger]

First, thank you for the positive response!

I don't think it's nonsensical to have a confidential amount but explicit asset, or vice-versa. These are explicitly supported modes of operation (the former is equivalent to pre-asset CT and may be useful for confidential pegs into CT chains, the latter I agree seems to be nonsensical).

That is really interesting! I didn't know about this mode of operation. Are there any test vectors available? I am curious to see if those can still be modeled in a way that makes the IMO more common usecases easier to deal with.

I will open a separate issue for this discussion!

It's unfortunate that libwally doesn't compile to WASM. Is there an open bug about this? We do target WASM in that library.

We didn't open a bug report but failing to compile to WASM wasn't the only thing that made us decide to go with libsecp256k1-zkp directly. We've also found that the APIs of libsecp256k1-zkp are easier to deal with. libwally for example depends on the global, static context instance (which isn't thread-safe to my knowledge) whereas libsecp256k1-zkp follows upstream more closely in its coding conventions and requires a dedicated instance to be passed in. Additionally - at least when it comes to dealing with elements - libwally doesn't add a whole lot on top of libsecp256k1-zkp? Sometimes it is almost the same functionality just with a different name :sweat_smile:

@thomaseizinger so, I've reviewed the entirety of rust-secp-zkp and it looks great, aside from a few nits (I opened some issues on your repo). If you want to replace our crate with this, I'm happy to do so.

That is great, thank you!

Given that we were not sure whether or not we can upstream our changes, we took some care to add our stuff in the least invasive way so we can continue to pull from upstream. As you can see from the history, we only added a couple of patches.

I just reset --hard our repo's master branch to your repo's master branch, then force-push this. The result of this would probably be a lot of confusion because our history would be identical to yours but there'd be no evidence of why.

If the confusion is the only issue, then we can just delete our repo after everything has been upstreamed? As mentioned above, we only added a couple of patches on top of a quite recent rust-secp256k1 (and are planning to update to latest HEAD) so most of the history is really just rust-secp256k1.

Is your intention to maintain the current history of your repo? If not, then another, IMO fairly easy way would be to reset --hard your repository to the latest HEAD of rust-secp256k1 and we simply PR our patches after that!

Semantically, it is really only three things:

1. A change to the API of SharedSecret
2. Vendoring the -zkp fork
3. Adding bindings for all the -zkp stuff

All the other patches are just fixing up stuff where we accidentally broke 1.29 support. We can squash those together so the result is just three patches (or two if (1) is not desired).

[apoelstra]

cc @jonasnick what are your thoughts here?

I don't think we care about the history of our rust-secp-zkp repo, we could just reset --hard it away (or push as a separate branch if Jonas wants to recreate his secp-zkp-dev midlayer between the FFI and Rust).

[jonasnick]

Thanks for your contribution.

Could our fork of rust-secp256k1 potentially replace https://github.com/ElementsProject/rust-secp256k1-zkp?

Is there anything wrong with rust-secp256k1-zkp despite being outdated? The main difference seems to be that rust-secp256k1-zkp is not a fork of rust-secp256k1 which is generally preferable because it doesn't unnecessarily duplicate code and interfaces.

[thomaseizinger]

Is there anything wrong with rust-secp256k1-zkp despite being outdated?

No there isn't! We didn't discover rust-secp256k1-zkp up until we were almost done with our work and it turned out that it didn't provide the bindings to the modules we were interested in anyway (generators, pedersen commitments and range proofs).

rust-secp256k1-zkp provides bindings for the schnorrsig module so these projects are actually more complementary than I initially thought! As such, asking for "replacing" it was probably not appropriate, sorry about that!

It would still be good if we could combine our efforts on this front!

The main difference seems to be that rust-secp256k1-zkp is not a fork of rust-secp256k1 which is generally preferable because it doesn't unnecessarily duplicate code and interfaces.

That is a good point! When we started this work, it wasn't fully clear how we will end up using it so we just started with a fork. Please also note this discussion regarding literal fork / extension.

[jonasnick]

Ah that's good to know. Just noticed now that you've updated your lib and by the way I agree that it's a good idea to optimize the lib for "ease of understanding (YMMV) and to prevent misuse as much as possible". Does comit-network/rust-secp256k1-zkp also support nostd?

I think it's fine to hard reset elementsproject/rust-secp256k1-zkp and push the current master to some archive branch. Before we do that, it would be ideal if comit-network/rust-secp-zkp would support Schnorr signatures. I don't know of anyone using rust-secp256k1-zkp (and they probably shouldn't) but would be nice to keep that functionality.

Please also note this discussion regarding literal fork / extension.

Thanks I hadn't seen that ... assuming you mean issue 1.

[jonasnick]

Before we do that, it would be ideal if comit-network/rust-secp-zkp would support Schnorr signatures. I don't know of anyone using rust-secp256k1-zkp (and they probably shouldn't) but would be nice to keep that functionality.

Oh I'm seeing now that rust-secp already has bip-340 Schnorr support already merged...nevermind.

[thomaseizinger]

Does comit-network/rust-secp256k1-zkp also support nostd?

Partly. For the sys crate, only RangeProof requires std (could be reduced to alloc) because it is dynamic in size: https://github.com/comit-network/rust-secp256k1-zkp/blob/b2a7d2af2fe2d15c23274bcba767bad7f5a3d4a7/secp256k1-zkp-sys/src/zkp.rs#L325-L351

I am not sure if we can get around that in any way? All the other low-level bindings should work in no-std.

The higher-level crate makes a few more trade-offs and allocates to provide APIs like this: https://github.com/comit-network/rust-secp256k1-zkp/blob/b2a7d2af2fe2d15c23274bcba767bad7f5a3d4a7/src/zkp/pedersen.rs#L110-L153

Or this: https://github.com/comit-network/rust-secp256k1-zkp/blob/b2a7d2af2fe2d15c23274bcba767bad7f5a3d4a7/src/zkp/surjection_proof.rs#L24-L30

These could potentially be optimized if we employ a "Structure of Arrays"-like strategy (potentially at zero cost with something like https://docs.rs/soak/0.2.0/soak/). That would allow us to retain the invariant of "need to provide arrays of generators, tags and secret keys of equal length" while avoiding the copying that we currently have to do to call the C API.

This is very much an optimization though and hence I haven't looked into it further. It also requires a fair bit of unsafe code and an additional dependency so I am not sure if it is worth it.

Please also note this discussion regarding literal fork / extension.

Thanks I hadn't seen that ... assuming you mean issue 1.

Yes! Sorry about the broken link. I created a new repository and moved the issues over because I didn't want it to be a GitHub fork now that we changed the strategy of how to integrate with rust-secp. Seems like those links are not forwarded now :/

[apoelstra]

We will probably change the surjection proof to also be variable-sized in the future. Right now there are consensus limits in Elements because of the fixed size, which I regret.

[jonasnick]

so I am not sure if it is worth it.

No, I think making the high level crate easy-to-use and easy-to-write is the right approach. Before pushing this to ElementsProject/rust-secp-zkp I'll have a closer look at the lib in the new year unless @apoelstra beats me to it.

[jonasnick]

I replaced rust-secp256k1-zkp with comit-network/rust-secp256k1-zkp. Also opened https://github.com/ElementsProject/rust-secp256k1-zkp/pull/9 in preparation of a release to crates.io.

[jonasnick]

Just pushed the new version 0.2.0 to crates.io.

[thomaseizinger]

Thanks @jonasnick !

[shesek]

libwally (which doesn't compile to WASM)

It does, officially supported as of https://github.com/ElementsProject/libwally-core/pull/249. Instructions are available here:

https://github.com/ElementsProject/libwally-core#webassembly

[thomaseizinger]

libwally (which doesn't compile to WASM)

It does, officially supported as of ElementsProject/libwally-core#249. Instructions are available here:

ElementsProject/libwally-core#webassembly

I should have probably been more specific. We were trying to compile it to WASM using the wasm32-unknown-unknown target of Rust. That is what is supported by wasm-bindgen and the whole wasm-pack tool stack.

That rules out emscripten and from what we experienced, emscripten support in Rust is currently completely broken anyway due to symbol naming changes in a recent LLVM version.

If I remember correctly, we even managed to compile the provided example to WASM. However, the rest of our code is written in Rust and not being able to integrate with that and compile the whole thing to WASM was a showstopper for us regarding libwally.

In the end, digging deeper revealed that specifically for the crypto required to handle CT on Elements, the functionality provided by libwally is just a thin wrappers around libsecp256k1-zkp. As such, there wasn't really any benefit for us in trying to link against libwally so we went for libsecp256k1-zkp directly.

[thomaseizinger]

We just opened a PR for integrating rust-secp256k1-zkp into rust-elements: https://github.com/ElementsProject/rust-elements/pull/70

Please post your feedback :)

[apoelstra]

rust-secp-zkp is now part of the ElementsProject org and we're well on our way to tightly integrating it with rust-elements.

Thanks so much!!