Getrandom issue even though local execution works

[drewstone]

We've pushed some new runtime modules to our testnet and are getting some new errors in execution. These modules deal with zero-knowledge verification and hashing. In particular, an OsRng is instantiated to seed random values for the verifier here.

Perhaps it is related to --wasm-execution Compiled that we have some nodes running with though we're a bit unsure. Any ideas? We're seeing errors such as:

Jun 14 23:49:13 berlin6 run_node.sh[22489]: 2021-06-14 23:49:13.677 ERROR tokio-runtime-worker runtime: panicked at 'Error: getrandom: this target is not supported', /cargo-home/registry/src/github.com-1ecc6299db9ec823/rand_core-0.5.1/src/os.rs:63:13

An example node execution is:

target/release/edgeware \
    --base-path /mnt/blockstorage/edgeware-db/edgeware \
    --node-key=$KEY \
    --chain=$CHAINSPEC \
    --validator \
    --unsafe-rpc-external \
    --unsafe-ws-external \
    --rpc-cors="all" \
    --name $NODE_NAME \
    --wasm-execution Compiled \
    -lrpc=trace

The system works locally and without problems. Transactions occur in a timely fashion however on the live testnet they take up to 30 seconds and potentially 60 seconds while slowing the chain down throughout the process. Any insight or ways to debug would be appreciated.

[drewstone]

I'll try out --wasm-execution interpreted-i-know-what-i-do and then without it to see if we achieve any speedup. I was under the assumption that our usage of getrandom was compatible in certain WASM environments.

[drewstone]

@burdges maybe you have an idea, inspecting old discussion here using the same transcripting system?

[rphmeier]

Hey Drew,

The Wasm execution of the runtime needs to be completely deterministic, so getrandom should not be allowed in any scenario. The real issue here is that it works at all in any case.

[drewstone]

Ok, glad that hopefully this provides good feedback @rphmeier. How then could I reasonably instantiate an prng for the verification? I was thinking it would be great if the RandomnessCoinFlip module could implement the RngCore + CryptoRng traits since I think those can work in no_std. Any thoughts on this? Or is this lib basically unusable unless I expose host functions to do the job.

Beyond that, I'm guessing that even if I use host functions that this functionality would never work in a parachain runtime right?

[drewstone]

Or at least if anyone knows of a prng module that can take a deterministic random seed through the RandomnessCoinFlip module and provide these traits so that verification can take place in a pallet.

[burdges]

Do not use RandomnessCoinFlip in production. It's basically not secure for anything.. just useful for tests.

[burdges]

What is the actual problem? Is this randomness fora proover or a verifier? As a rule, one cannot run proovers from PVFs anyways, but native off chain workers could run them just fine. Assuming it's a batch verifier.. I've not looked into bulletproofs closely recently, and never too closely at the batching, but usually protocols with random batching could actually be deterministically batched via Fiat-Shamir. It requires being careful however..

[rphmeier]

If you need a PRNG you could run ChaCha32 on the BABE VRF or epoch randomness. But this is public info.

[drewstone]

This is randomness for a verifier @burdges and we're using Aura so we don't have access to any BABE randomness @rphmeier. I hear you that that randomness may not be terribly secure, but does it really matter if I just need to instantiate a PRNG for verifier randomness?

[burdges]

It's verifiers I see.. We should discuss this with someone who looked more closely at bulletproofs, but..

I doubt VRFs help here, he needs Fiat-Shamir.

It's likely they're using merlin's transcript rng builder, which takes another Rng+CryptoRng, so folks give it OsRng. At this stage, the transcript much be built identically in the proover and the verifier, so maybe it maybe lacks the complete picture. It's likely save to do roughly

let mut seed_block_hash = [u8; 32]::default();
merlin::Transcript::new(b"foo")  // or whatever hash you like
.append_message(b"bar", all_transactions_bodies_with_proof.as_bytes())
.challenge_bytes(b"baz", &mut seed);
let mut fs_rng = ChaChaRng::from_seed(seed_block_hash);

You then pass fs_rng into bulletproof so it gets used in code that looks like

let mut rng = transcript
    .build_rng()
    .rekey_with_witness_bytes(b"witness1", witness_data)
    .rekey_with_witness_bytes(b"witness2", more_witness_data)
    .finalize(&mut fs_rng);

What this is doing is making sure that the whole all_transactions_bodies_with_proof aka block hash gets taken into account in the Fiat-Shamir transform.

[burdges]

Anyways your question is: How do I derandomize a bulletproofs batch verifier with Fiat-Shamir?

It probably just works but I cannot dig into the bullet proofs paper right now.

[drewstone]

Ok, I'm not using the batch verifier or range proofs, rather using their R1CS interfaces. I will read through this and continue to debug.. I think I'm finding ChaCha to be in the right direction too.

Thanks for the feedback so quickly!

[zhenfeizhang]

Ok, glad that hopefully this provides good feedback @rphmeier. How then could I reasonably instantiate an prng for the verification? I was thinking it would be great if the RandomnessCoinFlip module could implement the RngCore + CryptoRng traits since I think those can work in no_std. Any thoughts on this? Or is this lib basically unusable unless I expose host functions to do the job.

Beyond that, I'm guessing that even if I use host functions that this functionality would never work in a parachain runtime right?

I think I may have encountered this bug before. IIRC, my solution was to use ark_std::rand::{RngCore, SeedableRng};. It worked for arkworks version 0.2.0 (whose rand was a wrapper for rand version 0.7.x), rand_chacha version 0.2.0. Not sure this will solve your problem.

[burdges]

I'm not using the batch verifier or range proofs, rather using their R1CS interfaces

I think provers require real randomness for zero-knowledge so if you hit https://doc.dalek.rs/src/bulletproofs/r1cs/prover.rs.html#343 then you'll need it to run native on the parachain nodes.

You've presumably hit https://doc.dalek.rs/src/bulletproofs/r1cs/verifier.rs.html#374 so likely you could fork the bulletproofs crate and make Verifier::verify take an impl Rng or some seed for ChaChaRng, and then derive the seed from the block hash, or maybe just the transaction in the unbatched case.

Actually Verifier::verify never touches self.transcript after this point, so maybe even ChaChaRng::from_seed([083; 32]) turns out secure, but do not do this because we're really not sure. ;)

I only say likely because I've never checked whether this turns out secure. It's ages since I've looked into Bootle's paper or the bulletproofs paper that adds batching. In fact, it'll depend somewhat upon what your circuit actually does. @AlistairStewart could tell you stories about smart contracts that compute their own hash, but that's a story for another day.

You're likely find if all your doing is simple asset transfers, not scripting ala smart contracts.

[bkchr]

I think this can be closed, if not please reopen.