Rewrite ethashproof in Rust

[MaksymZavershynskyi]

https://github.com/nearprotocol/ethashproof is a fork of a fork of https://github.com/tranvictor/ethashproof While the original project is a great, according to the original author it was in PoC stage for a long time already. Also, it uses utils written by the same author github.com/tranvictor/ethutils to work with Ethereum-specific primitives.

To really own this code we need to understand it deeply, which means we would need to do a lot of in-depth reading of it, including adding lots of our own tests. Since there is a good chance the entire rainbow bridge CLI might be rewritten into Rust for security reasons, we might as well rewrite ethashproof in Rust first. In the process, we might greatly benefit from reusing existing Ethereum Rust utils, like Parity utils that might entirely replace github.com/tranvictor/ethutils which might make the code smaller and easier to verify/read. Also, we would need to write the spec of the ethashproof library in the process, because we would benefit from having a detailed spec of the bridge, overall.

---

Update 2020-08-30

Pros and Cos of Rust instead of JS

Pros

• We can combine this library with rust-ethash, see #284 .
  • Having one codebase that generates paths and verifies them will make granular testing much easier. If we rewrite ethashproof in JS then JS code will be used to generate paths, but Rust code will still be used to validate them.
  • We can reduce the total size of the code, since we might be able to share a lot of code with rust-ethash;
• It is safer to have low-level byte manipulations in strongly typed languages. JS is not meant for low-level byte manipulations as it can unintentionally coerce one type of numerical to another type;
• We might be able to reuse Parity libraries to reduce the code size even further and reduce the surface of errors too. We have already encountered errors in imported JS libraries that operate with Ethereum primitives while building the bridge. Had we used Parity libraries the chance of such errors might have been drastically smaller;
• It might be actually beneficial to have it in Wasm because we need to increase Wasm expertise in our company and have as many people comfortable with it as possible;
• Rust Wasm code can pass DAG paths to the contract as is without having going through bash serialization in JS;
• Rust interfaces have explicit error handling through returning Result, while JS have implicit through exception mechanism. Because we are using some JS libraries current bridge code is riddled with the following pattern, which is very unsafe because of the retries and generic try-catch mechanisms.

  for (var i =0; i < NUM_RETRIES; i++) {
  try {
  ...
  } catch (e)
  {
  console.log(e);
  }
  }

• JS code tends to produce bugs like this: https://github.com/near/rainbow-bridge-cli/issues/331

Cons

• Compiling Rust in Wasm might take 1-2 days of learning. However, @nearmax strongly believes this is the kind of knowledge that many people at NEAR should have.

[ailisp]

Just found we have some initial work on this: https://github.com/nearprotocol/rust-ethash

[ilblackdragon]

Curious considerations of using Rust vs Javascript? The benefit of using JS is that will allow to have pure functions directly in the browser. Not sure what is the performance hit obviously here. With Rust, we can compile to WASM and introduce into browser as well, but definitely more work.

[ailisp]

I suggest rust, because the go implementation took about 4 seconds to proof one block, with js that's going to be much longer.

With Rust, we can compile to WASM and introduce into browser as well, but definitely more work.

I think this case is simpler, because it's almost a single pure function to import

[MaksymZavershynskyi]

@ilblackdragon I have added Rust vs JS pros and cos into the description.

[ailisp]

I think we should definitely do rust-ethash based on pros and cons above

[MaksymZavershynskyi]

Here is another reason to start replacing components with Rust. Conversation is taken from Slack:

@Kouprin (wasn't able to run e2e tests on his machine)

I passed e2e test locally after downgrading yarn from 1.22.4 to 1.22.0. We need to understand why yarn version affects proofs calculation. I also found that sol-contracts bin-files are different with yarn 1.22.4. This may help to investigate. For now, we should add to README requirement to use yarn 1.22.0. To do that, it's needed to execute yarn set version 1.22.0.

@nearmax

Sad, yarn on my machine is 1.22.4 and it passes. It is not the first bug we had with npm and yarn. For example some of our stuff does not work with node version > 13. This is a pretty good reason for migrating our code to a language with more deterministic package management, like Rust. Fortunately we can do it incrementally by compiling Rust components with Wasm.

[ailisp]

Also a lesson i learnt, yarn probably has bug in handle git dependency cache, to ensure it's correctly resolve git dependencies, you need yarn cache clean (I think there was conversation with Chad, Mike and me on slack that finally found this). npm, AFAIK is more reliable because it doesn't have cache :D Also, it's important to commit and check yarn.lock isn't changed after every commit (it's set for rainbow-bridge-sol on ci) rainbow-bridge-lib and rainbow-bridge-cli does commit yarn.lock, but doesn't check yarn.lock stay unchanged on ci

On Sat, Sep 5, 2020 at 12:45 PM Maksym Zavershynskyi < notifications@github.com> wrote:

Here is another reason to start replacing components with Rust. Conversation is taken from Slack:

@Kouprin https://github.com/Kouprin (wasn't able to run e2e tests on his machine)

I passed e2e test locally after downgrading yarn from 1.22.4 to 1.22.0. We need to understand why yarn version affects proofs calculation. I also found that sol-contracts bin-files are different with yarn 1.22.4. This may help to investigate. For now, we should add to README requirement to use yarn 1.22.0. To do that, it's needed to execute yarn set version 1.22.0.

@nearmax https://github.com/nearmax

Sad, yarn on my machine is 1.22.4 and it passes. It is not the first bug we had with npm and yarn. For example some of our stuff does not work with node version > 13. This is a pretty good reason for migrating our code to a language with more deterministic package management, like Rust. Fortunately we can do it incrementally by compiling Rust components with Wasm.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/near/rainbow-bridge-cli/issues/150#issuecomment-687654395, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADFFFCALMNRWVXFT6T25VGTSEKIN7ANCNFSM4OSLQ4JQ .

[tranvictor]

To really own this code we need to understand it deeply, which means we would need to do a lot of in-depth reading of it, including adding lots of our own tests. Since there is a good chance the entire rainbow bridge CLI might be rewritten into Rust for security reasons, we might as well rewrite ethashproof in Rust first. In the process, we might greatly benefit from reusing existing Ethereum Rust utils, like Parity utils that might entirely replace github.com/tranvictor/ethutils which might make the code smaller and easier to verify/read. Also, we would need to write the spec of the ethashproof library in the process, because we would benefit from having a detailed spec of the bridge, overall. knowledge that many people at NEAR should have.

Which security reasons do you think Rust is better than Go? It seems Go is "secure" enough so the official ethereum client is written with...

On the other hand, ethashproof package doesn't need my ethutils package at all, you can use directly ethclient from go-ethereum package. I was using it in the commandline because i was lazy and was used to my own library.

[MaksymZavershynskyi]

Which security reasons do you think Rust is better than Go? It seems Go is "secure" enough so the official ethereum client is written with...

Rust prevents torn reads/writes which is very desirable for cryptographic tools. Since we might need to rewrite it from the scratch anyway and we have most of the code in Rust it makes sense to rewrite it into Rust to keep the number of languages that we need to maintain small. Also, we will be able to link it against our other Rust code if we ever need to.

[drewstone]

Is there somewhere to apply for grants for this work? Someone to reach out to, my email is drew@commonwealth.im

[ailisp]

@nearmax are we going to grant a bounty on this issue? This seems to out of our priority for a while

[drewstone]

Hello, we @edgeware-builders rewrote this library in Rust. It is extensible if you want to help out. We are also interested if this fulfills any bounty since we are trying to find grants to support this work.

https://github.com/rust-ethereum/ethash/pull/11

Is there any interest in providing support? Nonetheless, we hope that this is useful for you all. We use a different merkle tree to build DAG roots but similarly have generated 10 years of roots into the future if it helps anyone out.

@djsatok this might be useful for that issue as well.

[alexauroradev]

@drewstone , hey there! I was literally going through the rainbow-bridge issues today and looking into them. Let's connect via e-mail.

[alexauroradev]

250 related to this task