search

openethereum / sol-rs

Solaris - Solidity testing framework in Rust.
GNU General Public License v3.0
54 stars 14 forks source link

Solaris Tests Mirroring Blockchain Logic #26

Closed lght closed 6 years ago

lght commented 6 years ago

Not sure what the title of this issue should be.

There is currently a section of the main README.md that reads:

... will run exactly the same as on chain

Imo, there is no way we could know this at this point, nor has it been expressed to me that is a goal Solaris should aim for. Until we have a method to test / verify the claim that Solaris tests will perform exactly as contracts do on-chain, we should not be making such claims.

Perhaps doing two passes on test contracts, one on ParityEVM and one KEVM, will give us reasonable assurance that at least the interpreter portion behaves the same? Or make this two-pass an option for people highly concerned with correctness?

I believe there is still great utility in using Solaris to verify a contract's behavior in isolation. However, there are a few big differences in how Solaris tests work, and how the contract will run on a live blockchain.

At the moment, Solaris ignores all consensus and blockchain logic. For a tool meant to unit test smart contracts, this makes total sense (i.e. we aren't concerned with how contracts are added to the ledger, only contract behavior once it is on the blockchain).

Solaris is using Parity's EVM interpreter, default used by Parity nodes?, while other nodes on the blockchain may be using EVMJIT or GETH's EVM interpreter. Each interpreter will have subtle differences that may (not) affect a smart contract's execution.

Are the goals for Solaris extend to making it a fully comprehensive unit + integration (+ fuzzing?) test framework, that plugs in the EVM and blockchain logic from each client and chain? If so, then perhaps we can make claims that (eventually) behavior tested in Solaris will be the same on deployed chains.

TLDR: It may be safer to make smaller claims about what exactly Solaris tests will tell about the behavior of a smart contract.

kirushik commented 6 years ago

@lght I think we should close this, as discussed in Matrix channel. Extending README to better reflect the state of things is a good idea, though.