Feature Request: Incremental Compilation

dapphub / dapptools

Dapp, Seth, Hevm, and more

https://dapp.tools

2.1k stars 325 forks source link

Feature Request: Incremental Compilation #713

Open brockelmore opened 3 years ago

brockelmore commented 3 years ago

In large repos with many contracts, compilation can be the longest run time in the testing process. Since the compiler tries to be stateless, but dapptools does not, it would be helpful to watch for changes and determine if a contract needs to be recompiled. With how closely dapptools integrates git, I don't imagine this would be a massive lift.

MrChico commented 3 years ago

I dont see how this can be done from dapp rather than solc

brockelmore commented 3 years ago

Chris from the lang team seems to insist that it is devtool responsibility and that the compiler will not be adding it

d-xo commented 3 years ago

I think to do that we would need access to the full import tree? The compiler does not expose this afaik and building this ourselves seems like a non trivial piece of work.

With how closely dapptools integrates git, I don't imagine this would be a massive lift.

@brockelmore how do you see this working? Perhaps I'm missing something.

brockelmore commented 3 years ago

i mean there is the simple version and the complex version. the simple version is very dumb and basically is: changed & imported elsewhere? rebuild all contracts that depend on it. !changed ? don't rebuild. basically just a filter on the list of contracts to compile that if we have stored compilation do the simple thing and just exclude that from the contracts to build. this helps with monorepos as most of the time you don't want to actually compile every contract (and don't need to), but tools are such that it is time consuming to specify the specific contract. if it has to rebuild all dependencies that fine, a single contract is generally quick because of bytecode limits anyway

MrChico commented 3 years ago

It seems like to be able to do this we would need to at least:

store a copy (or hash) of the compiler input settings to see when they change
analyze the inheritance tree before compiling (flatten can only analyze it from the already compiled version, we would need to start parsing raw solidity)
introduce a way to merge compiler output jsons in a safe manner. This is useful in other contexts and has been discussed before #631.

With this we can compare the hash of the srccode and only recompile the necessary files in the way you describe.

If solc could give us just the inheritance tree, that would help a lot I think.

d-xo commented 3 years ago

opened a solc issue regarding the import graph: https://github.com/ethereum/solidity/issues/11716

d-xo commented 3 years ago

ok so it seems like we can use solc to parse the source files and then build the import graph without too much effort by processing the parsed soliditiy AST.