`__MACOSX` folder in zip causes solc compiler to break

manuelwedler commented 1 month ago

Original problem

A user reported an issue when verifying from a hardhat build:

I'm trying to verify contracts on meter testnet, after I uploaded the build-info folder generated by hardhat, some of the contracts get verified without issue, but others can't get verified and gave me the error above. build-info.zip (2.1 MB) This is the build-info I used It seems like a compilation error, but locally, when I compile with hardhat, there's no issue at all. Contracts on meter testnet that generates this error are: CErc20 0x040131D113B8939dd921490c4d892C07Cdef8BBF Comptroller 0x0615f2526e8E94DA2372452Aa3a6B29e8e655197 Any suggestions would be much appreciated

The UI displays the following error:

AWS Lambda error: Compiler error:
 [{"component":"general","errorCode":"7858","formattedMessage":"ParserError: Expected pragma, import directive or contract/interface/library/struct/enum/constant/function/error definition.\n --> __MACOSX/._build-info:1:1:\n  |\n1 | \u0000\u0005\u0016\u0007\u0000\u0002\u0000\u0000Mac OS X        \u0000\u0002\u0000\u0000\u0000\t\u0000\u0000\u00002\u0000\u0000 ...\n  | ^\n\n","message":"Expected pragma, import directive or contract/interface/library/struct/enum/constant/function/error definition.","severity":"error","sourceLocation":{"end":1,"file":"__MACOSX/._build-info","start":0},"type":"ParserError"},{"component":"general","errorCode":"7858","formattedMessage":"ParserError: Expected pragma, import directive or contract/interface/library/struct/enum/constant/function/error definition.\n --> __MACOSX/build-info/._0a48e614c1dbe379985d16d1b0a5f9c7.json:1:1:\n  |\n1 | \u0000\u0005\u0016\u0007\u0000\u0002\u0000\u0000Mac OS X        \u0000\u0002\u0000\u0000\u0000\t\u0000\u0000\u00002\u0000\u0000 ...\n  | ^\n\n","message":"Expected pragma, import directive or contract/interface/library/struct/enum/constant/function/error definition.","severity":"error","sourceLocation":{"end":1,"file":"__MACOSX/build-info/._0a48e614c1dbe379985d16d1b0a5f9c7.json","start":0},"type":"ParserError"}] - lamdbaRequestId: bbde6acd-9f02-4402-9ef4-c1cf5299a9a2

After debugging, I could figure out that issue comes from the zip being created from a Mac. Mac adds a __MACOSX folder to a zip file which is only visible on non-Mac machines. This folder is also not inteded to be read on a non-Mac machine.

The follwing happens when uploading the zip file. The server experiences a "extra-file-input-bug", and therefore adds all given source files to the compilation input. The solc compiler simply cannot read the folder and throws the above error.

First fix

I could fix the error message by excluding __MACOSX folders in the unzip function in validation.ts of lib-sourcify. We don't want the server to read such folders, so we can simply ignore it when unzipping.

Fix: #1534

Unfortunately, the server still cannot verify the contract using the zip after this fix.

Follow up problem: different bytecodes from additional empty sources

I could only verify the contracts after adding another folder level in the zip file (build-info/build-info/). This means the input given to solc contains an additional source: "build-info/build-info/": { "content": "" },

This might be an example of the extra-file-input-bug. Even though the content of the additional source is empty, the two inputs give different bytecodes.

As we can verify the contract with some change to the compiler input, we should investigate this further and make it possible to verify this contract without changing the structure of the zip file.

Tasks

To continue debugging, try the following:

[x] Ask user about hardhat repo to debug what input hardhat gives to solc
[x] Investigate some changes to the compiler input, e.g. rename the empty source path, additional empty sources, ...
[x] Try to get compilation working with solc 0.8.21, as there the bug might be fixed and the two inputs could give the same bytecode (given up on that, compilation is failing with YulException: Variable var_suToken is 1 too deep in the stack)

manuelwedler commented 1 month ago

I tried a few variations of the empty sources given to solc. So in the following every path stands for [path]: { "content": "" } in the sources of the standard json input. The non-empty sources were always the same as the ones generated by the server for the user's zip file. Everything was compiled with solc 0.8.19.

Correctly verifiable bytecode

build-info/ and build-info/build-info/
build-info/ and build-info/build-info/ and build-info/build-info/build-info/
No empty sources
build-info/ and build-info2/

Non-verifiable bytecode (same bytecode for all of these variations)

build-info/
build-info/ and empty
build-info/ and empty/
build-info/ and empty1/ and empty2/

Since giving no empty sources also produces the correct bytecode, my guess here is that hardhat didn't add any empty source to the compiler input and the build-info/ folder is just a leftover from the zip file creation. But this is hard to tell without having the original hardhat project.

manuelwedler commented 1 week ago

Unfortunately, the user didn't respond. Without the original hardhat project we cannot debug it further. Closing due to this.

The Solidity issue this problem originates from is here: https://github.com/ethereum/solidity/issues/14494 It should be fixed in newer Solidity versions

ethereum / sourcify