`out/Vm.sol/VmSafe.json: hGetContents: invalid argument (cannot decode byte sequence starting from 226)`

[msooseth]

This is the most weird bug I have yet to encounter. This hits our previous release, maybe more than one. If the static, released hevm 0.52 is run standalone as a static binary on a foundry project, we get:

$ forge clean
$ forge build
[⠊] Compiling...
[⠊] Compiling 25 files with 0.8.19
[⠰] Solc 0.8.19 finished in 3.82s
Compiler run successful!
$  ldd ./hevm
        not a dynamic executable
$ ./hevm version
0.52.0 [no git revision present]
$ ./hevm test
hevm: /home/matesoos/tmp/test/out/Vm.sol/VmSafe.json: hGetContents: invalid argument (cannot decode byte sequence starting from 226)

Note that the "starting from..." seems to be nonsensical, strace says:

strace ./hevm test
[....]
read(101, "\n                    \"id\": 14318"..., 8192) = 8192
read(101, "\"mutable\",\n                  \"na"..., 8192) = 8192
read(101, "            \"nodeType\": \"Structu"..., 8192) = 8192
write(2, "hevm: ", 6hevm: )                   = 6
write(2, "/home/matesoos/tmp/test/out/Vm.s"..., 126/home/matesoos/tmp/test/out/Vm.sol/VmSafe.json: hGetContents: invalid argument (cannot decode byte sequence starting from 226)) = 126
write(2, "\n", 1

and if you delete stuff from before it will simply fail at apparently the same place in the text, still indicating 226. Also, it seems to fail quite haphazardly,. I tried running with strace --string-limit=1000000000 ./hevm test to understand, but seems to have little to do with what's in the JSON where it says it's unhappy.

If i delete this file it will fail at:

$ rm /home/matesoos/tmp/test/out/Vm.sol/Vm.json
$ ./hevm test                                  
hevm: /home/matesoos/tmp/test/out/Vm.sol/VmSafe.json: hGetContents: invalid argument (cannot decode byte sequence starting from 226)

Also 226, I don't think that can be believed? If I replace that bad JSON with a good JSON, and delete out/Vm.sol/VmSafe.json, then:

$ rm /home/matesoos/tmp/test/out/Vm.sol/Vm.json
$ cp ./out/Base.sol/TestBase.json /home/matesoos/tmp/test/out/Vm.sol/VmSafe.json
$  ./hevm test                                                                   
Running 1 tests for src/contract.sol:MyContract
Exploring contract
Simplifying expression
Explored contract (7 branches)
Checking for reachability of 3 potential property violation(s)
[PASS] prove_add_value(address,uint256)

So these two files, out/Vm.sol/Vm.json and out/Vm.sol/VmSafe.json are the culprit. But I don't understand why.

Now comes the REALLY weird part. Let's run foundry clean & build. Then let's enter the nix-shell of hevm, navigate to this place, and run the SAME EXACT static binary:

$ cd /home/matesoos/development/hevm
$ nix-shell
[....]
$ cd /home/matesoos/tmp/test/                                                                          impure  
$ ldd ./hevm                                                                                           impure  
        not a dynamic executable
$ ./hevm test                                                                                          impure  
Running 1 tests for src/contract.sol:MyContract
Exploring contract
Simplifying expression
Explored contract (7 branches)
Checking for reachability of 3 potential property violation(s)
[PASS] prove_add_value(address,uint256)

So now the JSON parsing works perfectly fine... even though I'm running a static binary that previously didn't work. And that binary is a haskell binary and apparently the JSON parser was inside the haskell, and of course fully compiled-in. At this point, I was quite sure I'm losing my mind, so I decided to create a VM because I was afraid something is hooking into my IO and potentially replacing/changing what the process sees. So I created a VM (latest stable debian), then installed git, curl, foundry, and copied over the static binary for 0.52. I get the same exact issue. And it also works perfectly fine if I delete the two JSON files. So it's not something to do with my shell. It's actually some issue that likely is hitting other users, too, as long as they are using our static binaries.

What I don't understand is that if it's a static binary, why would it make a difference if I'm running it from under nix-shell?

[msooseth]

I can replicate the same exact behaviour in a virtual machine -- with it working from under nix-shell and not working otherwise.

Also, all previous releases that support foundry are broken:

• hevm-0.51.0
• hevm-0.51.1
• hevm-0.51.2
• hevm-0.51.3
• hevm-0.52

And result in the same exact issue:

hevm-0.51.0: /home/matesoos/tmp/test/out/Vm.sol/Vm.json: hGetContents: invalid argument (invalid byte sequence)

[msooseth]

Just a note that it's likely not the size of the two files out/Vm.sol/Vm.json and out/Vm.sol/VmSafe.json, since they are not the largest, at only 1.2/1.3MB, while e.g. ./out/console2.sol/console2.json is >4MB, and ./out/StdCheats.sol/StdCheats.json is 1.6MB

[msooseth]

Almost surely locale-related. Eventually we do a readFile at Solidity.hs line 350:

readSolc :: ProjectType -> FilePath -> FilePath -> IO (Either String BuildOutput)
readSolc pt root fp = do
  (readJSON pt (T.pack $ takeBaseName fp) <$> (readFile fp)) >>=
    \case
      Nothing -> pure . Left $ "unable to parse: " <> fp
      Just (contracts, asts, sources) -> do
        sourceCache <- makeSourceCache root sources asts
        pure (Right (BuildOutput contracts sourceCache))

And we likely fail there. I say likely because Haskell has such poor debugging facilities, one will never be able to know.

readFile's documentation says:

readFile :: FilePath -> IO Text

*Defined in ‘Data.Text.IO’* *(text-2.0.2)*

The  `readFile`  function reads a file and returns the contents of
 the file as a string.  The entire file is read strictly, as with
  `getContents` . 

Beware that this function (similarly to  `Prelude.readFile` ) is locale-dependent.
 Unexpected system locale may cause your application to read corrupted data or
 throw runtime exceptions about "invalid argument (invalid byte sequence)"
 or "invalid argument (invalid character)". This is also slow, because GHC
 first converts an entire input to UTF-32, which is afterwards converted to UTF-8. 

If your data is UTF-8,
 using  `Data.Text.Encoding.decodeUtf8`   `.`   `Data.ByteString.readFile` 
 is a much faster and safer alternative.

[msooseth]

OK, fixed it, needs to use the recommended Data.Text.Encoding.decodeUtf8 . Data.ByteString.readFile. I'll create a PR now.

[msooseth]

Can be closed once https://github.com/ethereum/hevm/pull/451 is merged

[msooseth]

Merged fix, closing.