Allow to use pipes to deal with big traces

hmijail commented 2 years ago

(This is probably a t8ntool problem, not a retesteth problem, but just in case)

Some tests take a long time to run and generate multi-GB traces. This makes it difficult to deal with those traces and requires a lot of disk space, particularly if one is starting implementation and only the beginning of the trace would be enough.

It would be useful to be able to pipe the traces out of retesteth, either from stdout (in a clean JSONL way) or from a named pipe. That way, another program can consume the pipe in typical unix fashion and block/kill the producer as needed.

Currently this is not doable, mainly because retesteth just moves around the files generated by t8ntool. So when retesteth starts writing its output, the long test has actually already finished and the huge file already exists on disk. Even if that was fixed, if using named pipes (instead of stdout), there is still the problem of needing to know in advance the filename that retesteth will write to - so that we can do mkfifo on that filename.

On the side of Geth's evm, the same problem exists because traces can only be generated to file, and the file's name is the transaction hash; so once again we don't know what filename to use in mkfifo. (I will try to report this to the Geth people)

hmijail commented 2 years ago

Geth issue https://github.com/ethereum/go-ethereum/issues/25544

winsvega commented 2 years ago

I think for the big files we can specify ranges of opcodes to export so then to disect the part where consensus issue occure.

winsvega commented 1 year ago

Now read file line by line. and if its too big to display then stops. if export is requested the file is copied from tmp dir to dest dir. by default tmp dir is set to /dev/shm. can be changed in config by "tmpDir" option

ethereum / retesteth

Allow to use pipes to deal with big traces #179