EOF test vectors format

[winsvega]

Following the EOF specific tests suggestion: the simple vectors format that will check if the given eof code valid or not generatable with t8n

https://github.com/ethereum/tests/pull/1148/files#diff-b95553933f4fb419dff01b27d138dfe26474822089f360eab090dffe60f4f23c

OUTDATED

"Filler"
{
    "testname" : 
    {
        "data" : [
            ":raw 0xeof",
            ":label code :raw 0xeof2",
            ":yul-eof {...}",
        ],
        "expect" : [
            {
                "indexes" : {
                        "data" : 1
                },
                "network" : [">=Berlin"],
                "result" : true
            },
           {
                "indexes" : {
                        "data" : ":label code"
                },
                "network" : [">=Berlin"],
                "result" : true
            },
           {
                "indexes" : {
                        "data" : ":label invalid_code"
                },
                "network" : [">=Berlin"],
                "result" : false,
                "expectException" : {
                    ">=Berlin" : "EOF_INVALID_HEADER"
                }
            }
        ]
    }
}

"Test"
{
    "testname" : 
    [
        {
            "index" : "0",
            "code" : "0xeof0",
            "results" : 
            {
                "Berlin" : {
                      "result" : false,
                      "exception" : "EOF_INVALID_HEADER"
                 },
                "Shanghai" : {
                      "result" : true
                 }
            }
        },
        {
            "index" : "1",
            "code" : "0xeof2",
            "results" : 
            {
                "Berlin" : {
                      "result" : false,
                      "exception" : "EOF_INVALID_HEADER"
                 }
                "Shanghai" : {
                      "result" : false,
                      "exception" : "EOF_INVALID_HEADER"
                 }
            }
        }
    ]
}

[lightclient]

Might also be interesting to have error codes passed out and standardized?

[winsvega]

yes. t8n tool report to stderror error message that we can check against

[winsvega]

Can also combine expect and data ibto one object and that seals one test code, one exception.

Or if its better in state test style when you can assign one exception for many test vectors?

[lightclient]

I think the filled vectors should be as simple and verbose as possible. I don't have any strong preference on how the filler looks or what optimizations are there.

One other comment: why is "testname" an array? Wouldn't you want each test to have a name and only one input and an output for each fork it is filled against?

[qbzzt]

Wouldn't you want each test to have a name and only one input and an output for each fork it is filled against?

Not necessarily, we might want to group related tests together.

[qbzzt]

"data" : [
            ":raw 0xeof",
            ":label code :raw 0xeof2",
            ":yul-eof {...}",
        ],

I would appreciate the ability to have multiple lines in raw code, because that will let me put comments at exactly the right place.

And of course I want to write fillers in yml, not JSON.

[shemnon]

Being able to comment within code strings would be a great help to pinpoint what exactly is being tested.

":label CALL :raw 0xef0001 010008 020002-000c-0002 030000 00 
# 7 inputs 1 output,
00000007-07010007 
59-59-59-59-59-59-59-b00001-50-b1
# No immediate data
f1-b1"

[lightclient]

Gently bumping again that I would much prefer the filled tests be direct mappings of sting -> test rather than to a list of tests. I've generated some reference tests of this format from the fuzzing corpus Martin put together: https://gist.github.com/lightclient/3442c744e03c58435c7b50013733481d

[winsvega]

Being able to comment within code strings would be a great help to pinpoint what exactly is being tested.

":label CALL :raw 0xef0001 010008 020002-000c-0002 030000 00 
# 7 inputs 1 output,
00000007-07010007 
59-59-59-59-59-59-59-b00001-50-b1
# No immediate data
f1-b1"

this is actually reasonable now with eof. I should introduce spaces and - support in a bytecode as we use raw bytecode now so often makes sense to make it more readable

[winsvega]

lets stick to this format of a filled test : https://gist.github.com/lightclient/3442c744e03c58435c7b50013733481d and in the test filler we can use the same logic as in state test. (many data vectors verified by expect section by labels) agregation by label is quite good. and can have many similar vectors in one file.

[winsvega]

Here is an example. I did a small change. Test vectors packed into test, since 1 test can have many vectors. (corpus) And each test has an info about filling tool, so this info is added to the test once.

One more notice. I remember people asked no int no bool fields in test jsons since it is a bother to parse for some. we can change bool fields to string.

https://github.com/ethereum/tests/pull/1148/files#diff-b95553933f4fb419dff01b27d138dfe26474822089f360eab090dffe60f4f23c

[holiman]

I remember people asked no int no bool fields in test jsons since it is a bother to parse for some.

I don't see how anyone can have a problem with booleans. With ints, sure, I that can happen on platforms where integer support is only 53 bits wide (javascript). But IMO there's no point in converting true into "true"

[winsvega]

is this good? EOFTests/efExample/efExample.json