ethereum / execution-spec-tests

A Python framework and collection of test cases to generate test vectors for Ethereum execution clients
https://ethereum.github.io/execution-spec-tests
MIT License
88 stars 62 forks source link

feat(pytest): directly create properties file and release tarball from fill #627

Closed danceratopz closed 1 week ago

danceratopz commented 2 weeks ago

🗒️ Description

Adds some release-specific requirements directly to fill command:

  1. If --output=fixtures.tgz is specified, then additionally create the tarball. From the help:

    --output OUTPUT       Directory path to store the generated test fixtures. Can be
                        deleted. If the specified path ends in '.tar.gz', then the
                        specified tarball is additionally created (the fixtures are
                        still written to the specified path without '.tar.gz' suffix).
                        Default: './fixtures'.
  2. Always writes a fixtures build properties file fixtures.ini to the output directory. A build name may be specified via --build-name:

    --build-name BUILD_NAME
                        Specify a build name for the fixtures.ini file, e.g.,
                        'stable'
  3. Uses 1. and 2. in .github/actions/build-fixtures/action.yaml.

Example fixture build properties file (--build-name=eip7692):

; File describing fixture build properties

[fixtures]
timestamp = 2024-06-13T14:35:59.172865
build = eip7692
commit = 5a91d041d2081f36cc12bbe6f5b896651655fab5
command_line_args = fill --output=fixtures_eip7692.tar.gz tests/berlin/ -k not Paris and not hive --build-name=eip7692

[packages]
pytest = 7.4.4
pluggy = 1.5.0

[plugins]
cov = 4.1.0
xdist = 3.6.1
metadata = 3.1.1
html = 4.1.1

[tools]
t8n = evm version 1.14.1-unstable-d6e91e2e-20240507
solc = 0.8.24+commit.e11b9ed9.Linux.gpp

[environment]
python = 3.10.12
platform = Linux-5.15.0-112-generic-x86_64-with-glibc2.35

🔗 Related Issues

None

✅ Checklist

danceratopz commented 2 weeks ago

Tested with this Github Actions Workflow: https://github.com/danceratopz/execution-spec-tests/actions/runs/9500351128/job/26183213458

Example artifact ("stable"): https://github.com/danceratopz/execution-spec-tests/actions/runs/9500351128/artifacts/1598139447 which has the fixtures.properties:

[fixtures]
timestamp = 2024-06-13T13:16:17.589569
build = stable
ref = refs/heads/feat/directly-create-release-tarball
commit = 097e5edc73b649c9eac82ec3dfbbe08ce31628e9
command_line_args = fill -n auto --evm-bin=evm --output=fixtures_stable.tar.gz --build-name stable

[packages]
pytest = 7.4.4
pluggy = 1.5.0

[plugins]
html = 4.1.1
metadata = 3.1.1
xdist = 3.6.1

[tools]
t8n = evm version 1.14.6-unstable-86150af2-20240612
solc = 0.8.21+commit.d9974bed.Linux.gpp

[environment]
python = 3.11.9
platform = Linux-6.5.0-1021-azure-x86_64-with-glibc2.35
ci = true
danceratopz commented 2 weeks ago

Renamed the "properties" file from fixtures.properties to fixtures.ini: We're using the python standard library's configparser module to write this file which creates a file in the .ini file format (this output is not valid .properties format due to the sections, e.g., [fixtures]).

This file is mainly intended for internal EEST use (consume) but might be useful to other people. Ini format should be readable by most languages, but feel free to change it to JSON (@marioevz, @spencer-tb).

pdobacz commented 1 week ago

@marioevz I've noticed that this might have changed the layout of the release artifact - inside the .tar.gz file one now has the intermediate dir with a suffix, i.e. fixtures_eip7692 dir, instead of just fixtures. Was this change intended?

Unless I've misconfigured sth on my end, you can take a look at such release here - fixtures_eip7692 inside. Compare with a previous one - fixtures inside.

marioevz commented 1 week ago

@pdobacz I think you're right, let me dig into the changes and see how we can fix it, thanks for pointing this out!