pytest-dev / pytest-randomly

:game_die: Pytest plugin to randomly order tests and control random.seed
MIT License
630 stars 30 forks source link

feature: add the option ``--randomly-seed-per-test`` to use a different seed for each test #617

Open brycedrennan opened 7 months ago

brycedrennan commented 7 months ago

Resolves #600

I'm tempted to make it the default behavior but this is safer.

brycedrennan commented 7 months ago

It's done but there is an inconsistency with how seeding works with faker. I'm not sure what would be least surprising for someone given the docs from faker. https://github.com/pytest-dev/pytest-randomly/blob/8b80740efd50c2785c8471cefba5fb7448e410f7/tests/test_pytest_randomly.py#L674-L692

https://faker.readthedocs.io/en/master/pytest-fixtures.html#seeding-configuration

alexandrupatrascu commented 6 months ago

hi folks, any chance this gets merged and released soon? reason why I'm asking is that I'm running pytest-rerunfailures alongside pytest-randomly and the re-runs use the same seed ID, causing the tests to re-use the same test data and not generate new inputs, hence they keep on failing.

thank you for the help!

brycedrennan commented 6 months ago

Sounds like you should change your test to either work with the same seed of data or change the data generator to not generate invalid data? I don't believe the changes proposed here would even help with your situation either way.

tamird commented 5 months ago

This would be very useful indeed.

In the meantime I worked around this in my own project using

@pytest.fixture(autouse=True)
# autouse fixtures with leading underscores are ignored. See https://github.com/pytest-dev/pytest/issues/12404.
def emulate_pytest_randomly_pull_request_617(request: pytest.FixtureRequest) -> None:  # noqa: PT004
    # pytest-randomly sets `randomly_seed` to an integer at pytest_configure time. See
    # https://github.com/pytest-dev/pytest-randomly/blob/8a3a241/src/pytest_randomly/__init__.py#L138.
    #
    # We include that integer to preserve the behavior of an explicit `--randomly-seed <int>` flag
    # passed on the command line while also giving each test a unique seed.
    request.config.option.randomly_seed += int(hashlib.sha512(request.node.nodeid.encode()).hexdigest(), base=16)