scrapinghub / spidermon

Scrapy Extension for monitoring spiders execution.
https://spidermon.readthedocs.io
BSD 3-Clause "New" or "Revised" License
528 stars 96 forks source link

fix flaky test on test_validators_schematics.py::test_choices #322

Closed lonly7star closed 1 year ago

lonly7star commented 2 years ago

This PR aims to fix the flaky test on test_validators_schematics.py::test_choices so the test could pass single run, multiple test run, and random test run. I'm doing this for a course project as a practice.

The result

The test won't pass on the single run with pytest, the following error raised assert (False, {'a':...in choices']}) == (False, {'a':...[1, 2, 3].']}) At index 1 diff: {'a': ['Value not in choices'], 'b': ['Value not in choices']} != {'a': ["Value must be one of ['a', 'b']."], 'b': ['Value must be one of [1, 2, 3].']}

Steps to reproduce the issue

  1. run the test file with pytest test_validators_schematics.py

Issue of the code

After examining the code I suspect the passed changes on contrib::Validation::Validator change how the error message is translated by contrib::validation::schematics::translator.py, which now raised a more detailed error message on VALUE_NOT_IN_CHOICES that caused an inconsistency on the testing error message.

Proposed solution

The convenience way is to changes the expect testing error message from {'a': ['Value not in choices'], 'b': ['Value not in choices']} to {'a': ["Value must be one of ['a', 'b']."], 'b': ['Value must be one of [1, 2, 3].']}

VMRuiz commented 1 year ago

Schematics is not longer supported