search

python / psf-salt

PSF infrastructure configuration
MIT License
115 stars 58 forks source link

Add automatic testing of the d.p.o redirects via Hurl #505

Closed AA-Turner closed 1 month ago

AA-Turner commented 1 month ago

Based on & includes commits from:

This is the PR that actually implements automatic testing. We have ~9,400 test cases for redirects generated from the regular expressions in the .conf. file based on the currently supported languages. Each test file 'spec' represents a different redirect rule or group of redirect rules. Hurl runs these in parallel, so all tests only take a second or so to run.

The only redirect rule I haven't implemented tests for is the "content-negotiation" rule as it is too dynamic.

A

jcamiel commented 1 week ago

Hi @AA-Turner maintainer of Hurl here. I'm a little blowed (and proud!) that Hurl have been chosen for this kind of tests for Python docs. Thanks for this!! I would like to know if something was missing in the language/tool when you say

The only redirect rule I haven't implemented tests for is the "content-negotiation" rule as it is too dynamic.

I'm interested in any feedbacks to see of we can improve some things...

AA-Turner commented 1 week ago

Thanks @jcamiel, it is very impressive that we run 9,810 tests in only 1.7 seconds!

The "content-negotiation" rule is thus:

https://github.com/python/psf-salt/blob/dd4033ebce678841b22a22f77b0dfc90e3450718/salt/docs/config/nginx.docs-redirects.conf#L177-L186

The way I wrote most other tests was reversing the regular expressions, to generate all valid inputs. Perhaps Hurl could do some form of property-based testing à la Hypothesis/QuickCheck? Though you may have better ideas here!

A

jcamiel commented 1 week ago

Thanks for the feedback, I'll try to improve things. Hurl's speed is really possible because of curl, can't take credit of it!