search

qt4cg / qt4tests

QT4 tests
https://qt4cg.org/
3 stars 7 forks source link

Isolate http(s): and other web resources in the test suite #140

Open ndw opened 2 months ago

ndw commented 2 months ago

We have tests that rely on external documents not under our control. For example, one of the tests for unparsed-text relies on the contents of https://www.w3.org/Consortium/mission.html which the W3C has recently removed.

All of the web resources we rely on should be stored in a local container that we control. For example, a Docker container running an Apache web server. Anything else leaves us endlessly at the whim of organizations outside our control.

michaelhkay commented 2 months ago

You're probably right; but it's also true that any attempt to make tests more predictable by isolating them from the vagaries of the real world mean that we're not exposing implementations to the vagaries of the real world.

In addition, I'm a bit concerned that you proposal blurs the boundary between what belongs in the test suite and what belongs in a test driver.

ndw commented 2 months ago

The unparsed-text-2002 test has nothing to do with the vagaries of the real world:

    <utf-8><xsl:value-of select="contains(unparsed-text('https://www.w3.org/Consortium/mission.html'),
                                 'long-term growth')"/></utf-8>
    <iso-8859-1><xsl:value-of select="contains(unparsed-text('https://www.w3.org/TR/1999/WD-font-19990902'),
                                 'Håkon Lie, W3C')"/></iso-8859-1>

At least nothing usefully to do with those vagaries. It's testing that the content returned by the first request is UTF-8 (I guess) and the second is ISO-8859-1. But we don't have any control over those resources. The first one is 404 now and if the W3C transcodes the second to UTF-8, then the test will pass uselessly.

If we ran our own server in a container, then the request could be to, for example http://localhost:3124/something and we could assure that something was returned in UTF-8 or ISO-8859-1 or anything else we wanted.

The XProc tests suite does this and it has worked very well. Previously, the tests were hosted at xproc.org which was managed by an organization sensitive to our needs (cough) but it was still problematic because sometimes the web doesn't work and if that causes a bunch of tests to fail in regression, that's frustrating and annoying. Ask me how I know!

This has nothing to do with the test driver it has to do with providing a test environment that's stable, reliable, and reproducible.