Closed aschoelling closed 1 year ago
I just did some more tests, and I think the error is somehow related to Gitlab. Rewrote the tests (minor changes only) to run standalone from a BASH-Script, the results are OK, no passing page with more than 5% difference anymore. I think this issue can be closed for now, unless someone has an idea on what is happening.
Thanks for doing more research and working around the issue.
Maybe you could post your solution so others can learn from your experience.
Cheers.
I'll see to it when my current workload gives me some room for it, for now I'm content with the knowledge that it works locally. Will update this issue when I got new information.
I use a sitemap.xml based Gitlab CI/CD Job, and I regularly get results tests declared ok with extreme differences between MisMatchThreshold (5% currently) and reported difference (I've seen values significantly higher than 50% being declared OK.
Configuration File for TEST:
The variables are filled in basic.js from GITLAB variables:
urls.json is created by Gitlab Job and works, I can see all pages in the generated report