Website needs a test suite

[pnijjar]

Probably all good software should have a test suite. I do not have a good sense of how to do this for websites, although I have heard buzzwords. Can somebody help set up the infrastructure for this, and then help me get started on how to test things and what to test?

[qedi-r]

I currently have my fork integrated into Travis-CI, which will automatically run on every checkin. Right now, all it's configured to run is htmlproofer, to do image, resource and link verification, but even that has found a few small issues I will create separate PRs for. We should be able to easily hang additional things off of that pipeline.

[qedi-r]

See https://travis-ci.org/qedi-r/WRvotes/jobs/419085787 for example.

[pnijjar]

Wow. This is a great start. Looking at the PR, I have some dumb newbie questions:

• Do we have to specify the build process for our site manually? Why did you need to create all of the Ward pages? Was it because of the symlinks? Does this ward list need to be checked into git, or can the script generate it each time Travis CI is run?
• How is this test suite hosted? Do we need our own server? Is it integrated from the travis-ci website? If so, does it cost money?
• Will we be able to ignore certain failures? (eg the fact that some candidate websites fail). Better yet, could we ignore failures for a time window and then stop ignoring them?
• Will we be able to have Travis CI rebuild the site using some custom test CSV files? I have noticed that most of the edge cases for the site can be made by specifying weird things in the CSVs and then checking whether certain things happen/don't happen. Is this hard to do with Travis, or easy?
• How many of the config files in your PR can be put into their own subdirectory? It would be nice not to pollute the top level. If we could drop some of the config into either /travis-ci or /scripts/travis-ci then that would be ideal.

Sorry if these are too many questions, or too newbie-ish.

[qedi-r]

1) The build scripts for our site just mirrors what github pages does, so that we can test it. Doesn't change it at all, and yes, the symlinks were wierd when going from host to test docker container, so I just generated them in the build. They could remain checked in and I could change the travis script to regenerate them, probably.

2) Tests are run by Travis CI with their hardware. They offer free testing services for open source projects.

3) Yes, we can specify some websites to ignore. I haven't done that yet, though.

4) This is easy, and is possible for people to do even before they submit a PR. I can write something up for how to do that.

5) the travis and Gemfiles need to stay at the top level, I think. The other three could be moved.

[pnijjar]

Would it be okay if you moved the files you can and then reroll? Then I will merge.

[qedi-r]

Sure. I'll see if I can add the excludes for number 3 above as well.

[qedi-r]

Okay. I have added a few failing campaign sites to be ignored, and moved the scripts into the scripts directory. I've also removed the ward symlink generation script. I'll submit a future PR to add some documentation on how others can use travis in their own forks to validate their changes are okay.