Implement regression tests machinery in the new site

[joaquintides]

Regression tests in the old site:

• https://www.boost.org/development/tests/develop/developer/summary.html
• https://www.boost.org/development/tests/develop/developer/issues.html
• https://www.boost.org/development/tests/master/developer/summary.html
• https://www.boost.org/doc/libs/develop/

are constructed periodically out of regression test reports uploaded by external contributors (notably Tom Kent) to an FTP server at results.boost.org.

• Regression test runner and reporter used by contributors: https://github.com/boostorg/regression/tree/develop/reports
• Tool for building the results tables at boost.org: https://github.com/boostorg/regression/tree/develop/library_status

We want to keep this functionality in the new website, which will probably require setting up an FTP server to the same name as now and running library_status periodically. After this, we'll have to consider how to link the produced HTML pages from the new website (not in the scope of this feature request).

[sdarwin]

A minimum viable product of this feature could be:

• display pages such as https://www.boost.org/development/tests/develop/developer/summary.html
• don't depend on wowbagger (the current webserver) to do that.

I've launched a server instance https://regression.boost.io (in progress). See https://regression.boost.io/develop/developer/summary.html This should be able to work with the existing CircleCI jobs and the current FTP server, only applying small modifications.

@kennethreitz @julioest, consider these URLs on boost.org https://www.boost.org/development/tests/master/developer/summary.html https://www.boost.org/development/tests/master/developer/issues.html https://www.boost.org/development/tests/develop/developer/summary.html https://www.boost.org/development/tests/develop/developer/issues.html https://www.boost.org/development/tests/* https://www.boost.org/development/running_regression_tests.html https://www.boost.org/development/testing.html

A version of those files now reside here: https://regression.boost.io/master/developer/summary.html https://regression.boost.io/master/developer/issues.html https://regression.boost.io/develop/developer/summary.html https://regression.boost.io/develop/developer/issues.html https://regression.boost.io/* https://regression.boost.io/running_regression_tests.html https://regression.boost.io/index.html

When visitors go to https://boost.io/development/tests/ could boost.io retrieve content from the back-end https://regression.boost.io and display it? Similar to the way profile photos are stored in S3 and then proxied to the browser.

You may like to customize the pages https://regression.boost.io/running_regression_tests.html and https://regression.boost.io/index.html further. Here are options.

1. They are plain HTML web pages. View-source in the browser, copy, modify the source HTML, and send me a copy. OR,
2. Decide on a location in this website-v2 repo. Put the files there. Modify them. Let me know and I will update the regression server.

What else is on the server..

• css/style-v2/ directory originating from https://github.com/boostorg/website/tree/master/style-v2
• images/favicon.ico
• images/space.png

So, the root directory on that web server contains:

css/
develop/
images/
index.html
master/
running_regression_tests.html

[joaquintides]

@sdarwin are you considering the fact that we'll need to host the FTP server (with the same name and access structure) so as not to disrupt the reporting process when the new website replaces the old one?

[grafikrobot]

@sdarwin & @PeterTurcan

The pages..

https://regression.boost.io/running_regression_tests.html https://regression.boost.io/index.html

..should be moved to pages in https://www.boost.io/doc/contributor-guide/testing/intro.html

[grafikrobot]

See https://github.com/boostorg/website-v2-docs/issues/218 for migrating the two info pages.

[sdarwin]

we'll need to host the FTP server (with the same name

(A restriction of DNS is you cannot host multiple servers simultaneously with the same name.)
Let's consider how the system currently works:

1. Tom uploads files to an ftp server
2. wowbagger has a cron to check on updated ftp files, and then triggers CircleCI
3. CircleCI downloads reports, processes, uploads a large zip file to wowbagger with all the results.

The way it could work in the new system:

1. Identical. No change or need to contact Tom.
2. add a cron on the new server regression.boost.io to "check" the files on ftp. However, one cron is enough. They are the same. So it's commented out at the moment.
3. CircleCI runs. Modify the CircleCI job so it uploads results to regression.boost.io also. To both servers. Done.

[joaquintides]

For the record, the FTP server at results.bost.org is detached from either the old or the new website and will continue to work after launch, so there's no disruption point to worry about here.