search

mediawiki-client-tools / mediawiki-dump-generator

Python 3 tools for downloading and preserving wikis
https://github.com/mediawiki-client-tools/mediawiki-scraper
GNU General Public License v3.0
93 stars 14 forks source link

Run tests and set up CI action #7

Open elsiehupp opened 2 years ago

elsiehupp commented 2 years ago

Right now tests don't run automatically, and there aren't any instructions on how to run them. The following should happen:

Additionally, we should do the following with the tests:

robkam commented 1 year ago

Setting up a MediaWiki test server at Miraheze would be easy.

yzqzss commented 1 year ago

I'm installing various versions of MediaWiki.
From 1.16.5(released on 2011-05-05) to 1.39.X(current LTS release).

Currently, these versions are already installed:

I will install 1.27 LTS, 1.31 LTS, 1.35 LTS and 1.39 LTS soon.

For testing Unicode and XML compatibility, some wikis's language are set to Chinese, and some sites have special characters and Emoji in the title.

Could someone help me insert test data into wikis once set up? Upload some articles and files, the title and content should be as weird as possible, xD.

We can also install wikis with different "features" like http 301 https, SSL expired, Special:Export disabled, some APIs disabled, wiki with WAF, etc.

yzqzss commented 1 year ago

OK:

elsiehupp commented 1 year ago

I had originally intended the prepare-for-publication branch to serve this purpose, and I did make some progress on getting that branch to pass the tests, but I got sidetracked on other things, and I got a bit overwhelmed, so that branch is increasingly out of date.

One of the other things I was working on with the prepare-for-publication branch was switching from using the same config dictionary for every function call and instead using typed parameters, with mypy type checking to help with debugging.

While I do still think that mypy type checking would be helpful, I was trying to do way too much all at once, which is probably a big part of why I got overwhelmed and stalled in the process...

As far as salvaging the work I did on the prepare-for-publication branch goes, one place to start could be copying over the tests themselves, since the issue was partially getting the tests themselves to work, with the application code not being the main problem.

It would probably also help to create one or more additional branches, so that any salvageable changes could be brought over piecemeal.

How does this sound? I can try and take a stab at it myself, but as I've mentioned I'm super not on top of things (hence why I'm glad you all have been working together so well without me).

WolfgangFahl commented 1 year ago

See https://github.com/WolfgangFahl/pymediawikidocker where this is fully automated and where the extensions to be used may be specified while at so usecases revolving around problems with extensions might be tested.

See https://wiki.bitplan.com/index.php/Pymediawikidocker#Examples for examples

See https://github.com/WolfgangFahl/pymediawikidocker/actions for the CI usage Defined in https://github.com/WolfgangFahl/pymediawikidocker/blob/main/.github/workflows/build.yml Which simply uses

scripts/install
scripts/test

as the CI commands - which starts python commands to run the whole show using the excellent https://github.com/gabrieldemarmiesse/python-on-whales library

elsiehupp commented 1 year ago

Rereading the initial checklist, it looks like the main thing left to do is adding instructions on how to run the tests locally. Is there anything else I'm missing?

robkam commented 4 months ago

Save The Web Project's MediaWiki test sites (http://group0.mediawiki.demo.save-web.org/ and so on) are down.