wikiteam3 v4 release

[yzqzss]

Fixes

• fix: --xmlapiexport infinite loop on page with missing revision
• fix: config is not passed to log_error()
• fix: parentid should be a positive integer
• fix: truncated API response for "allrevisions" causes infinite loop (https://github.com/mediawiki-client-tools/mediawiki-scraper/issues/166)
• fix: yield bug in red_titles()
• fix: Unable to traverse Special:Filelist
• fix: can't find API URL from HTML (new MW versions)
• ...

feats

• feat: ensure params["limit"] is int and >= 2
• feat: save timestamp metadata to images.txt
• feat: cli: --disable-image-verify (https://github.com/mediawiki-client-tools/mediawiki-scraper/issues/170)
• feat: cli: add --image-timestamp-interval to download images uploaded in the given time interval (https://github.com/mediawiki-client-tools/mediawiki-scraper/issues/151)
• feat: cli: add --user-agent to custom UA
• feat: --ia-wbm-booster Incremental image dump powered by web.archive.org (experimental)
• feat: write .mark file after dumped
• feat: image: dynamic load config.delay
• feat: If there are archives on IA in the last year, exit
• feat: a trick to get original file for fandom.com
• feat: more robust convert URLs to prefix filename
• ...

refactor

countless ...

drop legacy code

• gui.py
• wikia.py
• launcher.py
• ...

Special:Export

• Do not strip sha1 tags from XML

Breaking changes

• image: change the magic NULL string from False to null
• drop .desc xmlfile support

Drop launcher

Shifts compression responsibilities from the launcher to the uploader.

dependencies

• remove: all useless dependencies
• add: python-slugify
• add (uploader): zstd

refactor uploader

• Use zstd to compress .xml for better compression speed, compression ratio, decompression speed, streaming decompression. （with -{19..22} --ultra --long=31)
• Instead of packaging the whole wikidump into wikidump&history.7z, we upload each file separately for better parsing and management in the future.
• By default, the compression tasks of each process will be queued to avoid overload/OOM. (socket lock) (--parallel to disable it)
• Upload logo. (optional)
• Stop converting language codes to human readable formats.
• IA ITEM medadata: add 'upload-state' and 'scanner'.
• ...

---

https://pypi.org/project/wikiteam3/

[yzqzss]

Point of Conflict.

https://github.com/mediawiki-client-tools/mediawiki-scraper/commit/eeab93a4888491dcf6d818b93e13968b4a27d152

[randomnetcat]

Several things:

• This is a truly massive PR. Could you consider splitting it up into smaller ones? I wouldn't even know where to start reviewing this.
• Just glancing at the changes, you use the "saveweb" project in URLs several times, which is not this project.
• From reading the description, it sounds like you're breaking format compatibility with upstream wikiteam and older versions of this project. That's not a good idea, since people are depending on that (at least I am in some of my code).

[elsiehupp]

This is the sort of thing where I will defer to everyone else.

In general I would say that anything breaking backwards compatibility should be dependent on implementation of build-versioning that would allow users to reliably target older versions.

Right now this repository does not even have version-tagged GitHub builds, let alone versioned PyPI builds, so at this point breaking backwards compatibility is a no-go.

As for drastically refactoring the code… that's fine, and probably for the better, as long as there is build-versioning in place to protect existing users.

[elsiehupp]

Regarding format compatibility: I think introducing a new default format is fine as long as the existing upstream format continues to be supported alongside it for a substantial "bridge" period, with (a) an ability to convert existing dumps to the new format, and (b) there are strong "deprecation" nudges encouraging users to migrate.

Refactoring (and abstracting much of the backend) could of course facilitate this, hence why I'm supportive of refactoring more generally.

Basically what I'm saying is that introducing a new data format should be dependent on first establishing a stable public API for the backend, which currently does not exist.

[robkam]

This is too complex for my level of comprehension - except that this PR for wikiteam3 is not in that repository.

[elsiehupp]

Here's my thought…

@yzqzss why don't you open a new Pull Request from an earlier commit on this branch? (I think you have to create a branch from that commit in order to do so.)

This would be much easier to approach if it wasn't a gigantic total total rewrite all at once, and breaking it into chunks this way would help.

If you're not interested in doing thing, to be fair, we could try and do so ourselves, but you're more familiar with your own code than we are.