Release DumpGenerator 2.0 (tracking)

[GoogleCodeExporter]

This is a tracking bug for getting DumpGenerator 2.0 released. Code is 
currently in the works and is intended to be a rewrite of the original version 
(supposedly called 1.0).

Outline of what features are going to be available in 2.0: 
https://meta.wikimedia.org/wiki/WikiTeam/Dumpgenerator_rewrite

Original issue reported on code.google.com by ad...@alphacorp.tk on 12 Nov 2013 at 6:35

• Blocked on: #17, #29, #33, #34, #41, #44, #71

[GoogleCodeExporter]

Original comment by ad...@alphacorp.tk on 12 Nov 2013 at 6:38

• Now blocked on: #17, #29, #33, #34, #41, #44, #71

[GoogleCodeExporter]

Wow, are you really working on this? :D

The original version was too hacky and incomplete to be >= 1, it was 0.1. :) I 
think this would deserve to be the first actual release, 1.0. A critical 
feature should be elegance and compatibility achieved by relying on robust 
libraries for the actual interaction with the wikis.
So what libraries are you going to use? Betacommand suggested pywikibot, but 
mwclient and wikitools are options too. 
https://www.mediawiki.org/wiki/API:Client_code#Python Factors for the choices 
are 1) cross-platform compatibility and ease of install (including size), 2) 
how easy it is to override some critical methods like the user agent, delays, 
reaction to some status codes.
For the remaining HTTP requests outside the libraries, python-requests.org 
seems a god bet.

Original comment by nemow...@gmail.com on 12 Nov 2013 at 7:43

[GoogleCodeExporter]

My goal is to not depend on too much third-party libraries and use those solely 
within Python itself. Imagine being someone trying to take part in the 
archiving, only to realise that you have to download and configure things that 
you never heard of, and it can get quite irritating. :)

So, cross-platform compatibility is my highest priority here. Hopefully this is 
the first step towards getting "normal" Windows users to just download and run 
it without problems (and possible headaches).

Of course, the code that I currently have is still quite tailored for 
bleeding-edge MediaWiki. As I get the main features working, support for older 
releases will soon be added for it.

I did reveal a little bit about one feature that this new DumpGenerator will 
have: Auto-updating of the script! The revnum.json file in SVN right now is 
intended to be the control file to make (or possibly, force) people update to 
the latest version and ensure that they don't encounter bugs.

Don't expect anything ready too soon though, I am still trying to make the 
basic functionality work first before adding those that are desired. One thing 
is for sure: We are moving away from re.search and other regular expressions 
and using proper libraries.

Once it is actually usable, I will put it in SVN so that everyone can help in 
fixing the final issues before the final release.

P.S. There might be some bias towards Dumps on demand in terms of some 
functionalities though. :D

Original comment by ad...@alphacorp.tk on 12 Nov 2013 at 8:32

[GoogleCodeExporter]

No longer going to work on this anymore.

Original comment by ad...@alphacorp.tk on 23 Nov 2013 at 1:06

• Changed state: Invalid