Bump version to 0.5.0-alpha with some improvements (Part 2)

[yzqzss]

• pass requests session to mwclient (https://github.com/WikiTeam/wikiteam/pull/441, 405b32ac9612923f85d5a4da237f2b807ab49ab8)

  FIX: #40 (not tested yet)

• FIX: Send/Recive twice download requests to the image URL by deleting duplicate code block.(a329903faa526176611bc43c93538ebd1c179b8c)
• add to_stdout parameter to logerror() for printing errors to stdout (0ce1d8820e6013807d0e187941047e55c4ea730f)
• FIX #23: index url computation fails on some wikis (94e62691c11fb301988756eb8f9d6e24c7eab536)
• make requests.session to use --retries value (0e317e213f6ca55964a255f9a9437086ebffa6af)
• remove redundant retry logic (c4c853f508f46ee44e4b138b21efe0090f5de254)

  requests.session already has built-in retry mechanism

• save more metadata(size, sha1) into images.txt (30176d556311179873842da034448b58b674e302)
• introduce sha1file() (e8adcbef151693dc6a2d2e1c3a1c10d74517f933)
• feat (image dump): (5edd3ace52035985926e0837f75a7555058904d6)
  • validate image size after requests fetched it: if len(r.content) == size
  • show progress and prrint more info
  • better resume:

    also FIX: #15 Improved the resume mechanism. First check whether the file and file.desc exist, and then check whether the size and sha1 of the file correspond to images.txt. If any check fails, the file and the .desc of the file will be downloaded again. If all pass, the download of this file is skipped. You can even delete random pictures and .desc files and try to resume again.

  • bump version to 0.5.0-alpha:

    because the format of images.txt has changed.

  • pre-work for incremental image dump:

    51

  • remove start param from generatorImageDump()
  • the images resume mechanism has changed. we don't need start for resuming anymore.
• other minor improvements:
  • Change UA
  • more error log
  • ...

[randomnetcat]

It would probably be better to split this into smaller PRs?

Also, the change to the on-disk format to make it inconsistent with the original wikiteam tools seems perhaps not ideal (though I don't know for sure whether that's already been done separately in this fork).

[yzqzss]

It would probably be better to split this into smaller PRs?

OK, I'll do that later.

Also, the change to the on-disk format to make it inconsistent with the original wikiteam tools seems perhaps not ideal (though I don't know for sure whether that's already been done separately in this fork).

Writing size and sha1 (they are from API/index.php) to images.txt is necessary to make image/file downloads more stable, to support checking file integrity and to support incremental downloads (incremental downloads are a later work in progress).

images.txt: before: FileName\tFileURL\tUploader after: FileName\tFileURL\tUploader\tSize\tSha1

wikiteam3 seems to have made some formatting changes, such as changing config.txt to config.json. This PR change prevented wikiteam3 from running --resume --images on <0.5.0-alpha Dump, but I think it was worth it, after all the previous version even saved 404/502/429 etc. error response as normal files...

[elsiehupp]

wikiteam3 seems to have made some formatting changes, such as changing config.txt to config.json. This PR change prevented wikiteam3 from running --resume --images on <0.5.0-alpha Dump, but I think it was worth it, after all the previous version even saved 404/502/429 etc. error response as normal files...

IIRC the reason I changed the config file to JSON is that using a standardized, well-supported configuration format like JSON simplifies the client code and makes it easier to debug. I don't think I changed much other than having it use JSON, and Python JSON just serializes and de-serializes the config dictionary as-is.

[elsiehupp]

Regarding versioning: version numbers only really make a difference with dependency management, once we publish this on PyPI. (The README currently instructs users to force-overwrite the install, making version increments irrelevant.)

I originally intended Issue 7 and the prepare-for-publication branch to serve the purpose of getting things ready for PyPI, though I got a bit sidetracked along the way, and the branch is now wildly out of date.

Since PyPI distribution would make Wikiteam3 useful to a lot more people, I do think it's a good medium-term goal, if you'd like to discuss it on Issue 7.

[yzqzss]

It would probably be better to split this into smaller PRs?

I've split this PR: #66 #67 #68 #69 #70
30176d5 and 5edd3ac are dependent on them and need to wait for these PRs to be merged before making new PRs for both of them.

[robkam]

PRs https://github.com/elsiehupp/wikiteam3/pull/66 https://github.com/elsiehupp/wikiteam3/pull/67 https://github.com/elsiehupp/wikiteam3/pull/68 https://github.com/elsiehupp/wikiteam3/pull/69 https://github.com/elsiehupp/wikiteam3/pull/70 now merged.

[yzqzss]

30176d5 and 5edd3ac are dependent on them and need to wait for these PRs to be merged before making new PRs for both of them.

88 created.