Flickr-Foundation / flickypedia

A tool to copy CC-licensed images from Flickr to Wikimedia Commons
https://www.flickr.org/tools/flickypedia/
Apache License 2.0
8 stars 2 forks source link

Use the `lastrevid` parameter to speed up the Backfillr Bot #425

Open alexwlchan opened 6 months ago

alexwlchan commented 6 months ago

The way I'm making the edits in the Backfillr Bot is not optimal:

When it processes a file, it fetches the current SDC using the API, works out the diff, then posts that back to WMC. There are two problems with this approach:

If I were to use the SDC from the snapshot with the baserevid parameter, I'd be able to cut out a significant number of API calls, and only call the "get SDC" API when the SDC has changed since the snapshot was taken.

alexwlchan commented 6 months ago

This is a nice idea in theory, but I can't work out what, if any, conflict detection it's doing, at least when adding new statements.

The problem is that if we write a new statement, it doesn't have an ID, and Wikimedia Commons will happily add that as a new statement w/o checking to see if it conflicts with or duplicates an existing statement. Grr.