Optimize the number of X API calls for large media libraries

[jdaymude]

Description As described in the README under Troubleshooting and Gotchas ERROR 403, xbox2local calls the X API once for authentication, once per page of screenshot results, and once per page of game clip results each time it is run. For users with large media libraries on Xbox Live, this may exceed the 60 requests per hour allowed by X API's free tier. The suggested workaround of using --media_type to download screenshots and game clips in separate xbox2local calls (straddling over an hour reset) still may not solve the problem since some users may have over 60 pages of results for screenshots or game clips independently.

Since results pages are served in reverse-chronological order, a first improvement would stop xbox2local from querying result pages if it has reason to believe that all older pages are results it has already downloaded. This would require an extension of the download_history.json file, most likely. However, this does not solve the problem of first-time users with large media libraries; it only helps users who start with libraries small enough to be managed by the 60 requests per hour that may outgrow this limit over time.

Alternatively, with some research into how pagination and continuationToken information is structured, it may be possible to let the user provide a date to start downloading from, skipping unwanted calls. This is more amorphous though, and I'm not sure how it would actually work.

Can you help develop and maintain this feature? Yes.

[jdaymude]

This remains an open issue, though the outlook is more optimistic after #15 which updated the backend API from X API to OpenXBL. At the time of this writing, the free tier of OpenXBL allows 150 requests per hour, which is significantly more generous than the 60 req/hr allowed by the (now deprecated) free tier of X API.

The media entries are still paginated in reverse-chronological order (most recent first), which means that without some clever trick, first-time users will still have to access all pages (perhaps one media type at a time) with no optimizations from "skipping" pages (if such a thing is even possible).

If any optimizations are possible, they will likely depend on information that can be captured in a deeper history of metadata; see #16.

[jdaymude]

TL;DR: This issue can be closed since the 150 requests per hour limit on OpenXBL's free tier is sufficient for all reasonable cases.

In detail, xbox2local makes the following calls to OpenXBL API each time it's run:

1. One call to the /api/v2/dvr/screenshots endpoint per page of screenshot results, each of which has 100 screenshots.
2. One call to the /api/v2/dvr/gameclips endpoint per page of game clip results, each of which has 100 game clips.
3. One call to the /api/v2/dvr/gameclips/delete/{gameClipId} endpoint per deleted game clip.

As of #19, xbox2local must scan the entire set of screenshots and game clips on the Xbox network before proceeding with downloads. For this to exceed 150 requests/hour, the media library must contain at least 150 requests * 100 media per page = 15,000 combined screenshots and game clips. For the sake of worst case analysis, suppose all 15,000 media are 1MiB screenshots (an exaggerated lower bound), totaling ~15GiB of storage. Actual screenshots are ~3MiB in 1080p and >10MiB in 4K, and game clips are an order of magnitude larger. However, the Xbox network has a 10GiB SDR media storage quota. Thus, we can reasonably assume that OpenXBL's free tier (150 requests/hour) should be sufficient for any real media library.

xbox2local guarantees monotonic progress for game clip deletions; i.e., any successful deletion is tracked and will not need to be repeated in the future. It sends as many game clip deletion requests as it can before reaching the request limit, at which point it gracefully halts. It can be rerun after the next hour's request quota reset to continue requesting any deletions that did not succeed the first time around. (And, if you're impatient, you can of course do these deletions from the Captures app on your Xbox).