AO3 HTTP 429 Exception logging

[stranger-danger-zamu]

It's pretty easy to hit a HTTP 429 Status Code if you are grabbing a lot of fics on AO3 (>200). That's on me and is fine.

The issue is that it's just printed to stdout instead of stderr.

> fanficfare -i ao3-fics -p 2> errors
............................................................................................................................................................URL(https://archiveofourown.org/works/30528462) Failed: Exception (HTTP Error in FFF '429 Client Error: Too Many Requests for url: https://archiveofourown.org/works/30528462/navigate?view_adult=true'(429)). Run URL individually for more detail.
URL(https://archiveofourown.org/works/36258934) Failed: Exception (HTTP Error in FFF '429 Client Error: Too Many Requests for url: https://archiveofourown.org/works/36258934/navigate?view_adult=true'(429)). Run URL individually for more detail.
URL(https://archiveofourown.org/works/23538241) Failed: Exception (HTTP Error in FFF '429 Client Error: Too Many Requests for url: https://archiveofourown.org/works/23538241/navigate?view_adult=true'(429)). Run URL individually for more detail.
URL(https://archiveofourown.org/works/36573415) Failed: Exception (HTTP Error in FFF '429 Client Error: Too Many Requests for url: https://archiveofourown.org/works/36573415/navigate?view_adult=true'(429)). Run URL individually for more detail.
URL(https://archiveofourown.org/works/32391694) Failed: Exception (HTTP Error in FFF '429 Client Error: Too Many Requests for url: https://archiveofourown.org/works/32391694/navigate?view_adult=true'(429)). Run URL individually for more detail.
.....................................................................................................

Is there a way to get the exceptions to be logged into stderr instead of stdout? Or better yet a configuration option to output failed links/URLs to a different file (ie. a dead letter queue)?

[JimmXinu]

Not presently.

Somebody wanted color coded output a while back, so 'warnings' are blue and 'failures' red with --color.

warn and fail could optionally instead write to stderr, but it looks like it's yet another py2/py3 incompatibility point. Code at https://github.com/JimmXinu/FanFicFare/blob/main/fanficfare/cli.py#L219

You also get a lot more output on stderr with --debug/-d.

Please don't download hundreds of stories at a time. We used to be able to download from ffnet without huge hassles. I'd hate to see AO3 go the same way.

[stranger-danger-zamu]

Ah, that's a shame. I guess I'll be working around with some pipes for filtering.

Please don't download hundreds of stories at a time. We used to be able to download from ffnet without huge hassles. I'd hate to see AO3 go the same way.

Ah, this is just a initial data ingest. After this initial load is done, it'll be all daily deltas going forward (which has been less than 5 a day).

Oh well, thanks for the quick response.