Problem: the script quit on me after ~100-150 pages of pulling IDs.
Fixes:
Load existing seen_ids from the CSV output file provided. Means we will add to the existing file and avoid duplicate IDs.
Make seen_ids a set. Minor improvement, but it's mostly look up, so a set is better.
Retries on 429. A 429 Retry Later causes the script to think there are no more works and subsequently quit out. This retries with a 5s delay until we get a non-429 response.
Problem: the script quit on me after ~100-150 pages of pulling IDs.
Fixes:
seen_idsfrom the CSV output file provided. Means we will add to the existing file and avoid duplicate IDs.seen_idsaset. Minor improvement, but it's mostly look up, so a set is better.429 Retry Latercauses the script to think there are no more works and subsequently quit out. This retries with a 5s delay until we get a non-429 response.