issues
search
Police-Data-Accessibility-Project
/
automatic-archives
MIT License
1
stars
1
forks
source link
Mc fix archives bug
#19
Closed
maxachis
closed
1 week ago
maxachis
commented
1 week ago
Fixes
None. Bug fixes and QoL improvements.
Description
Fixed bugs preventing Automatic Archives from Automatically Archiving.
Refactored code to be generally more comprehensible.
Added enhancements to improve visibility of actions in console.
Removed the process whereby multiple workers (up to 100) were archiving data in parallel. Because the internet archives will
temporarily block an IP address if more than 15 requests are made within a minute
, additional workers do not improve performance and merely add complexity.
Testing
Run
cache_url.py
with necessary environment variables.
Performance
Some additional overhead due to refactoring
Jobs will take longer, in part because archive action must be throttled to prevent internet archive from
temporarily blocking the IP address
.
Docs
Not applicable.
Fixes
Description
Testing
cache_url.py
with necessary environment variables.Performance
Docs