search

blekhmanlab / rxivist

API providing access to papers and authors scraped from biorxiv.org
https://rxivist.org
GNU Affero General Public License v3.0
59 stars 11 forks source link

Write cache-warming script to crawl the site #206

Closed rabdill closed 5 years ago

rabdill commented 5 years ago

Prepare for potentially big traffic by making sure Cloudfront is configured conservatively and has stored as many pages as possible somewhere

rabdill commented 5 years ago

https://github.com/rabdill/rxivist/commit/90160584ce7bba2a17450a422d3f1bf01f32f56a