loculus-project / loculus

An open-source software package to power microbial genomic databases
https://loculus.org
GNU Affero General Public License v3.0
37 stars 2 forks source link

We need to reduce the frequency at which we poll Github for curated data updates #3268

Open corneliusroemer opened 1 day ago

corneliusroemer commented 1 day ago

Github doesn't like how often we're polling it. We should hit github pages for this and not raw user content:

02:50:19    ERROR (trigger_submission_to_ena.py:  71) - Failed to retrieve file due to requests exception: 403 Client Error: Forbidden for url: https://raw.githubusercontent.com/pathoplexus/ena-submission/main/test/approved_ena_submission_list.json

I saw this in staging. It's also happening in production.

We currently hit Github every 2 minutes from that one deployment alone.