Improve feed polling efficiency

snarfed / bridgy-fed

🌉 A bridge between decentralized social network protocols

https://fed.brid.gy/

Creative Commons Zero v1.0 Universal

489 stars 28 forks source link

Improve feed polling efficiency #791

Open snarfed opened 5 months ago

snarfed commented 5 months ago

RSS/Atom feed polling looks pretty wasteful with datastore queries and reads right now. It launched on 1/9, and you can see the cost increase pretty visibly. Need to figure out what's going on and tighten it up.

snarfed commented 5 months ago

Added a short circuit on seen ids and ETag/Last-Modified based HTTP requests. Not seeing as big an improvement in cost as I'd expect so far. Will see.

snarfed commented 1 week ago

related: #1149