search

zyachel / quetre

A libre front-end for Quora
https://quetre.iket.me
GNU Affero General Public License v3.0
439 stars 31 forks source link

[Feature request] Store scraped data #99

Open KaKi87 opened 1 year ago

KaKi87 commented 1 year ago

Hello,

It would be nice of Quetre to store scraped data in a database so that it doesn't ever need to scrape it again.

Here's what happened to me today :

  1. I visited a successfully loaaded Quetre page :
  2. Shared it with someone else ;
  3. Hours later, the same page returned 503 to the person I shared it with.

Thanks

zyachel commented 1 year ago

well, there is an option to cache responses using Redis.
In case of my instance, I have set the cache expiry time to one hour. I could increase that time if you want.

Caching for infinite time, however, would blow up RAM usage. Furthermore, it would be akin to keeping stale answers, which is not optimal. So, I don't think it's a sound idea.

if you're worried about a particular answer not existing in the future, you can always use archive.org and save the answer forever.

As for instances being rate limited, I'm exploring ways to minimise it.

KaKi87 commented 1 year ago

Why not caching on disk, of not with MySQL or Postgres then with SQLite for example, which would easily allow something like a month of cache ?

Thanks