search

mozilla-magnet / magnet-metadata-service

A service that returns metadata for given URLs
Other
8 stars 10 forks source link

Caching system #17

Closed arcturus closed 8 years ago

arcturus commented 8 years ago

To speed up analysis we could add a caching system, there is a branch now adding this, but it's a bit obsolete.

wilsonpage commented 8 years ago

Do you think we could just use nginx for this? 😛

Might just be a few lines. Although if not dynamic, we may not be able to honour the cache headers returned from the source.

arcturus commented 8 years ago

Actually, I think that honoring the cache headers is the most important part.

Is the only way developers setting up beacons will have to handle the refresh of their creations.

samgiles commented 8 years ago

We could use Varnish if we don't use batching - we could even use it with a Fastly trial for London demo

arcturus commented 8 years ago

In my mind I had this idea of not having to use any http cache. But a custom cache that stores the result of the parsing.

Something like redis, with key the url and content the result of the parsing, taking into account that you can use records with ttl.

What do you think @samgiles ?