Caching

[oskarrough]

In order to avoid reaching the Discogs API rate limits we should avoid more requests than necessary.

For instance, if you go to a release, then back to the homepage and finally visit the same release again, a second uneccesary GET request is made.

• can we avoid calling it twice? who is responsible? discogs? ember data?

Another idea is to sync the the store to a local database like pouchdb, localstorage or whatever they're called these days. That could be cool, too. First look in local db, is it there, yes return, otherwise fetch.

nicetohave

[oskarrough]

Seems this is exactly what https://www.emberjs.com/api/ember-data/3.0/classes/DS.RESTAdapter/methods/shouldBackgroundReloadAll?anchor=shouldBackgroundReloadRecord is for.

[4www]

True that releases, labels and artists are not data that must be updated too often on Discogs. I bet even search results could be cached.

It somehow sounds appealing, but caching mechanisms are hard, to manage and reason about. I would not focus too much on this part for now, but definitely keep it in mid if we hit a limit that we cannot bypass.

Eventually soon. We could also look into how to get accepted for more requests. Maybe implementing user login with oauth could help. That feature would also be interesting to get the logged in releases in a tab.

All this said, I think that not reloading a model too much when it has been loaded in the current session is a good idea, and might even be a one liner in a config (¡hoôope!).

[oskarrough]

All this said, I think that not reloading a model too much when it has been loaded in the current session is a good idea, and might even be a one liner in a config (¡hoôope!).

Yes! See https://github.com/internet4000/explorer-discogs/pull/15

But even with this in place, if you visit this page you'll reach the API limit immediately (try refreshing). It makes thirty Discogs requests because of all the relationships. Perhaps we need to make sure they're not loaded automatically.

https://deploy-preview-14--explorer-discogs.netlify.com/artists/203017

Requests are throttled by the server by source IP to 60 per minute for authenticated requests, and 25 per minute for unauthenticated requests, with some exceptions.

Some even go all the way…… https://github.com/hendriks73/coxy

[4www]

rate limiting is max 60 queries for authed queries. https://www.discogs.com/developers/#page:home,header:home-rate-limiting

So since we're using the same API key you and I, not a lot.

I think maybe with oauth, so one user has their own limit of 60req/sec.

Also we should not load all relaed data automatically, but have an expand button or better, a sub route :]

[oskarrough]

Af few examples of solving this with node.js+redis

https://www.compose.com/articles/api-caching-with-redis-and-nodejs/ https://community.risingstack.com/redis-node-js-introduction-to-caching/ https://medium.com/ibm-watson-data-lab/building-an-api-cache-with-redis-and-node-3fbad5675d5d

[oskarrough]

Proxy API test

https://glitch.com/edit/#!/edapi?path=server.js:45:17 https://edapi.glitch.com/

[oskarrough]

Fixed by adding the proxy #21