sensible app will use caching

[tansaku]

getting all the data from the 3rd party end points is time consuming and will require sensible caching of some sort. I lean towards dbs where we can have finer grained control than caches. When adding a project I imagine all associated metric would be stored in the db and then could be either manually refreshed or have a once a day job to ensure things don't get too out of date ...

I guess the current code in this repo could be a single gem, or we could just slap a rails app around it and then it's just a service within the app ...

Ability to pass in a list of projects not so important? Well even if not used in the main app would be used in rake task for scheduled job to run through and update all projects ...

[armandofox]

i've actually thought about this.

i think the right model is for each plugin/widget to simply assume that its database table (see the design doc in the wiki) contains up-to-date information, and be agnostic about how it got there; and each widget provides an entry point that scrapes data from a remote API and deposits it in said table.

then the concern of when to run the scraper jobs is totally separate from the itneractive performance of the app.

[tansaku]

I would of thought running scraper jobs at a particular frequency was quite important to the interactive performance of the app. If you have to wait while you manually request updates through the interactive interface it will be quite a drag.

Whereas if one has set the job frequency appropriately for how the interactive interface is used, e.g. a once a day check then the users will find the interface already with the data they need each time they come to it.

Anyway, it's all pretty easy with the design we've gone for now the update everything methods you have in the controller can easily be pulled into services - the key thing is we've nicely separated the persistence mechanism from the metric calculation which was my key concern so I'll close this issue