search

scraperwiki / odata-tool

Access your ScraperWiki data in tools like Tableau, QlikView and Excel Power Query
3 stars 0 forks source link

Hitting "Refresh" in Tableau should automagically update the underlying data #20

Open drj11 opened 10 years ago

drj11 commented 10 years ago

At the moment if a Tableau user hits Refresh on an OData source, it merely reloads the data from the dataset (over the OData endpoint). The data itself will not be fresh. Even with data that is updated on a schedule, the data won't be fresh until the next scheduled update of the data has happened.

So to get fresh data into Tableau there is a two step process:

1) wait for the ScraperWiki dataset to be updated, or update it manually; and, 2) press Refresh in Tableau.

At least in some circumstances it would be nice if hitting Refresh in Tableau triggered an update of the underlying data. The example which triggered this issue is the earthquake example from the GeoJSON tool. It's plausible that someone would want Refresh to show them the most recent earthquakes.

We wouldn't want to do this all the time, for various reasons, for example: the data update process might be destructive; the data update process might be very expensive; the data might represent some useful snapshot and updating it would destroy it's usefulness.

We'll need some way to configure how and when to automatically fetch updated data.

frabcus commented 10 years ago

This sounds terrifying - like a reverse version of the update hooks.

Francis

On Tue, Sep 09, 2014 at 04:18:55AM -0700, David Jones wrote:

At the moment if a Tableau user hits Refresh on an OData source, it merely reloads the data from the dataset (over the OData endpoint). The data itself will not be fresh. Even with data that is updated on a schedule, the data won't be fresh until the next scheduled update of the data has happened.

So to get fresh data into Tableau there is a two step process:

1) wait for the ScraperWiki dataset to be updated, or update it manually; and, 2) press Refresh in Tableau.

At least in some circumstances it would be nice if hitting Refresh in Tableau triggered an update of the underlying data. The example which triggered this issue is the earthquake example from the GeoJSON tool. It's plausible that someone would want Refresh to show them the most recent earthquakes.

We wouldn't want to do this all the time, for various reasons, for example: the data update process might be destructive; the data update process might be very expensive; the data might represent some useful snapshot and updating it would destroy it's usefulness.

We'll need some way to configure how and when to automatically fetch updated data.

Reply to this email directly or view it on GitHub: https://github.com/scraperwiki/odata-tool/issues/20

drj11 commented 10 years ago

Technically we could do something simple like have a symlink called odata-refresh in the dataset. When the odata tool gets invoked, it would run the odata-refresh script if there was one.

drj11 commented 10 years ago

read what @frabcus wrote while I was writing my previous comment. hmm. now you say that, it does sound like a reverse version of update hooks.

IanHopkinson commented 10 years ago

I was thinking update hook when we first spoke about this. This functionality is key to the Tableau workflow, in my opinion. Would be essentially for the Google Sheet/Google Forms use case

drj11 commented 10 years ago

in that sense though, cgi scripts, which we support, are already like a reverse version of update hooks. One of the reasons that update hooks were bad, is that the underlying data was changing far more frequently (say, every hour) then it was ever accessed (say, once ever, or once per month).

drj11 commented 10 years ago

Possible user interface (thought of a while ago, recording here so i don't forget):

odata-tool detects when the source data is "automatic update enabled", and if so, offers either two URLs (one auto update, one not), or a checkbox that flips between the two URLs. odata-cgi is modified so that it senses which URL is being used and does the automatic update if required.

frabcus commented 10 years ago

Wonder if we should have standard update convention for this - maybe related to adding queuing too.

More UI for this is a bit confusing/horrid...

On Mon, Sep 15, 2014 at 01:43:02AM -0700, David Jones wrote:

Possible user interface (thought of a while ago, recording here so i don't forget):

odata-tool detects when the source data is "automatic update enabled", and if so, offers either two URLs (one auto update, one not), or a checkbox that flips between the two URLs. odata-cgi is modified so that it senses which URL is being used and does the automatic update if required.

Reply to this email directly or view it on GitHub: https://github.com/scraperwiki/odata-tool/issues/20#issuecomment-55564894

drj11 commented 10 years ago

On the UI: fundamentally the user may have to decide whether they want the data to change or not. We can't tell.