Query versioning

[arikfr]

• Each query should link to its parent, in case it's a fork of an existing query.
• When changing a query, store last N versions and let the user go back to an older version.

[tsibley]

:+1: This would be great, and the lack of it was one of the key cons on my list of deciding whether or not to adopt redash.

[ChiragKParmar]

+1

[tsibley]

Along the same lines, comparing results between queries could be very useful too. A "Compared to previous version" tab next to the "Results" tab would be neat-o. I often export results to CSV files and compare them with daff.

[tsibley]

Daff has a JS library, so it could perhaps even be integrated directly into the frontend.

[rohanpd]

@whummer has a PR #1113 to implement a changes table. This will be storing the versions of the query but not currently presenting a UI to compare versions.

[arikfr]

@tsibley I'm planning to give access to past query results (they are stored anyway). Will keep daff in mind when it happens.

[WesleyBatista]

I was wondering if would not be better change the approach. The main objective is to have the queries versioned, right? We know that Git is a great versioning system. We could use it.

In my conception we can use GitHub as a "queries source".

Basically it would work as follows: In Redash settings we would have to set up:

• REDASHGITHUBREPO
• REDASHGITHUBUSERNAME
• REDASHGITHUBACCESS_TOKEN

In the UI, on the query creation page, we would have a field where we would put the path to the query For example: path/to/query.sql.

Our handler will hit the GitHub servers at https://api.github.com/repos/{repo}/contents/{path_to_query}, given the auth previously get.

I even started to make a prototype, but not finished it. If someone find it is worth, please feel free to fork me.

I think we could save a good development time with this idea and get the queries versioning working 😄

[tsibley]

@WesleyBatista It's a good idea, and one I've sketched out on paper for working with a local clone with bidirectional sync rather than assuming Github. I definitely like the idea of being able to look at query history in Git with my normal tools.

At the same time, such a change would require a substantial rework of the current backing datastore for queries and query-related data, and this might be prohibitive to a short-term fix. There's also nothing in particular that precludes doing a pluggable, Git-based query store later even if the project implements a non-Git query versioning scheme first for the short term.

[arikfr]

Using Git definitely crossed my mind and I even wrote a script that exports all queries as files to a folder that is managed with Git.

But the challenge for this feature is not only with how to store queries but also in building the appropriate UI, which using Git doesn't solve. I think the right way to go will be to have an optional Git store, but not mandatory.

[WesleyBatista]

Totally agree with you guys! This can not be considered as a final solution. Surely it is an idea to be worked. I just guessed that worth raise the thoughts. The idea is to provide an optinal query source. We would still use the queries stored in-house.

But I was wondering about the requirements to get the idea working. What I thought til now was:

1. a "query url" text field on the query source page.
2. store the query url in the query record.

We can hide the feature on the early steps requiring to set REDASH_ENABLE_EXTERNAL_QUERY_SOURCE to True on the environment.

To store the "query url" we can create a migration to add a 'options' column on queries table like we have on data_sources.

As next steps we could think in improve the UI. The final one could be something more like a modal, navigating through the directories retrieved from the repo. Maybe add support to gists too... I dont know, just wondering.

So the tech debits that I found:

• queries stored outside redash will not be able to be scheduled unless we change some points on the scheduler (redash.tasks.execute_query, redash.tasks.QueryExecutor and redash.tasks.enqueue_query) to accept
• cache the queries (regarding the scheduler). request only if it changed on the source.
• improve the UI

Please, sorry if I am being too ingenuous and missed something (I probably did it). What do you guys think?

[WesleyBatista]

And what do you think about the queries.options column? I think that it actually would be like a "tech credit" :stuck_out_tongue_winking_eye: With a JSON serializable available on queries record level we can open doors to get other features/ideas.

[tsibley]

@WesleyBatista I think an external query source likely wants to be more general than "fetch from a URL". Just like there are different data sources, there should probably be different query providers you can plug in and use. One provider might fetch from a URL, one from git, and another from a database itself. One way to design it might be to have the default query provider be a database provider pointed at re:dash's own database.

I expect, as you identified as a possibility, that any solution would want to cache query sources somewhere and have a means for updating/invalidating that cache. The benefit of using the current queries table as the local cache is that other parts of redash can remain the same which simplifies the overall changes necessary.

A design that's appealing to me (and common in many areas) is something like a cascading provider list where, when it comes time to update a query source, each provider is tried in turn to provide the query source until one is able to. When it is, the query source is cached and the cache provider will be able to return it sooner the next time as long as it hasn't expired.

[ghost]

hello ... I would give the redash project development especially ui of it ... help me for optimized architecture? i want create a folder beside "app" in client => client/myProject

[carnevlu]

any news about this features ?

[frqnck]

Any new about the? It would be a great option to have and would make Redash so compelling.

Also instead of limiting this to just Github, it would be preferable to open this to any generic Git repo server so that any self-hosted, or SAS services Git servers out there can be used (over SSH)-- and even allow a localhost repos on a Redash instance...

[ruchimann]

Any updates here.

[susodapop]

Nothing to share at this time. Later this year we plan to have a more transparent roadmap / engagement with contributors. We'll announce that on the user forum.

[gallamine]

Any update here?

[susodapop]

Not at this time.

[joe42]

I saw that https://github.com/mozilla/redash/pull/149 was merged, that adds query versioning, or? If so which version of redash will this be included in? I don't see it in the version we are using.

[susodapop]

Mozilla haven't upstreamed this to the primary redash repository so there's no roadmap to include their change here.

[Auric-Manteo]

Are the query versions already stored in the database so I can access them manually?

[jaoxford]

Are there any more updates on this? This would be a very useful feature to have.