getredash / redash

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
http://redash.io/
BSD 2-Clause "Simplified" License
26.45k stars 4.38k forks source link

Support result reuse in Athena data sources #7202

Open dtaniwaki opened 1 month ago

dtaniwaki commented 1 month ago

What type of PR is this?

Description

Currently, redash makes a query everytime the query page opens and it costs a lot if the query is too heavy.

On the other hand, there is a feature that efficiently uses query results in Athena. https://docs.aws.amazon.com/athena/latest/ug/reusing-query-results.html

pyathena (>=2.18.0) supports the option. https://github.com/laughingman7743/PyAthena/blob/6b8f0e94abd09115ba1277616ca84372e2f15d56/pyathena/common.py#L112-L113 It seems the reason of updating the major version is dropping python versions. and as far as I tried, queries work fine.

We may want to turn on/off the option per query but it will break the compatibility with the other data sources, so I made it a data source option. Could you consider adding this feature?

How is this tested?

Run an Athena query with this option with manually container image.

Related Tickets & Documents

N/A

Mobile & Desktop Screenshots/Recordings (if there are UI changes)

Screenshot_2024-10-23_at_1_49_15
dtaniwaki commented 3 days ago

@lucydodo @justinclift Could you consider merging this feature?

lucydodo commented 1 day ago

It looks good, but since the version of the pyathena package goes up quite a bit, so we should provbably look into the possibility of other side effectes before merging.