search

DataJunction / dj

A metrics platform.
http://datajunction.io
MIT License
35 stars 15 forks source link

Adding 'use_materialized' flag to sql endpoints #1161

Closed skastel closed 2 months ago

skastel commented 2 months ago

Summary

To address https://github.com/DataJunction/dj/issues/1158 this adds an use_materialized flag to the sql endpoints so that clients can set to False and request SQL that bypasses any materialized data. By default, no behavior is changing, the default value is still True for this flag.

Test Plan

Unit tests cover this code already, focus is on making sure

Deployment Plan

Nothing special.

netlify[bot] commented 2 months ago

Deploy Preview for thriving-cassata-78ae72 canceled.

Name Link
Latest commit 22243476c841464db4f9cb9005ddcd9120e33adb
Latest deploy log https://app.netlify.com/sites/thriving-cassata-78ae72/deploys/66df7c1ad31559000890d11b
agorajek commented 2 months ago

Looks fine to me, but I am still wondering what would be the use case for not using materialized tables?

skastel commented 2 months ago

Thanks for adding this @skastel! Could you add an additional unit test for the use_materialized=False case, when a materialized table exists for the node?

Just added a unit test for this @shangyian

skastel commented 2 months ago

Looks fine to me, but I am still wondering what would be the use case for not using materialized tables?

@agorajek By allowing a client to get the full definition of the node/measure/metric and bypassing any materializations the client can see the underlying data sources. We'll likely use this in a few places where we're trying to do our own materializations and caching in a parallel headless BI system, but relying on DJ for definitions.

agorajek commented 2 months ago

Looks fine to me, but I am still wondering what would be the use case for not using materialized tables?

@agorajek By allowing a client to get the full definition of the node/measure/metric and bypassing any materializations the client can see the underlying data sources. We'll likely use this in a few places where we're trying to do our own materializations and caching in a parallel headless BI system, but relying on DJ for definitions.

Sad panda... but that makes sense.