search

fivetran / dbt_amplitude

Fivetran's Amplitude dbt package
https://fivetran.github.io/dbt_amplitude/
Apache License 2.0
3 stars 3 forks source link

[Feature] Move the "change event date range" upstream #10

Closed jon-openspace closed 1 year ago

jon-openspace commented 1 year ago

Is there an existing feature request for this?

Describe the Feature

Currently, "change event date range" does not apply to the staging models. Because of this, dbt build takes a very long time to complete. On my Amplitude Dataset, there are ~65M events since 2019, and it takes ~15 minutes to run the amplitude models. Most of this time is spent generating the staging models.

Describe alternatives you've considered

I created a temporary clone of the raw event table and pointed the package towards it, and that drastically reduced the amount of time it tool to build the models. However, I cannot do this programmatically with dbt, as we keep our raw tables in a separate snowflake database that the dbt role cannot write tables to.

Are you interested in contributing this feature?

Anything else?

No response

fivetran-joemarkiewicz commented 1 year ago

Hi @jon-openspace thanks so much for opening this feature request!

I think this is a completely reasonable request and would greatly improve the performance of the package and downstream transformations. This would result in a bit different behavior for our staging models from our other packages (we typically try not to filter out data in the staging models); however, this package should be an exception since the event data can grow in volume and manage to be unusable with the run time if we don't filter out in the staging model.

I will accept this Feature and plan to provide an update once our team starts working on this. Thanks again!

fivetran-reneeli commented 1 year ago

Feature added in the latest release https://github.com/fivetran/dbt_amplitude_source