Closed benigls closed 1 year ago
This is indeed a resource intensive model as it is based on int_recurly__account_rolling_totals
which is based on int_recurly__transactions_date_spine
, where both tables have around 350 million records. In Snowflake the query took a long time to complete, 2820.44 seconds.
I'm now comparing it a similar model in Stripe stripe__daily_overview to try and find differences/similarities and maybe come up with a more efficient way.
Thanks for opening this issue @benigls, and thanks for taking a look to dig in @dimoschi!
This is great for us to know and I imagine there should be a way we can optimize this query to be more performant. I will be sure to dig into this as well and explore a possible solution to hopefully be rolled out in our upcoming sprint.
Thanks for bringing this issue to our attention @benigls and examining this problem more closely @dimoschi!
It looks like the main culprit might be the multiple OVER functions happening in the daily_overview
.
I'm wondering if we take those resource-intensive functions out of the final model, separate them into additional intermediate model(s), then somehow separate them out before we get to the end model and join the two models together), we can reduce the load time and the query runs.
My original thought is we could bring both of the OVER CTEs into one intermediate model, see if it can run on its own. If that still has trouble running, we can split them up into multiple CTEs in two intermediate models.
Then the only operation we have to do is the join to account_overview
in the final model and adding the passthrough columns.
Would either of you be able to take a look and try breaking that final model logic up? Otherwise we can attempt to tackle this in an upcoming sprint!
I'll do my best to look into it until the end of this week, but cannot promise anything.
Hello @benigls and @dimoschi!
I've attempted the above solution to try and resolve the query execution issue. Before I send over the proposed PR, I just want to double-check what your specific dbt environment setup is looking like. Some of our code we're deploying and testing is dependent on your current version, so we might need to make some modifications if you're using a specific version of dbt.
Could you provide the following details:
Thanks for all your help and patience in resolving this issue!
Thank you for taking care of this @fivetran-avinash!
Let me know if you need further info.
Thanks @benigls!
I think your versions are up-to-date with what can run on our branch.
Here is the link to the branch with the updated changes: https://github.com/fivetran/dbt_recurly/tree/bug/account-daily-overview-query-resources-exceeded
You can clone the repo, fetch this specific branch above, and explore whether these code changes make a difference in resource execution on your data.
Let me know how it goes and if you have any questions!
Hi @fivetran-avinash. I just tested the branch and all the models are running in my environment and data. 🚀 Thank you!
Side note: I noticed that it took 3.5 minutes to run all the models. I wonder, is there a way to add a condition in the base layer to only process accounts with an existing subscription? The reason for asking is that in our case, we only care about the accounts with a subscription, which is ~1/4 of the entire account population. I think this is a common pattern with companies with a free-tier plan.
Hi @benigls thanks for testing out the branch and that is great to hear that the models are running with your data now!!
The majority of our team is on PTO until the new year, but we will be sure to pick this up and plan to integrate into the next release once we come back in the new year.
In the meantime, I think your suggestion for filtering out accounts without active subscriptions is completely reasonable and would add value for all of the package users. Realistically, we would probably add a variable to allow customers to configure if they want the filter applied or not. In order for us to ensure we can get this fix out I am going to move the filter request into a new feature request (FR #11) where we will be able to scope it out and integrate into a future sprint. Be sure to follow the linked FR to see when we are able to integrate the filter and test out the branch once it is available.
Thanks again for all your help!
Closing this issue since the branch has been merged.
Is there an existing issue for this?
Describe the issue
After running all the recurly models, I got a memory exceeded error on the last model, recurly__account_daily_overview
Relevant error log or model output