fix: use table instead of MV for section_video_engagement MV

openedx / aspects-dbt

The dbt project for Open edX Aspects!

Apache License 2.0

2 stars 4 forks source link

fix: use table instead of MV for section_video_engagement MV #99

Closed Ian2012 closed 2 months ago

Ian2012 commented 2 months ago

Description

This PR reduces the dependencies from the section/subsection_video_engagement MV, so that it doesn't get OOM killed on large datasets.

Performance gains:

Before:

0 rows in set. Elapsed: 0.283 sec. Processed 11.21 thousand rows, 2.36 MB (39.64 thousand rows/s., 8.35 MB/s.)
Peak memory usage: 35.93 MiB

After:

0 rows in set. Elapsed: 0.062 sec. Processed 8.48 thousand rows, 1.97 MB (136.64 thousand rows/s., 31.79 MB/s.)
Peak memory usage: 12.89 MiB.

openedx-webhooks commented 2 months ago

Thanks for the pull request, @Ian2012! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

openedx-webhooks commented 2 months ago

@Ian2012 Even though your pull request wasn’t merged, please take a moment to answer a two question survey so we can improve your experience in the future.