feat: move second-tier MVs into dbt project (FC-0033) - Githubissues

openedx / aspects-dbt

The dbt project for Open edX Aspects!

Apache License 2.0

2 stars 4 forks source link

feat: move second-tier MVs into dbt project (FC-0033) #34

Closed SoryRawyer closed 8 months ago

SoryRawyer commented 10 months ago

Now that dbt-clickhouse supports materialized views, this PR moves MV creation and maintenance to the Aspects dbt project. MV definitions are included for:

xapi_events_all_parsed
completion_events
enrollment_events
forum_events
grading_events
problem_events
video_playback_events

openedx-webhooks commented 10 months ago

Thanks for the pull request, @SoryRawyer! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

SoryRawyer commented 9 months ago

@bmtcril this is ready for review. The dbt version upgrades seem to work without issue and the MVs seem to match what I have with the alembic-created tables.

bmtcril commented 9 months ago

Sorry I was waiting for this to merge, can you pull in these new ones in as well? Then we should be able to make the changeover. https://github.com/openedx/tutor-contrib-aspects/pull/529

SoryRawyer commented 9 months ago

Just to be sure, are you talking about the user profile table? I don't see any views in that PR and I don't think dictionaries are supported by dbt-clickhouse.

SoryRawyer commented 9 months ago

Sorry, I just noticed that the user profile table is mentioned in this issue as being included in the alembic migrations. I think the only MV in our migrations that isn't listed in that issue is the most recent blocks MV. Do you want me to include that in this PR?

bmtcril commented 9 months ago

Hmmm yeah I think we should move everything we can that's not a "base needed to write to" table. If you could confirm about the dictionaries that would be great. Seems like another thing ripe for an upstream PR.

bmtcril commented 9 months ago

(FWIW I don't think the dictionaries thing is blocking this at all, just curious about it for future work)

SoryRawyer commented 9 months ago

Apologies for the delay - dictionaries aren't supported at the moment, but that does seem like something we could contribute to dbt-clickhouse.

Ian2012 commented 9 months ago

@SoryRawyer can you handle Brian comment around the Json extract float, and squash the commits?

SoryRawyer commented 9 months ago

@bmtcril were you imagining the xapi_events_all_parsed being part of this PR as well? I think that might be the last MV in the alembic migrations.

bmtcril commented 9 months ago

@SoryRawyer sure, let's move it 👍

openedx-webhooks commented 8 months ago

@SoryRawyer 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.