Count number of values in tables for ETL

tl-its-umich-edu / unizin-validation

Unizin Validation Scripts

1 stars 4 forks source link

Count number of values in tables for ETL #2

Closed jonespm closed 4 years ago

jonespm commented 5 years ago

We want to have a count of the values in the tables needed for the ETL process. @zqian will provide the tables.

According to IQ data mapping document: here are the list of relevant UDW tables: COURSE_DIM, ASSIGNMENT_DIM, USER_DIM, SUBMISSION_FACT, SUBMISSION_DIM, SUBMISSION_COMMENT_DIM,ENROLLMENT_TERM_DIM, SCORE_FACT,PSEUDONYM_DIM,

jonespm commented 5 years ago

This could also probably just check all of the tables counts. There's for sure a command to get all tables, then we'd just iterate over them.

zqian commented 4 years ago

Looks good to me! I thought there should be a "flag setting" step so that if any of those table count is zero, the warning flag should be set true, and surfaced via Jenkins endpoint.

Maybe embed the task in another issue?

ssciolla commented 4 years ago

This has been resolved by PR #7. Work on flags/checks has moved to issue #8.