mitodl / ol-data-platform

Pipeline definitions for managing data flows to power analytics at MIT Open Learning
BSD 3-Clause "New" or "Revised" License
36 stars 6 forks source link

Update OCW Resources Report to Include Learning Resource Types and Resource Title #1205

Closed pt2302 closed 1 month ago

pt2302 commented 1 month ago

What are the relevant tickets?

Resolves https://github.com/mitodl/hq/issues/3894#issuecomment-2231482794.

Description (What does it do?)

This PR adds the new fields learning_resource_types and resource_title to the OCW resource report. Note that an empty array of learning_resource_types is treated as null. In addition, the fields in the intermediate resources model have been alphabetized for improved readability.

How can this be tested?

Run the following commands; the tests should all pass

dbt build --select staging.ocw  --vars 'schema_suffix: <your name>' --target dev_production
dbt build --select intermediate.ocw --vars 'schema_suffix: <your name>' --target dev_production

Run the following query in https://mitol.galaxy.starburst.io/query-editor to see the result of the above tables:

SELECT * FROM ol_data_lake_production.ol_warehouse_production_<your name>_intermediate.int__ocw__resources

Then, verify that the new fields are present in the table, using a query like

select resource_uuid, resource_title, learning_resource_types from ol_warehouse_production_<your name>_intermediate.int__ocw__resources
KatelynGit commented 1 month ago

the organization change looks good too