mitodl / ol-data-platform

Pipeline definitions for managing data flows to power analytics at MIT Open Learning
BSD 3-Clause "New" or "Revised" License
36 stars 6 forks source link

Add cybersource payment transaction fields to combined order mart and its upstream #1212

Closed rachellougee closed 1 month ago

rachellougee commented 1 month ago

What are the relevant tickets?

https://github.com/mitodl/hq/issues/4775

Description (What does it do?)

Adding a few cybersource transaction fields and unit_price used in https://bi.odl.mit.edu/queries/1189 and https://bi.odl.mit.edu/queries/1190/ to martscombinedorders and its upstream models

How can this be tested?

dbt build --select +martscombinedorders

or

dbt build --select intmitxonlineecommerce_order dbt build --select +intmitxonlineecommerce_transaction dbt build --select intmicromastersorders dbt build --select intmitxproecommerce_order dbt build --select +intmitxproecommerce_receipt dbt build --select +intbootcampsecommerce_receipt dbt build --select +intbootcampsecommerce_order dbt build --select martscombinedorders

rachellougee commented 1 month ago

My only question is whether we should remove the fields from staging as well? the main concern is are they currently being used elsewhere? But builds, definitions, and code look great so approving

Thanks for the review. To answer your questions about removing these fields from staging, I've replaced their references with intermediate so there shouldn't be anywhere else. The usage can be found in the lineage graph in the right corner https://mitodl.github.io/ol-data-platform/#!/model/model.open_learning.stg__bootcamps__app__postgres__ecommerce_receipt

https://mitodl.github.io/ol-data-platform/#!/model/model.open_learning.stg__mitxpro__app__postgres__ecommerce_receipt

I figured it's easier to handle these fields directly in intermediate so we don't have to copy paste these fields from staging to intermediate.