mitodl / ol-data-platform

Pipeline definitions for managing data flows to power analytics at MIT Open Learning
BSD 3-Clause "New" or "Revised" License
36 stars 6 forks source link

Update metadata object keys to organize raw metadata files #1180

Closed quazi-h closed 2 months ago

quazi-h commented 2 months ago

What are the relevant tickets?

https://github.com/mitodl/hq/issues/4067

Description (What does it do?)

With the way the object keys are currently configured, metadata filenames include both the source_system and course_id rather than creating a new path for each source_system and course_id pair. These changes update the course_metadata_object_key to make sure each source_system has its on unique destination path.

How can this be tested?

Let the course metadata get processed in production, double check the file structure/path of the resulting files in S3.