search

knowledgesystems / pipelines-scrum

Repository for tracking uncategorizable issues related to backend pipelines work
0 stars 0 forks source link

Use Trigger for S3 Pull/Push + CDM Job #1317

Closed averyniceday closed 1 week ago

averyniceday commented 1 month ago

Done Condition (What do we need? Why do we need it? Keep this is small as possible!)

Work in conjunction with Chris to add FileSensor and/or S3 push notification to trigger next step in the cycle

Pipeline should no longer run at specified timeslots (excluding final pull of CDM data for overnight merging)

Technical Description (How are we going to achieve the above)

Potential Issues

Dependencies

Technical Requirements

Outside People/Teams

Changes

callachennault commented 2 weeks ago

https://github.com/clinical-data-mining/dags/pull/15 https://github.com/clinical-data-mining/cdm-cbioportal-etl/pull/74