datahub-project / datahub

The Metadata Platform for your Data and AI Stack
https://datahubproject.io
Apache License 2.0
9.93k stars 2.94k forks source link

feat(mcl-upgrade): implement resume & urn pagination #11889

Closed david-leifker closed 2 days ago

david-leifker commented 3 days ago

Implementing resume functionality for upgrades using the AbstractMCLStep. Examples include edge status and CLL job which can be long running and benefit from resuming. Additionally logging for the last processed urn as it progresses. Added unit test for the resume capability.

Checklist