GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.11k stars 931 forks source link

Spanner to Source basic impl #1707

Open aksharauke opened 1 week ago

aksharauke commented 1 week ago

The template does the basic reverse replication of multiple shards. Follow-up PRs to include:

  1. More UT
  2. More IT
  3. DLQ handling
  4. Session file absence
  5. Custom transform porting
codecov[bot] commented 5 hours ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 42.25%. Comparing base (dd00fe4) to head (e3ec440). Report is 88 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1707 +/- ## ============================================ + Coverage 41.25% 42.25% +1.00% - Complexity 2939 3162 +223 ============================================ Files 771 791 +20 Lines 45114 46137 +1023 Branches 4818 4935 +117 ============================================ + Hits 18612 19496 +884 - Misses 24935 25047 +112 - Partials 1567 1594 +27 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.59% <ø> (+2.35%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.32% <ø> (-0.11%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `74.97% <ø> (+0.93%)` | :arrow_up: | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `51.44% <ø> (+1.02%)` | :arrow_up: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.14% <ø> (+2.57%)` | :arrow_up: | [see 45 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1707/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)