GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.16k stars 976 forks source link

tests: Adding Forward Migration Tests #2001

Open asthamohta opened 2 weeks ago

asthamohta commented 2 weeks ago

Tests added:

  1. Sequence
  2. Dropped table
  3. Mixed IT: 1. Foreign Keys. 2. Table Dropped. 3. Column Rename. 4. DLQ Retry. 5. Missing PK 6. Indexes
codecov[bot] commented 2 weeks ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 52.93%. Comparing base (1bef5ff) to head (bdd8da5). Report is 12 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #2001 +/- ## ============================================ + Coverage 45.30% 52.93% +7.62% + Complexity 3631 1367 -2264 ============================================ Files 836 378 -458 Lines 49922 20661 -29261 Branches 5260 2090 -3170 ============================================ - Hits 22615 10936 -11679 + Misses 25631 9045 -16586 + Partials 1676 680 -996 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `67.93% <ø> (+1.27%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.88% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `76.65% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `86.37% <ø> (+0.02%)` | :arrow_up: | [see 491 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/2001/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)

🚨 Try these New Features:

asthamohta commented 2 weeks ago

The End-to-End test checks for multiple conditions at the same time in the same table unlike other tests. It checks for, for examples FK,Column Rename+,tables dropped,Indexes etc, all on the same table

manitgupta commented 2 weeks ago

multiple conditions at the same time in the same table unlike other tests. It checks for, for examples FK,Column Rename+,tables dropped,Indexes etc, all on the same table

How does that add any additional value v/s checking these operations on a different sets of tables?

It's not clear to me what is the definition of end-to-end..

Can we define this please?

darshan-sj commented 2 weeks ago

In our existing ITs we test for column rename and DLQ handling for FK/Interleaving separately. We had a bug in our template where the DLQ handling was not happening properly when the columns are renamed, and it dint get caught through ITs. So, we wanted to add one IT which covers all the cases simultaneously.