Add IT tests for handling schema update during spanner cdc to big query pipeline running

[ShuranZhang]

Add IT tests for handling schema update during a spanner cdc to bigquery pipeline running. Previously a base case test that doesn't contain any schema change was added in https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1705 . This pull request adds two more IT tests following above workflow:

• Add a column(disabled): add a column in spanner source table -> add a column in corresponding bq table ->wait for 15mins(this is due to limitation of BigQuery and BQ team recommends wait >10 mins, see more here)->insert a row to source table with new column filled->verify bq table's new column is successfully populated. Due to the above bq limitation, this test requires sleeping at least 10 mins in test body to ensure no flakiness. Thus I disabled this test to avoid just hang the thread and do hardcoded sleep for such a long time.
• Add a table: add a source table in spanner -> insert a new row to new source table -> verify that the new changelog table in created and filled with correct information in bq. Different from updating an existing bq table's schema, add a completely new table does not suffer from the high latency.

I have run all IT tests together in parallel for >15 times locally and no flakiness observed. We can add more test cases later if tests in this pr are running stable.

I also added a 3 minutes dlq retry parameters to all IT tests in case there are any availability/transient errors happening on the spanner side.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 51.20275% with 142 lines in your changes missing coverage. Please review.

Project coverage is 42.93%. Comparing base (6b094c2) to head (8209708). Report is 16 commits behind head on main.

Files with missing lines	Patch %	Lines
...tobigquery/schemautils/SpannerToBigQueryUtils.java	53.52%	24 Missing and 9 partials :warning:
...eamstobigquery/SpannerChangeStreamsToBigQuery.java	0.00%	28 Missing :warning:
...reamstobigquery/schemautils/SchemaUpdateUtils.java	60.00%	16 Missing and 6 partials :warning:
...hangestreamstobigquery/schemautils/TypesUtils.java	58.33%	17 Missing and 3 partials :warning:
...erchangestreamstobigquery/model/ModColumnType.java	33.33%	18 Missing :warning:
...bigquery/FailsafeModJsonToTableRowTransformer.java	0.00%	12 Missing :warning:
...streamstobigquery/BigQueryDynamicDestinations.java	0.00%	4 Missing :warning:
...ates/spannerchangestreamstobigquery/model/Mod.java	62.50%	2 Missing and 1 partial :warning:
...gestreamstobigquery/model/TrackedSpannerTable.java	83.33%	1 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1730 +/- ## ============================================ + Coverage 42.83% 42.93% +0.10% - Complexity 3424 3455 +31 ============================================ Files 824 827 +3 Lines 48147 48275 +128 Branches 5168 5196 +28 ============================================ + Hits 20624 20728 +104 - Misses 25847 25860 +13 - Partials 1676 1687 +11 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `62.81% <ø> (ø)` | | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.90% <ø> (ø)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.05% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `67.63% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `83.75% <ø> (ø)` | | | [Files with missing lines](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [.../beam/it/gcp/bigquery/BigQueryResourceManager.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=it%2Fgoogle-cloud-platform%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbeam%2Fit%2Fgcp%2Fbigquery%2FBigQueryResourceManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-aXQvZ29vZ2xlLWNsb3VkLXBsYXRmb3JtL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9iZWFtL2l0L2djcC9iaWdxdWVyeS9CaWdRdWVyeVJlc291cmNlTWFuYWdlci5qYXZh) | `72.85% <ø> (ø)` | | | [...ngestreamstobigquery/schemautils/OptionsUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fschemautils%2FOptionsUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvc2NoZW1hdXRpbHMvT3B0aW9uc1V0aWxzLmphdmE=) | `61.90% <ø> (-1.74%)` | :arrow_down: | | [...igquery/schemautils/SpannerChangeStreamsUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fschemautils%2FSpannerChangeStreamsUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvc2NoZW1hdXRpbHMvU3Bhbm5lckNoYW5nZVN0cmVhbXNVdGlscy5qYXZh) | `80.08% <100.00%> (+9.18%)` | :arrow_up: | | [...gestreamstobigquery/model/TrackedSpannerTable.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fmodel%2FTrackedSpannerTable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvbW9kZWwvVHJhY2tlZFNwYW5uZXJUYWJsZS5qYXZh) | `61.53% <83.33%> (+13.39%)` | :arrow_up: | | [...ates/spannerchangestreamstobigquery/model/Mod.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fmodel%2FMod.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvbW9kZWwvTW9kLmphdmE=) | `48.07% <62.50%> (+48.07%)` | :arrow_up: | | [...streamstobigquery/BigQueryDynamicDestinations.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2FBigQueryDynamicDestinations.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvQmlnUXVlcnlEeW5hbWljRGVzdGluYXRpb25zLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...bigquery/FailsafeModJsonToTableRowTransformer.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2FFailsafeModJsonToTableRowTransformer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvRmFpbHNhZmVNb2RKc29uVG9UYWJsZVJvd1RyYW5zZm9ybWVyLmphdmE=) | `0.00% <0.00%> (ø)` | | | [...erchangestreamstobigquery/model/ModColumnType.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fmodel%2FModColumnType.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvbW9kZWwvTW9kQ29sdW1uVHlwZS5qYXZh) | `33.33% <33.33%> (ø)` | | | [...hangestreamstobigquery/schemautils/TypesUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fschemautils%2FTypesUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvc2NoZW1hdXRpbHMvVHlwZXNVdGlscy5qYXZh) | `58.33% <58.33%> (ø)` | | | [...reamstobigquery/schemautils/SchemaUpdateUtils.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree&filepath=v2%2Fgooglecloud-to-googlecloud%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftemplates%2Fspannerchangestreamstobigquery%2Fschemautils%2FSchemaUpdateUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvZ29vZ2xlY2xvdWQtdG8tZ29vZ2xlY2xvdWQvc3JjL21haW4vamF2YS9jb20vZ29vZ2xlL2Nsb3VkL3RlbGVwb3J0L3YyL3RlbXBsYXRlcy9zcGFubmVyY2hhbmdlc3RyZWFtc3RvYmlncXVlcnkvc2NoZW1hdXRpbHMvU2NoZW1hVXBkYXRlVXRpbHMuamF2YQ==) | `60.00% <60.00%> (ø)` | | | ... and [2 more](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1730?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | |

[Abacn]

DLPTextToBigQueryStreamingIT.testDLPTextToBigQuery failed due to #1836