GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

Adding init seq to set session timezone. #1623

Closed VardhanThigle closed 3 months ago

VardhanThigle commented 3 months ago

Adding Initialization Sequence for the JDBC connection.

Summary of Changes

Session Timezone: The PR sets the session time zone to UTC by default to retrieve timestamp in UTC. This is asper the contract exposed by the Reader to always output timestamp in UTC (irrespective of the source settings).The most idomatic way to achieve this via jdbc would be to pass a Calendar object initialized to UTC to java.sql.ResultSet.getTimestamp(String, Calendar) api (which is what we do in MysqlJdbcValueMappings), but due to bugs like Bug#95644 , Bug#96276, Bug#93444, etc. in the older drivers, we are also setting the session timezone to UTC.

Testing

This change has been tested on an end to end migration setup where MySQL server was in IST timezone and the timestamp was migrated to spanner by verifying that the migrated timestamp obtained from Spanner was as per the correct timezone offsets applied.

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 83.33333% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 45.10%. Comparing base (6d4dbf8) to head (422ca47).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1623 +/- ## ============================================ + Coverage 41.31% 45.10% +3.78% + Complexity 2888 711 -2177 ============================================ Files 752 297 -455 Lines 43705 15966 -27739 Branches 4656 1581 -3075 ============================================ - Hits 18056 7201 -10855 + Misses 24127 8229 -15898 + Partials 1522 536 -986 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `58.39% <83.33%> (-2.64%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `73.38% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `48.48% <ø> (-0.12%)` | :arrow_down: | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `81.73% <83.33%> (+<0.01%)` | :arrow_up: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [.../io/jdbc/iowrapper/config/JdbcIOWrapperConfig.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2Fconfig%2FJdbcIOWrapperConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9jb25maWcvSmRiY0lPV3JhcHBlckNvbmZpZy5qYXZh) | `100.00% <100.00%> (ø)` | | | [...iowrapper/config/defaults/MySqlConfigDefaults.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2Fconfig%2Fdefaults%2FMySqlConfigDefaults.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9jb25maWcvZGVmYXVsdHMvTXlTcWxDb25maWdEZWZhdWx0cy5qYXZh) | `100.00% <100.00%> (ø)` | | | [...source/reader/io/jdbc/iowrapper/JdbcIoWrapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2FJdbcIoWrapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9KZGJjSW9XcmFwcGVyLmphdmE=) | `90.35% <66.66%> (-0.65%)` | :arrow_down: | ... and [471 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1623/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)