search

GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

feat: Adds SourceDBToSpanner PostgreSQL Adapter #1754

Closed thiagotnunes closed 1 week ago

thiagotnunes commented 1 month ago

Adds implementation for reading source PostgreSQL database.

We purposefully left out parts of the PostgreSQL adapter support, because this pull request was becoming too large. This is what still needs to be done after this PR is merged:

codecov[bot] commented 1 month ago

Codecov Report

Attention: Patch coverage is 89.72603% with 30 lines in your changes missing coverage. Please review.

Project coverage is 49.77%. Comparing base (6b094c2) to head (899530d). Report is 1 commits behind head on main.

Files Patch % Lines
...ctadapter/postgresql/PostgreSQLDialectAdapter.java 87.15% 21 Missing and 2 partials :warning:
...owmapper/provider/PostgreSQLJdbcValueMappings.java 87.50% 3 Missing :warning:
...ud/teleport/v2/options/OptionsToConfigBuilder.java 92.85% 0 Missing and 2 partials :warning:
...loud/teleport/v2/templates/PipelineController.java 0.00% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1754 +/- ## ============================================ + Coverage 42.83% 49.77% +6.94% + Complexity 3424 1273 -2151 ============================================ Files 824 363 -461 Lines 48147 19738 -28409 Branches 5168 1978 -3190 ============================================ - Hits 20624 9825 -10799 + Misses 25847 9254 -16593 + Partials 1676 659 -1017 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.03% <89.72%> (+0.21%)` | :arrow_up: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `∅ <ø> (∅)` | | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `75.05% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `67.63% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `84.09% <89.72%> (+0.34%)` | :arrow_up: | | [Files](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [...source/reader/io/jdbc/iowrapper/JdbcIoWrapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2FJdbcIoWrapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9KZGJjSW9XcmFwcGVyLmphdmE=) | `93.15% <100.00%> (+0.09%)` | :arrow_up: | | [.../io/jdbc/iowrapper/config/JdbcIOWrapperConfig.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2Fconfig%2FJdbcIOWrapperConfig.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9jb25maWcvSmRiY0lPV3JhcHBlckNvbmZpZy5qYXZh) | `100.00% <100.00%> (ø)` | | | [...ce/reader/io/jdbc/iowrapper/config/SQLDialect.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2Fconfig%2FSQLDialect.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9jb25maWcvU1FMRGlhbGVjdC5qYXZh) | `100.00% <100.00%> (ø)` | | | [...pper/config/defaults/PostgreSQLConfigDefaults.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Fiowrapper%2Fconfig%2Fdefaults%2FPostgreSQLConfigDefaults.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL2lvd3JhcHBlci9jb25maWcvZGVmYXVsdHMvUG9zdGdyZVNRTENvbmZpZ0RlZmF1bHRzLmphdmE=) | `100.00% <100.00%> (ø)` | | | [.../reader/io/jdbc/rowmapper/JdbcSourceRowMapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fjdbc%2Frowmapper%2FJdbcSourceRowMapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9qZGJjL3Jvd21hcHBlci9KZGJjU291cmNlUm93TWFwcGVyLmphdmE=) | `100.00% <100.00%> (ø)` | | | [.../v2/source/reader/io/schema/SourceTableSchema.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fschema%2FSourceTableSchema.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9zY2hlbWEvU291cmNlVGFibGVTY2hlbWEuamF2YQ==) | `100.00% <100.00%> (ø)` | | | [...eader/io/schema/typemapping/UnifiedTypeMapper.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fschema%2Ftypemapping%2FUnifiedTypeMapper.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9zY2hlbWEvdHlwZW1hcHBpbmcvVW5pZmllZFR5cGVNYXBwZXIuamF2YQ==) | `100.00% <100.00%> (ø)` | | | [...ypemapping/provider/PostgreSQLMappingProvider.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Fsource%2Freader%2Fio%2Fschema%2Ftypemapping%2Fprovider%2FPostgreSQLMappingProvider.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvc291cmNlL3JlYWRlci9pby9zY2hlbWEvdHlwZW1hcHBpbmcvcHJvdmlkZXIvUG9zdGdyZVNRTE1hcHBpbmdQcm92aWRlci5qYXZh) | `100.00% <100.00%> (ø)` | | | [...leport/v2/transformer/SourceRowToMutationDoFn.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Ftransformer%2FSourceRowToMutationDoFn.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvdHJhbnNmb3JtZXIvU291cmNlUm93VG9NdXRhdGlvbkRvRm4uamF2YQ==) | `97.82% <100.00%> (+0.04%)` | :arrow_up: | | [...ud/teleport/v2/options/OptionsToConfigBuilder.java](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree&filepath=v2%2Fsourcedb-to-spanner%2Fsrc%2Fmain%2Fjava%2Fcom%2Fgoogle%2Fcloud%2Fteleport%2Fv2%2Foptions%2FOptionsToConfigBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform#diff-djIvc291cmNlZGItdG8tc3Bhbm5lci9zcmMvbWFpbi9qYXZhL2NvbS9nb29nbGUvY2xvdWQvdGVsZXBvcnQvdjIvb3B0aW9ucy9PcHRpb25zVG9Db25maWdCdWlsZGVyLmphdmE=) | `90.90% <92.85%> (+3.76%)` | :arrow_up: | | ... and [3 more](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | | ... and [483 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1754/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)