GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

Upgrade jackson version #1739

Closed Polber closed 1 month ago

Polber commented 1 month ago

Apache Beam depends on Jackson 2.15.4 as of https://github.com/apache/beam/pull/31473 which is breaking at least the JMS To PubSub template when deserializing pipeline options due to

java.lang.NoSuchFieldError: READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE
  at com.fasterxml.jackson.databind.deser.std.EnumDeserializer.createContextual(EnumDeserializer.java:211)
  ...
  at org.apache.beam.sdk.options.PipelineOptionsFactory.computeDeserializerForMethod(PipelineOptionsFactory.java:1739)
  ...

which was a field added in 2.15.4

Polber commented 1 month ago

Not sure why checks aren't running. Looking into it

Polber commented 1 month ago

Might be this https://www.githubstatus.com/

Abacn commented 1 month ago

wait, at least need to add this line in several places per #1681

RowJsonUtils.increaseDefaultStreamReadConstraints(100 * 1024 * 1024);
Polber commented 1 month ago

Job is queued: https://github.com/GoogleCloudPlatform/DataflowTemplates/actions/runs/10000088771 Definitely a github issue. Don't love that it doesn't block submission

Abacn commented 1 month ago

there will be integration test failures which I observed during 2.57.0rc1 validation, need to fix some code path, see https://github.com/GoogleCloudPlatform/DataflowTemplates/pull/1739#issuecomment-2237776829

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 42.31%. Comparing base (d7db191) to head (84d72c6).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1739 +/- ## ============================================ - Coverage 42.32% 42.31% -0.01% + Complexity 3182 3180 -2 ============================================ Files 794 794 Lines 46244 46244 Branches 4951 4951 ============================================ - Hits 19572 19568 -4 - Misses 25074 25077 +3 - Partials 1598 1599 +1 ``` | [Components](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | Coverage Δ | | |---|---|---| | [spanner-templates](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `63.69% <ø> (-0.01%)` | :arrow_down: | | [spanner-import-export](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `64.42% <ø> (-0.03%)` | :arrow_down: | | [spanner-live-forward-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `74.97% <ø> (ø)` | | | [spanner-live-reverse-replication](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `51.84% <ø> (ø)` | | | [spanner-bulk-migration](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform) | `82.79% <ø> (ø)` | | [see 2 files with indirect coverage changes](https://app.codecov.io/gh/GoogleCloudPlatform/DataflowTemplates/pull/1739/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleCloudPlatform)
Polber commented 1 month ago

there will be integration test failures which I observed during 2.57.0rc1 validation, need to fix some code path, see #1739 (comment)

@Abacn I already included these changes when I upgraded beam in #1695.

IT's passed other than a couple Kafka IT's which have already been failing - I'm looking into those, but error is unrelated to the jackson upgrade.