search

apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.71k stars 4.2k forks source link

Update vendored calcite to eliminate vulnerability from shaded log4j:1.2.17 and protobuf-java:3.19.2 #26403

Open OlgaGorlova opened 1 year ago

OlgaGorlova commented 1 year ago

Hi team,

The beam-vendor-calcite-1_28_0 contains a bunch of shaded dependencies with major security vulnerabilities. For example, log4j:1.2.17 and protobuf-java:3.19.2.

Are there any plans to upgrade the vendored calcite?

We are using beam-sdks-java-extensions-sql:2.46.0 and beam-sdks-java-extensions-sql—zetasql:2.46.0 dependencies in our code. We use SqlTransform class which internally references classes from beam-vendor-calcite-1_28_0.

Looks like beam-vendor-calcite is created as fat-jar, and the log4j:1.2.17 is shaded into this beam-vendor-calcite-1_28_0:0.2. So, we cannot exclude log4j:1.2.17 from this vendor jar and we cannot even override it with the latest versions either. We cannot use any other beam-vendor-calcite version as beam-vendor-calcite-1_28_0:0.2 is the latest version. We cannot exclude beam-vendor-calcite because internally SqlTransform class references classes from beam-vendor-calcite-1_28_0 such as org.apache.beam.vendor.calcite.v1_28_0.com.google.common.collect.ImmutableList.

And, if we include this vendor jar, it gets flagged vulnerable by our security scan due to shaded log4j:1.2.17. Hence, it would be really helpful if the log4j:1.2.17 was excluded or replaced with its latest version in beam-vendor-calcite-1_28_0.

Please note that there are more vulnerable dependencies shaded in beam-vendor-calcite-1_28_0, but log4j:1.2.17 is currently the most critical one:

Abacn commented 1 year ago

vendor calcite 1.28 should not have shaded 1og4j. It is excluded:

https://github.com/apache/beam/blob/1a51ba5edebef9622c016331071764830adffc94/vendor/calcite-1_28_0/build.gradle#L94

org.apache.beam.vendor.calcite.v1_28_0.com.google.common.collect.ImmutableList is a misuse. It should be

org.apache.beam.vendor.guava.v26_xxx.com.google.common.collect.ImmutableList

Let me do an audit of com.google.common that is not under vendor guava space.

oleksandr-mashchenko-db commented 1 year ago

Thank you for the feedback, @Abacn. Did you have a chance to perform audit of com.google.common ?

Abacn commented 1 year ago

Did you have a chance to perform audit of com.google.common ?

Yes, done in #26463. The change will be available in Beam v2.48.0. Though I don't think this resolves the main issue of this Issue. Dot the log4j dependency, users should be able to overwrite the dependency to a newer version because it is not shaded in vendor-calcite-1.28.0. Will confirm.

OlgaGorlova commented 1 year ago

Hi @Abacn, thank you for looking into this. You're right, I also don't think it will resolve the issue with security scans. However, our security scan is finding the reference to log4j under beam-vendor-calcite-1_28_0-0.2.jar/META-INF/maven/log4j/* . Looks like the package is built as a fat jar. Is there a reason for that?

Abacn commented 1 year ago

That is the maven metadata file (pom.xml) for log4j. The fat jar does not include log4j. beam-vendor-calcite-1_28_0-0.2.jar is our vendored dependency. Beam vendors a few dependencies which shaded all necessary transitive dependencies in a fat jar. This avoids conflict introduced by user dependencies.

I setup a project using SqlTransform, and running mvn dependency:tree there is no log4j dependency:

[INFO] com.github.yathu:dependencyTriageTest:jar:1.0-SNAPSHOT
[INFO] +- org.apache.beam:beam-sdks-java-core:jar:2.46.0:compile
[INFO] |  +- org.apache.beam:beam-model-pipeline:jar:2.46.0:compile
[INFO] |  |  \- org.conscrypt:conscrypt-openjdk-uber:jar:2.5.2:compile
[INFO] |  +- org.apache.beam:beam-model-job-management:jar:2.46.0:compile
[INFO] |  +- org.apache.beam:beam-vendor-grpc-1_48_1:jar:0.1:compile
[INFO] |  +- org.apache.beam:beam-vendor-guava-26_0-jre:jar:0.1:compile
[INFO] |  +- net.bytebuddy:byte-buddy:jar:1.12.14:compile
[INFO] |  +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] |  +- com.google.errorprone:error_prone_annotations:jar:2.10.0:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.14.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.14.1:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.14.1:compile
[INFO] |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.7:compile
[INFO] |  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  +- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile
[INFO] |  \- joda-time:joda-time:jar:2.10.10:compile
[INFO] +- org.apache.beam:beam-sdks-java-extensions-sql:jar:2.46.0:compile
[INFO] |  +- org.checkerframework:checker-qual:jar:3.27.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-avro:jar:2.46.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-join-library:jar:2.46.0:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-sql-udf:jar:2.46.0:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.15:compile
[INFO] |  +- org.apache.commons:commons-csv:jar:1.8:compile
[INFO] |  +- org.apache.beam:beam-vendor-calcite-1_28_0:jar:0.2:compile
[INFO] |  +- com.alibaba:fastjson:jar:1.2.69:compile
[INFO] |  +- org.codehaus.janino:janino:jar:3.0.11:compile
[INFO] |  +- org.codehaus.janino:commons-compiler:jar:3.0.11:compile
[INFO] |  +- org.mongodb:mongo-java-driver:jar:3.12.11:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.46.0:compile
[INFO] |  |  +- com.google.cloud.bigdataoss:gcsio:jar:2.2.6:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-storage-v2:jar:2.2.2-alpha:compile
[INFO] |  |  |  |  \- com.google.api.grpc:proto-google-cloud-storage-v2:jar:2.2.2-alpha:compile
[INFO] |  |  |  +- io.opencensus:opencensus-impl:jar:0.31.0:compile
[INFO] |  |  |  |  +- io.opencensus:opencensus-impl-core:jar:0.31.0:compile
[INFO] |  |  |  |  \- com.lmax:disruptor:jar:3.4.2:compile
[INFO] |  |  |  +- io.opencensus:opencensus-exporter-stats-stackdriver:jar:0.31.0:compile
[INFO] |  |  |  |  +- io.opencensus:opencensus-contrib-exemplar-util:jar:0.31.0:compile
[INFO] |  |  |  |  +- io.opencensus:opencensus-contrib-resource-util:jar:0.31.0:compile
[INFO] |  |  |  |  \- io.opencensus:opencensus-exporter-metrics-util:jar:0.31.0:compile
[INFO] |  |  |  \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.31.0:compile
[INFO] |  |  +- com.google.apis:google-api-services-cloudresourcemanager:jar:v1-rev20220828-2.0.0:compile
[INFO] |  |  +- org.apache.httpcomponents:httpcore:jar:4.4.14:compile
[INFO] |  |  \- org.apache.httpcomponents:httpclient:jar:4.5.13:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.46.0:compile
[INFO] |  |  +- org.apache.beam:beam-sdks-java-expansion-service:jar:2.46.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.14.1:compile
[INFO] |  |  |     \- org.yaml:snakeyaml:jar:1.33:compile
[INFO] |  |  +- org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.46.0:compile
[INFO] |  |  +- org.apache.beam:beam-sdks-java-extensions-arrow:jar:2.46.0:compile
[INFO] |  |  +- com.google.api:gax:jar:2.23.0:compile
[INFO] |  |  +- com.google.api:gax-grpc:jar:2.23.0:compile
[INFO] |  |  +- com.google.api:gax-grpc:jar:testlib:2.23.0:compile
[INFO] |  |  +- com.google.api:gax-httpjson:jar:0.108.0:compile
[INFO] |  |  +- com.google.api:api-common:jar:2.6.0:compile
[INFO] |  |  +- com.google.apis:google-api-services-healthcare:jar:v1-rev20220818-2.0.0:compile
[INFO] |  |  +- com.google.apis:google-api-services-pubsub:jar:v1-rev20220904-2.0.0:compile
[INFO] |  |  +- com.google.cloud:google-cloud-bigquerystorage:jar:2.31.0:compile
[INFO] |  |  |  +- com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:jar:0.155.0:compile
[INFO] |  |  |  +- com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta2:jar:0.155.0:compile
[INFO] |  |  |  +- org.json:json:jar:20220924:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:jar:0.155.0:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta2:jar:0.155.0:compile
[INFO] |  |  |  \- com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1:jar:2.31.0:compile
[INFO] |  |  +- com.google.cloud.bigtable:bigtable-client-core:jar:1.26.3:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-common-protos:jar:2.3.2:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-bigtable-v2:jar:1.27.1:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:jar:1.27.1:compile
[INFO] |  |  |  +- com.google.cloud:google-cloud-core-http:jar:1.95.4:compile
[INFO] |  |  |  |  \- com.google.http-client:google-http-client-appengine:jar:1.39.2:compile
[INFO] |  |  |  +- com.google.cloud.bigtable:bigtable-metrics-api:jar:1.26.3:compile
[INFO] |  |  |  +- io.opencensus:opencensus-contrib-grpc-util:jar:0.28.0:compile
[INFO] |  |  |  \- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
[INFO] |  |  +- com.google.cloud:google-cloud-bigtable:jar:2.19.0:compile
[INFO] |  |  |  +- com.google.cloud:google-cloud-bigtable-stats:jar:2.19.0:compile
[INFO] |  |  |  +- io.grpc:grpc-rls:jar:1.52.1:runtime
[INFO] |  |  |  +- com.google.cloud:google-cloud-monitoring:jar:3.11.0:compile
[INFO] |  |  |  \- com.google.api.grpc:proto-google-cloud-monitoring-v3:jar:3.11.0:runtime
[INFO] |  |  +- com.google.cloud:google-cloud-core:jar:2.10.0:compile
[INFO] |  |  +- com.google.cloud:google-cloud-core-grpc:jar:2.10.0:compile
[INFO] |  |  +- com.google.cloud.datastore:datastore-v1-proto-client:jar:2.9.0:compile
[INFO] |  |  |  \- com.google.http-client:google-http-client-protobuf:jar:1.41.8:compile
[INFO] |  |  +- com.google.cloud:google-cloud-firestore:jar:3.7.10:compile
[INFO] |  |  |  \- com.google.cloud:proto-google-cloud-firestore-bundle-v1:jar:3.7.10:compile
[INFO] |  |  +- com.google.cloud:google-cloud-pubsub:jar:1.123.2:compile
[INFO] |  |  +- com.google.cloud:google-cloud-spanner:jar:6.36.0:compile
[INFO] |  |  |  +- com.google.cloud:grpc-gcp:jar:1.3.2:compile
[INFO] |  |  |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:6.36.0:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:6.36.0:compile
[INFO] |  |  |  +- com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:6.36.0:compile
[INFO] |  |  |  \- com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:6.36.0:compile
[INFO] |  |  +- com.google.code.gson:gson:jar:2.9.1:compile
[INFO] |  |  +- com.google.oauth-client:google-oauth-client:jar:1.34.1:compile
[INFO] |  |  +- io.grpc:grpc-alts:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-api:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-auth:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-core:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-census:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-context:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-grpclb:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-netty:jar:1.52.1:compile
[INFO] |  |  |  +- io.netty:netty-codec-http2:jar:4.1.79.Final:compile
[INFO] |  |  |  |  \- io.netty:netty-codec-http:jar:4.1.79.Final:compile
[INFO] |  |  |  +- io.netty:netty-handler-proxy:jar:4.1.79.Final:runtime
[INFO] |  |  |  |  \- io.netty:netty-codec-socks:jar:4.1.79.Final:runtime
[INFO] |  |  |  \- io.netty:netty-transport-native-unix-common:jar:4.1.79.Final:runtime
[INFO] |  |  +- io.grpc:grpc-netty-shaded:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-stub:jar:1.52.1:compile
[INFO] |  |  +- io.grpc:grpc-xds:jar:1.52.1:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:1.105.2:compile
[INFO] |  |  +- com.google.api.grpc:grpc-google-cloud-pubsublite-v1:jar:1.10.0:compile
[INFO] |  |  +- com.google.guava:guava:jar:31.1-jre:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.14.1:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.14.1:compile
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.77.Final:compile
[INFO] |  |  |  +- io.netty:netty-common:jar:4.1.77.Final:compile
[INFO] |  |  |  +- io.netty:netty-resolver:jar:4.1.77.Final:compile
[INFO] |  |  |  +- io.netty:netty-buffer:jar:4.1.77.Final:compile
[INFO] |  |  |  +- io.netty:netty-transport:jar:4.1.77.Final:compile
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.77.Final:compile
[INFO] |  |  +- io.netty:netty-tcnative-boringssl-static:jar:2.0.52.Final:compile
[INFO] |  |  |  +- io.netty:netty-tcnative-classes:jar:2.0.52.Final:compile
[INFO] |  |  |  +- io.netty:netty-tcnative-boringssl-static:jar:linux-x86_64:2.0.52.Final:compile
[INFO] |  |  |  +- io.netty:netty-tcnative-boringssl-static:jar:linux-aarch_64:2.0.52.Final:compile
[INFO] |  |  |  +- io.netty:netty-tcnative-boringssl-static:jar:osx-x86_64:2.0.52.Final:compile
[INFO] |  |  |  +- io.netty:netty-tcnative-boringssl-static:jar:osx-aarch_64:2.0.52.Final:compile
[INFO] |  |  |  \- io.netty:netty-tcnative-boringssl-static:jar:windows-x86_64:2.0.52.Final:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-bigquerystorage-v1:jar:2.31.0:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:jar:2.19.0:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-datastore-v1:jar:0.104.4:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-firestore-v1:jar:3.7.10:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:1.105.2:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:6.36.0:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-cloud-spanner-v1:jar:6.36.0:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-common-protos:jar:2.14.0:compile
[INFO] |  |  +- org.apache.arrow:arrow-memory-core:jar:5.0.0:compile
[INFO] |  |  +- org.apache.arrow:arrow-vector:jar:5.0.0:compile
[INFO] |  |  |  +- org.apache.arrow:arrow-format:jar:5.0.0:compile
[INFO] |  |  |  \- com.google.flatbuffers:flatbuffers-java:jar:1.12.0:compile
[INFO] |  |  +- com.google.http-client:google-http-client-gson:jar:1.41.2:compile
[INFO] |  |  \- org.threeten:threetenbp:jar:1.4.4:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-bigtable-v2:jar:2.19.0:compile
[INFO] |  |  +- com.google.auto.value:auto-value-annotations:jar:1.10.1:compile
[INFO] |  |  +- com.google.auto.value:auto-value:jar:1.10.1:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] |  |  +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] |  |  \- com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] |  +- com.google.apis:google-api-services-bigquery:jar:v2-rev20220924-2.0.0:compile
[INFO] |  +- com.google.api.grpc:proto-google-cloud-pubsublite-v1:jar:1.10.0:compile
[INFO] |  +- com.google.cloud:google-cloud-pubsublite:jar:1.10.0:compile
[INFO] |  |  +- io.opencensus:opencensus-api:jar:0.31.1:compile
[INFO] |  |  +- io.grpc:grpc-protobuf-lite:jar:1.52.1:compile
[INFO] |  |  +- com.google.api.grpc:proto-google-iam-v1:jar:1.9.0:compile
[INFO] |  |  +- io.grpc:grpc-googleapis:jar:1.52.1:runtime
[INFO] |  |  +- io.opencensus:opencensus-proto:jar:0.2.0:compile
[INFO] |  |  +- io.grpc:grpc-services:jar:1.52.1:compile
[INFO] |  |  +- com.google.re2j:re2j:jar:1.6:compile
[INFO] |  |  +- com.google.android:annotations:jar:4.1.1.4:runtime
[INFO] |  |  +- org.codehaus.mojo:animal-sniffer-annotations:jar:1.22:runtime
[INFO] |  |  +- io.perfmark:perfmark-api:jar:0.26.0:runtime
[INFO] |  |  +- commons-logging:commons-logging:jar:1.2:compile
[INFO] |  |  +- io.opencensus:opencensus-contrib-http-util:jar:0.31.1:compile
[INFO] |  |  +- com.google.flogger:google-extensions:jar:0.7.4:compile
[INFO] |  |  +- com.google.flogger:flogger:jar:0.7.4:compile
[INFO] |  |  \- org.checkerframework:checker-compat-qual:jar:2.5.3:compile
[INFO] |  +- org.apache.beam:beam-sdks-java-io-mongodb:jar:2.46.0:compile
[INFO] |  +- com.google.protobuf:protobuf-java:jar:3.21.12:compile
[INFO] |  \- com.google.protobuf:protobuf-java-util:jar:3.21.12:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] +- org.slf4j:slf4j-jdk14:jar:1.7.30:runtime
[INFO] +- org.hamcrest:hamcrest-core:jar:2.1:compile
[INFO] |  \- org.hamcrest:hamcrest:jar:2.1:compile
[INFO] +- org.hamcrest:hamcrest-library:jar:2.1:compile
[INFO] +- junit:junit:jar:4.13-beta-3:compile
[INFO] +- org.apache.beam:beam-runners-direct-java:jar:2.46.0:runtime
[INFO] +- org.mockito:mockito-core:jar:3.7.7:test
[INFO] |  +- net.bytebuddy:byte-buddy-agent:jar:1.10.19:test
[INFO] |  \- org.objenesis:objenesis:jar:3.1:runtime
[INFO] +- org.apache.beam:beam-runners-portability-java:jar:2.46.0:runtime
[INFO] |  +- org.apache.beam:beam-runners-java-fn-execution:jar:2.46.0:compile
[INFO] |  +- org.apache.beam:beam-runners-java-job-service:jar:2.46.0:runtime
[INFO] |  \- org.apache.beam:beam-sdks-java-harness:jar:2.46.0:runtime
[INFO] +- org.apache.beam:beam-runners-flink-1.14:jar:2.46.0:runtime
[INFO] |  +- org.apache.beam:beam-runners-core-java:jar:2.46.0:compile
[INFO] |  +- org.apache.beam:beam-runners-core-construction-java:jar:2.46.0:compile
[INFO] |  |  \- io.github.classgraph:classgraph:jar:4.8.104:compile
[INFO] |  +- args4j:args4j:jar:2.33:runtime
[INFO] |  +- org.apache.flink:flink-clients_2.12:jar:1.14.3:runtime
[INFO] |  |  +- commons-cli:commons-cli:jar:1.3.1:runtime
[INFO] |  |  \- org.apache.flink:flink-shaded-force-shading:jar:14.0:runtime
[INFO] |  +- org.apache.flink:flink-streaming-java_2.12:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-file-sink-common:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-scala_2.12:jar:1.14.3:runtime
[INFO] |  |  |  +- org.scala-lang:scala-reflect:jar:2.12.7:runtime
[INFO] |  |  |  +- org.scala-lang:scala-library:jar:2.12.7:runtime
[INFO] |  |  |  +- org.scala-lang:scala-compiler:jar:2.12.7:runtime
[INFO] |  |  |  |  \- org.scala-lang.modules:scala-xml_2.12:jar:1.0.6:runtime
[INFO] |  |  |  \- com.twitter:chill_2.12:jar:0.7.6:runtime
[INFO] |  |  |     \- com.twitter:chill-java:jar:0.7.6:runtime
[INFO] |  |  +- org.apache.flink:flink-shaded-guava:jar:30.1.1-jre-14.0:runtime
[INFO] |  |  \- org.apache.commons:commons-math3:jar:3.5:runtime
[INFO] |  +- org.apache.flink:flink-core:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-annotations:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-shaded-asm-7:jar:7.1-14.0:runtime
[INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.3.2:runtime
[INFO] |  |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:runtime
[INFO] |  |  |  \- com.esotericsoftware.minlog:minlog:jar:1.2:runtime
[INFO] |  |  \- commons-collections:commons-collections:jar:3.2.2:runtime
[INFO] |  +- org.apache.flink:flink-metrics-core:jar:1.14.3:runtime
[INFO] |  +- org.apache.flink:flink-java:jar:1.14.3:runtime
[INFO] |  +- org.apache.flink:flink-runtime:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-rpc-core:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-rpc-akka-loader:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-queryable-state-client-java:jar:1.14.3:runtime
[INFO] |  |  +- org.apache.flink:flink-hadoop-fs:jar:1.14.3:runtime
[INFO] |  |  +- commons-io:commons-io:jar:2.8.0:runtime
[INFO] |  |  +- org.apache.flink:flink-shaded-netty:jar:4.1.65.Final-14.0:runtime
[INFO] |  |  +- org.apache.flink:flink-shaded-jackson:jar:2.12.4-14.0:runtime
[INFO] |  |  +- org.apache.flink:flink-shaded-zookeeper-3:jar:3.4.14-14.0:runtime
[INFO] |  |  +- org.javassist:javassist:jar:3.24.0-GA:runtime
[INFO] |  |  \- org.lz4:lz4-java:jar:1.8.0:runtime
[INFO] |  +- org.apache.flink:flink-optimizer:jar:1.14.3:runtime
[INFO] |  +- org.apache.beam:beam-model-fn-execution:jar:2.46.0:compile
[INFO] |  \- org.apache.beam:beam-sdks-java-fn-execution:jar:2.46.0:compile
[INFO] \- org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.46.0:runtime
[INFO]    +- org.apache.beam:beam-sdks-java-io-kafka:jar:2.46.0:runtime
[INFO]    |  +- org.springframework:spring-expression:jar:5.3.25:runtime
[INFO]    |  |  \- org.springframework:spring-core:jar:5.3.25:runtime
[INFO]    |  |     \- org.springframework:spring-jcl:jar:5.3.25:runtime
[INFO]    |  +- io.confluent:kafka-avro-serializer:jar:5.3.2:runtime
[INFO]    |  |  +- io.confluent:common-config:jar:5.3.2:runtime
[INFO]    |  |  \- io.confluent:common-utils:jar:5.3.2:runtime
[INFO]    |  |     \- com.101tec:zkclient:jar:0.10:runtime
[INFO]    |  \- io.confluent:kafka-schema-registry-client:jar:5.3.2:runtime
[INFO]    +- com.google.cloud.bigdataoss:util:jar:2.2.6:compile
[INFO]    |  +- com.google.api-client:google-api-client-jackson2:jar:1.32.2:compile
[INFO]    |  \- com.google.apis:google-api-services-iamcredentials:jar:v1-rev20210326-1.32.1:compile
[INFO]    +- com.google.flogger:flogger-system-backend:jar:0.7.3:compile
[INFO]    +- com.google.api-client:google-api-client:jar:2.0.0:compile
[INFO]    |  \- com.google.http-client:google-http-client-apache-v2:jar:1.42.1:compile
[INFO]    +- com.google.apis:google-api-services-clouddebugger:jar:v2-rev20220318-2.0.0:runtime
[INFO]    +- com.google.apis:google-api-services-dataflow:jar:v1b3-rev20220920-2.0.0:runtime
[INFO]    +- com.google.apis:google-api-services-storage:jar:v1-rev20220705-2.0.0:compile
[INFO]    +- com.google.auth:google-auth-library-credentials:jar:1.15.0:compile
[INFO]    +- com.google.auth:google-auth-library-oauth2-http:jar:1.15.0:compile
[INFO]    +- com.google.http-client:google-http-client:jar:1.42.3:compile
[INFO]    \- com.google.http-client:google-http-client-jackson2:jar:1.42.3:compile
DebajitKumarPhukan commented 9 months ago

@Abacn FYI, below "critical" vulnerabilities still exist along with the log4j vulnerability and are getting blocked by JFrog XRay scans.

org.yaml:snakeyaml:1.24 com.fasterxml.jackson.core:jackson-databind:2.10.0

Apache Beam SQL Extension Version

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-extensions-sql</artifactId>
    <version>2.51.0</version>
</dependency>

Which Internally Uses Below Version

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-vendor-calcite-1_28_0</artifactId>
    <version>0.2</version>
</dependency>
Abacn commented 9 months ago

@Abacn FYI, below "critical" vulnerabilities still exist along with the log4j vulnerability and are getting blocked by JFrog XRay scans.

org.yaml:snakeyaml:1.24

com.fasterxml.jackson.core:jackson-databind:2.10.0

Apache Beam SQL Extension Version


<dependency>

    <groupId>org.apache.beam</groupId>

    <artifactId>beam-sdks-java-extensions-sql</artifactId>

    <version>2.51.0</version>

</dependency>

Which Internally Uses Below Version


<dependency>

    <groupId>org.apache.beam</groupId>

    <artifactId>beam-vendor-calcite-1_28_0</artifactId>

    <version>0.2</version>

</dependency>

Thanks for reminding, both jackson and snakeYAML vulnerabilities are indeed well-known and should be fixed by upgrades. The obstacle is that upgrading calcite for beam SQL is known to be nontrivial due to breaking changes (happened last time vendor upgrade) so it is expected to take more effort than the other two vendor dependency beam currently does. We currently do not have bandwidth and encourage contributors could test with new calcite versions.

Generally, vendor calcite and its dependency are shaded and only used by beam SQL artifact and vulnerabilities usually not exploitable (disclaimer - not professional opinion about security). Other part of beam and user code do not depend on the dependencies of vulnerabilities. e.g. Beam Java core depends on more recent version of jackson databind.

Nevertheless it should be possible to upgrade these transitive dependencies and keep calcite version which requires much less effort. Could initiate that when I got time.