search

apache / parquet-java

Apache Parquet Java
https://parquet.apache.org/
Apache License 2.0
2.49k stars 1.37k forks source link

Bump org.apache.arrow:arrow-vector from 16.0.0 to 16.1.0 #1354

Closed dependabot[bot] closed 1 month ago

dependabot[bot] commented 1 month ago

Bumps org.apache.arrow:arrow-vector from 16.0.0 to 16.1.0.

Commits
  • 7dd1d34 MINOR: [Release] Update versions for 16.1.0
  • ca3e9d1 MINOR: [Release] Update .deb/.rpm changelogs for 16.1.0
  • 63a4287 MINOR: [Release] Update CHANGELOG.md for 16.1.0
  • 5874287 GH-41594: [Go] Support reading date64 type & properly validate list-like ty...
  • 81940c6 GH-41431: [C++][Parquet][Dataset] Fix repeated scan on encrypted dataset (#41...
  • be62f6c GH-41577: [Java][Packaging] Add org.apache.arrow.memory.core to --add-opens=j...
  • 6cfebb9 GH-41562: [C++][Parquet] Decoding: Fix num_value handling in ByteStreamSplitD...
  • dcfeceb GH-41140: [C#] Account for offset and length in union arrays (#41165)
  • b28633c MINOR: [Release] Manually commit fix to avoid cherry-pick conflict for changi...
  • ac8af22 GH-41566: [CI][Packaging] macOS wheel for Catalina fails to build on macOS ar...
  • Additional commits viewable in compare view


Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options
You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
pjfanning commented 4 weeks ago

@Fokko Arrow 16.1.0 breaks Java 8 support. See https://github.com/apache/arrow/issues/41717

Fokko commented 4 weeks ago

@pjfanning That's not good, thanks for bringing it to the attention here. Looking at the stack trace it is around ipc, which I don't think we use in Parquet:

[info] org.apache.pekko.stream.connectors.googlecloud.bigquery.storage.scaladsl.BigQueryArrowStorageSpec *** ABORTED *** (0 milliseconds)
[info]   java.lang.NoSuchMethodError: java.nio.ByteBuffer.rewind()Ljava/nio/ByteBuffer;
[info]   at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:704)
[info]   at org.apache.arrow.vector.ipc.message.MessageSerializer.deserializeSchema(MessageSerializer.java:213)
[info]   at org.apache.pekko.stream.connectors.googlecloud.bigquery.storage.impl.SimpleRowReader.<init>(ArrowSource.scala:66)
[info]   at org.apache.pekko.stream.connectors.googlecloud.bigquery.storage.scaladsl.BigQueryArrowStorageSpec.$anonfun$new$1(BigQueryArrowStorageSpec.scala:35)
pjfanning commented 4 weeks ago

There is also https://github.com/apache/arrow/pull/41718#issuecomment-2119816144

java.lang.NoSuchMethodError: java.nio.ByteBuffer.position(I)Ljava/nio/ByteBuffer;
    at org.apache.arrow.memory.ArrowBuf.setBytes(ArrowBuf.java:907)
        at org.apache.arrow.vector.BaseVariableWidthVector.setSafe(BaseVariableWidthVector.java:1183)
Fokko commented 4 weeks ago

It looks like we're only using the type-system for schema conversion:

grep -R "import org.apache.arrow" . | sort | uniq
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.DateUnit;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.FloatingPointPrecision;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.TimeUnit;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.ArrowTypeVisitor;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Binary;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Bool;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Date;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Decimal;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.FloatingPoint;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Int;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Interval;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Null;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Struct;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Time;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Timestamp;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Union;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType.Utf8;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.Field;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.FieldType;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaConverter.java:import org.apache.arrow.vector.types.pojo.Schema;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaMapping.java:import org.apache.arrow.vector.types.pojo.Field;
./parquet-arrow/src/main/java/org/apache/parquet/arrow/schema/SchemaMapping.java:import org.apache.arrow.vector.types.pojo.Schema;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.DateUnit;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.FloatingPointPrecision;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.IntervalUnit;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.TimeUnit;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.UnionMode;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.pojo.ArrowType;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.pojo.Field;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.pojo.FieldType;
./parquet-arrow/src/test/java/org/apache/parquet/arrow/schema/TestSchemaConverter.java:import org.apache.arrow.vector.types.pojo.Schema;