Release Apache ORC 1.8.5

wgtmac commented 1 year ago

[x] GitHub Action CI Check (Java 8/11/17) and Ubuntu/MacOS/Windows.
- https://github.com/apache/orc/actions/workflows/build_and_test.yml?query=branch%3Abranch-1.8

[x] Docker Tests (CentOS7, Debian10/11, Ubuntu18/20/22, Fedora37)


./run-all.sh local branch-1.8
Launching centos7 as
Launching debian10 as
Launching debian11 as
Launching ubuntu18 as
Launching ubuntu20 as
Launching ubuntu22 as
Launching fedora37 as
Launching debian10_jdk=11 as
Launching ubuntu20_jdk=11 as
Launching ubuntu20_jdk=11_cc=clang as

Test start: Thu Aug 31 11:30:20 CST 2023 End: Thu Aug 31 13:25:46 CST 2023

tail -n5 logs/test ==> logs/centos7-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 161.36 sec Built target test-out Finished centos7 at Thu Aug 31 11:42:05 CST 2023

==> logs/debian10-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 163.23 sec Built target test-out Finished debian10 at Thu Aug 31 11:53:39 CST 2023

==> logs/debian10_jdk=11-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 160.73 sec Built target test-out Finished debian10_jdk=11 at Thu Aug 31 13:03:45 CST 2023

==> logs/debian11-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 159.30 sec Built target test-out Finished debian11 at Thu Aug 31 12:05:43 CST 2023

==> logs/fedora37-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 161.55 sec Built target test-out Finished fedora37 at Thu Aug 31 12:52:21 CST 2023

==> logs/ubuntu18-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 150.69 sec Built target test-out Finished ubuntu18 at Thu Aug 31 12:16:18 CST 2023

==> logs/ubuntu20-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 154.95 sec Built target test-out Finished ubuntu20 at Thu Aug 31 12:28:25 CST 2023

==> logs/ubuntu20_jdk=11-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 165.14 sec Built target test-out Finished ubuntu20_jdk=11 at Thu Aug 31 13:16:02 CST 2023

==> logs/ubuntu20_jdk=11_cc=clang-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 164.70 sec Built target test-out Finished ubuntu20_jdk=11_cc=clang at Thu Aug 31 13:25:46 CST 2023

==> logs/ubuntu22-test.log <== 100% tests passed, 0 tests failed out of 8

Total Test time (real) = 155.67 sec Built target test-out Finished ubuntu22 at Thu Aug 31 12:39:52 CST 2023


- [x] [Apache Spark 3.4.x Integration Test with 1.8.5 SNAPSHOT](https://github.com/wgtmac/spark/pull/4)
- [x] [Apache Iceberg 1.2.x Integration Test with 1.8.5 SNAPSHOT](https://github.com/wgtmac/iceberg/pull/4)
- [x] [Vote Tag Creation](https://github.com/apache/orc/releases/tag/v1.8.5-rc0)
- [x] [Upload RC0 Vote Artifacts](https://dist.apache.org/repos/dist/dev/orc/v1.8.5-rc0)
- [x] [Send RC0 Vote Email](https://lists.apache.org/thread/pnhho9d20g74orhqtdl2yrzbcd7m54od)
- [x] [Apache Spark 3.4.x Integration Integration Test with RC0](https://github.com/wgtmac/spark/pull/5)
- [x] [Apache Iceberg 1.2.x Integration Test with RC0](https://github.com/wgtmac/iceberg/pull/5)
- [x] [Send RC0 Vote Result](https://lists.apache.org/thread/27yfnm3822tt3nq9fqcvnm1dx6nrxn7d)
- [x] [Publish to Apache Download](https://www.apache.org/dyn/closer.cgi/orc/orc-1.8.5/orc-1.8.5.tar.gz)
- [x] [Release Tag Creation](https://github.com/apache/orc/releases/tag/v1.8.5)
- [x] Publish to Maven Central
- [x] Create next milestone
- [x] Register `1.8.5` to https://reporter.apache.org/addrelease.html?orc

dongjoon-hyun commented 1 year ago

Thank you, @wgtmac .

BTW, it seems that there is no new commit in branch-1.8 since v1.8.4 (@guiyanakuang )

Do you want to backport some?

dongjoon-hyun commented 1 year ago

Also, cc @williamhyun .

If there is no community need to release v1.8.5 now, we can delay the due date of this milestone from September 16, 2023 to some future date.

wgtmac commented 1 year ago

Yes, I have noticed the same thing. Let me spend some time to see if there is anything required to backport. @dongjoon-hyun

wgtmac commented 1 year ago

After a rough check, I didn't see any commit that worth backporting (bug fix or minor improvement). So I suggest to delay the release if this does not change till the due date. @dongjoon-hyun @williamhyun

dongjoon-hyun commented 1 year ago

Got it. +1 for your decision, @wgtmac .

wgtmac commented 1 year ago

It seems that I have a lot of issues running the docker test on the branch-1.8.

tail -n10 logs/*test*
==> logs/centos7-test.log <==
[INFO] No dependency problems found
[INFO]
[INFO] --- maven-assembly-plugin:3.4.1:single (make-assembly) @ orc-benchmarks-hive ---
[INFO] Reading assembly descriptor: src/assembly/uber.xml
./mvnw: line 102:  5650 Killed                  "${MVN_BIN}" "$@"
make[2]: *** [java/CMakeFiles/java_build.dir/build.make:64: java/shims/orc-shims-1.8.5-SNAPSHOT.jar] Error 137
make[2]: *** Deleting file 'java/shims/orc-shims-1.8.5-SNAPSHOT.jar'
make[1]: *** [CMakeFiles/Makefile2:591: java/CMakeFiles/java_build.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
FAILED centos7

==> logs/debian10-test.log <==
[ 70%] Building CXX object lib/fs/CMakeFiles/fs_obj.dir/filesystem_sync.cc.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [lib/fs/CMakeFiles/fs_obj.dir/build.make:76: lib/fs/CMakeFiles/fs_obj.dir/filesystem_sync.cc.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:1421: lib/fs/CMakeFiles/fs_obj.dir/all] Error 2
make[3]: *** [Makefile:141: all] Error 2
make[2]: *** [CMakeFiles/libhdfspp_ep.dir/build.make:113: libhdfspp_ep-prefix/src/libhdfspp_ep-stamp/libhdfspp_ep-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:295: CMakeFiles/libhdfspp_ep.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
FAILED debian10

==> logs/debian10_jdk=11-test.log <==
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ orc-core ---
[INFO] Compiling 93 source files to /root/build/java/core/test-classes
./mvnw: line 102:  6730 Killed                  "${MVN_BIN}" "$@"
make[2]: *** [java/CMakeFiles/java_build.dir/build.make:64: java/shims/orc-shims-1.8.5-SNAPSHOT.jar] Error 137
make[2]: *** Deleting file 'java/shims/orc-shims-1.8.5-SNAPSHOT.jar'
make[1]: *** [CMakeFiles/Makefile2:591: java/CMakeFiles/java_build.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
FAILED debian10_jdk=11

==> logs/debian11-test.log <==
[ 21%] Building CXX object lib/proto/CMakeFiles/proto_obj.dir/ClientNamenodeProtocol.pb.cc.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [lib/proto/CMakeFiles/proto_obj.dir/build.make:218: lib/proto/CMakeFiles/proto_obj.dir/ClientNamenodeProtocol.pb.cc.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:1537: lib/proto/CMakeFiles/proto_obj.dir/all] Error 2
make[3]: *** [Makefile:160: all] Error 2
make[2]: *** [CMakeFiles/libhdfspp_ep.dir/build.make:132: libhdfspp_ep-prefix/src/libhdfspp_ep-stamp/libhdfspp_ep-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:447: CMakeFiles/libhdfspp_ep.dir/all] Error 2
make: *** [Makefile:182: all] Error 2
FAILED debian11

==> logs/fedora37-test.log <==
[ 21%] Building CXX object lib/proto/CMakeFiles/proto_obj.dir/ClientNamenodeProtocol.pb.cc.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [lib/proto/CMakeFiles/proto_obj.dir/build.make:214: lib/proto/CMakeFiles/proto_obj.dir/ClientNamenodeProtocol.pb.cc.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:1449: lib/proto/CMakeFiles/proto_obj.dir/all] Error 2
make[3]: *** [Makefile:146: all] Error 2
make[2]: *** [CMakeFiles/libhdfspp_ep.dir/build.make:86: libhdfspp_ep-prefix/src/libhdfspp_ep-stamp/libhdfspp_ep-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:405: CMakeFiles/libhdfspp_ep.dir/all] Error 2
make: *** [Makefile:166: all] Error 2
FAILED fedora37

==> logs/ubuntu18-test.log <==
[INFO] Excluding org.slf4j:slf4j-simple:jar:1.7.36 from the shaded jar.
./mvnw: line 102:  6626 Killed                  "${MVN_BIN}" "$@"
make[2]: *** [java/shims/orc-shims-1.8.5-SNAPSHOT.jar] Error 137
java/CMakeFiles/java_build.dir/build.make:63: recipe for target 'java/shims/orc-shims-1.8.5-SNAPSHOT.jar' failed
make[2]: *** Deleting file 'java/shims/orc-shims-1.8.5-SNAPSHOT.jar'
CMakeFiles/Makefile2:585: recipe for target 'java/CMakeFiles/java_build.dir/all' failed
make[1]: *** [java/CMakeFiles/java_build.dir/all] Error 2
Makefile:162: recipe for target 'all' failed
make: *** [all] Error 2
FAILED ubuntu18

==> logs/ubuntu20-test.log <==
[INFO]
[INFO] --- maven-assembly-plugin:3.4.1:single (make-assembly) @ orc-benchmarks-core ---
[INFO] Reading assembly descriptor: src/assembly/uber.xml
[INFO] Building jar: /root/build/java/bench/core/orc-benchmarks-core-1.8.5-SNAPSHOT-uber.jar
./mvnw: line 102:  6008 Killed                  "${MVN_BIN}" "$@"
make[2]: *** [java/CMakeFiles/java_build.dir/build.make:64: java/shims/orc-shims-1.8.5-SNAPSHOT.jar] Error 137
make[2]: *** Deleting file 'java/shims/orc-shims-1.8.5-SNAPSHOT.jar'
make[1]: *** [CMakeFiles/Makefile2:581: java/CMakeFiles/java_build.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
FAILED ubuntu20

==> logs/ubuntu20_jdk=11-test.log <==
[INFO] Including org.tukaani:xz:jar:1.9 in the shaded jar.
[INFO] Including org.jodd:jodd-core:jar:3.5.2 in the shaded jar.
[INFO] Including org.openjdk.jmh:jmh-core:jar:1.20 in the shaded jar.
[INFO] Including net.sf.jopt-simple:jopt-simple:jar:4.6 in the shaded jar.
[INFO] Including org.openjdk.jmh:jmh-generator-annprocess:jar:1.20 in the shaded jar.
[INFO] Including org.scala-lang:scala-library:jar:2.12.15 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.14.2 in the shaded jar.
[INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2 in the shaded jar.
[INFO] Including org.objenesis:objenesis:jar:3.2 in the shaded jar.

==> logs/ubuntu20_jdk=11_cc=clang-test.log <==
      value = readByte();
            ~ ^~~~~~~~~~
/root/orc/c++/src/ByteRLE.cc:469:36: error: implicit conversion changes signedness: 'signed char' to 'char' [-Werror,-Wsign-conversion]
              data[position + i] = readByte();
                                 ~ ^~~~~~~~~~
3 errors generated.
make[2]: *** [c++/src/CMakeFiles/orc.dir/build.make:213: c++/src/CMakeFiles/orc.dir/ByteRLE.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:487: c++/src/CMakeFiles/orc.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
FAILED ubuntu20_jdk=11_cc=clang

==> logs/ubuntu22-test.log <==
[ 21%] Building CXX object lib/proto/CMakeFiles/proto_obj.dir/ClientNamenodeProtocol.pb.cc.o
c++: fatal error: Killed signal terminated program cc1plus
compilation terminated.
make[5]: *** [lib/proto/CMakeFiles/proto_obj.dir/build.make:214: lib/proto/CMakeFiles/proto_obj.dir/ClientNamenodeProtocol.pb.cc.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:1449: lib/proto/CMakeFiles/proto_obj.dir/all] Error 2
make[3]: *** [Makefile:146: all] Error 2
make[2]: *** [CMakeFiles/libhdfspp_ep.dir/build.make:86: libhdfspp_ep-prefix/src/libhdfspp_ep-stamp/libhdfspp_ep-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:405: CMakeFiles/libhdfspp_ep.dir/all] Error 2
make: *** [Makefile:166: all] Error 2
FAILED ubuntu22

@guiyanakuang I see you have used ./run-all.sh apache branch-1.8 to run the docker test for the 1.8.4 release. Was the run-all.sh script on the branch-1.8 or main branch?

cc @dongjoon-hyun if you have any advise.

dongjoon-hyun commented 1 year ago

Hi, @wgtmac . You need toreinit the OS images first inside branch-1.8.

$ git checkout branch-1.8

$ cd docker

$ ./reinit.sh

Otherwise, it will try to use the pre-built images which has the latest dev environment for Apache ORC 1.9+.

https://hub.docker.com/r/apache/orc-dev

Also, you need to run the docker script of branch-1.8.

Was the run-all.sh script on the branch-1.8 or main branch?

guiyanakuang commented 1 year ago

I was running the command on the branch-1.8. Judging from the error, it may be that the memory allocated by docker is insufficient.

dongjoon-hyun commented 1 year ago

If then, you can increase the memory because the all OSes run at the same time. I'd recommend that. Or, run one by one via run-one.sh.

dongjoon-hyun commented 1 year ago

BTW, for me, it looks like a dev environment issue exists a the same time.

==> logs/ubuntu20_jdk=11_cc=clang-test.log <==
/root/orc/c++/src/ByteRLE.cc:469:36: error: implicit conversion changes signedness: 'signed char' to 'char' [-Werror,-Wsign-conversion]
              data[position + i] = readByte();
                                 ~ ^~~~~~~~~~

wgtmac commented 1 year ago

BTW, for me, it looks like a dev environment issue exists a the same time.

==> logs/ubuntu20_jdk=11_cc=clang-test.log <==
/root/orc/c++/src/ByteRLE.cc:469:36: error: implicit conversion changes signedness: 'signed char' to 'char' [-Werror,-Wsign-conversion]
              data[position + i] = readByte();
                                 ~ ^~~~~~~~~~

I agree. I think we need to backport https://github.com/apache/orc/commit/acb8d6cad138a8fc629053d487b55017d148ff9d

wgtmac commented 1 year ago

Thanks @guiyanakuang and @dongjoon-hyun! I have managed to run the docker test by increasing the memory and running them in sequence.

dongjoon-hyun commented 1 year ago

Thank you again, @wgtmac . Since this is done, let me close this issue too.

apache / orc

Release Apache ORC 1.8.5 #1583