Improve GraalVM Reachability Metadata and corresponding nativeTest related unit tests

[linghengqian]

Feature Request

For English only, other languages will not be accepted.

Please pay attention on issues you submitted, because we maybe need more details. If no response anymore and we cannot make decision by current information, we will close it.

Please answer these questions before submitting your issue. Thanks!

Is your feature request related to a problem?

• apache/shardingsphere#21347.

Describe the feature you would like.

• In the current master branch, for the sharding, readwrite-splitting, mask, shadow and encrypt features in standalone mode and H2Database, there are already corresponding GraalVM Reachability Metadata and corresponding nativeTest-related unit tests located under GraalVM Native Image.

• The previous PR did not cover further situations. We need to add additional unit tests and corresponding GraalVM Reachability Metadata for the following functions. Some unit tests will require the GraalVM Reachability Metadata of testcontainers.

  • Modes except Local
    • [x] apache/zookeeper
    • [x] etcd-io/etcd
  • Distributed Transactions
    • [x] atomikos/transactions-essentials
    • [x] jbosstm/narayana
    • [x] apache/incubator-seata
  • Databases except h2database
    • [x] mysql:9.0.1-oraclelinux9
    • [x] postgres:16.3-bookworm
    • [x] opengauss/opengauss:5.0.0
    • [x] mcr.microsoft.com/mssql/server:2022-CU14-ubuntu-22.04
    • [ ] gvenzl/oracle-free and gvenzl/oracle-xe
    • [ ] apache/doris? Where is the docker image of runtime?
    • [ ] prestodb/presto
    • [x] apache/hive:4.0.0
    • [x] clickhouse/clickhouse-server:24.6.2.17
  • Other features
    • [x] DistSQL
    • [x] Logging - qos-ch/logback

• In the current master branch, ShardingSphere temporarily hosts the GraalVM Reachability Metadata of some third-party libraries. These irrelevant json entries should be submitted to https://github.com/oracle/graalvm-reachability-metadata and then deleted from the ShardingSphere side. This sub-work is initially planned to be completed before the release of GraalVM CE for JDK 29.

  • [ ] com.github.ben-manes.caffeine:caffeine:2.9.3
    • Partially located at https://github.com/oracle/graalvm-reachability-metadata/pull/388 .
  • [ ] com.zaxxer:HikariCP:4.0.3
  • [ ] org.apache.calcite:calcite-core:1.35.0
  • [ ] org.apache.groovy:groovy:4.0.10
  • [ ] org.apache.shardingsphere.elasticjob:elasticjob-lite-core:3.0.4
    • this is addressed at https://github.com/apache/shardingsphere-elasticjob/issues/2258 .
  • [ ] org.apache.zookeeper:zookeeper:3.9.0
  • [x] org.glassfish.jaxb:jaxb-runtime:2.3.1
    • Removed directly in apache/shardingsphere#29384, no need to commit to GraalVM Reachability Metadata Repository side.
  • [ ] org.slf4j:slf4j-api:1.7.36
  • [ ] com.mysql:mysql-connector-j:8.0.31
  • [ ] ch.qos.logback:logback-core:1.2.12
  • [ ] io.netty:netty-transport-classes-epoll:4.1.99.Final
  • [ ] org.jboss.narayana.jta:jta:5.12.4.Final
  • [ ] com.atomikos:transactions:6.0.0
  • [ ] com.h2database:h2:2.2.224
  • [ ] com.github.docker-java:docker-java-api:3.3.6
  • [ ] org.apache.seata:seata-all:2.1.0

• The current issue involves several user-side experience issues.

  • [ ] Remove the requirement for --report-unsupported-elements-at-runtime of buildArgs in GraalVM Native Build Tools. This requires further breaking ShardingSphere's compatibility with previous versions under GraalVM Native Image.
  • [x] Increased GraalVM CE version for handling NativeTest and building ShardingSphere Proxy Native from GraalVM CE 23.0.2 For JDK 17.0.9 to GraalVM CE 24.0.2 For JDK 22.0.2.

[linghengqian]

• I'm working on this issue.

[linghengqian]

user-side experience:

• GraalVM CE 23.1.1 still fails to solve the problem that occurred at https://github.com/oracle/graal/issues/7500 in GraalVM CE 23.1.0 . I have refactored a small unit test at https://github.com/linghengqian/shardingsphere-graalvm-v2311-native-test .
  • Under investigation at apache/shardingsphere#28609 . This involves breaking changes to the GraalVM Truffle framework.
  • update in 2023.12.16: Maybe https://github.com/oracle/graal/pull/8039 .
  • update in 2024.01.18: Fixed by #29764 .
  • update in 2024.06.07: Fixed by #31630 .
• Due to the impact of https://github.com/oracle/graal/pull/7474 in GraalVM CE 24.0.0 for JDK 22, I believe that to remove --report-unsupported-elements-at-runtime, then we must set a batch of --initialize-at-run-time=xxx.xxx.xxx in buildArgs like Spring Framework.
  • update in 2024.07.18: Maybe https://github.com/oracle/graal/pull/8815 .

[linghengqian]

Databases other than H2database:

• Oracle JDBC Driver com.oracle.database.jdbc:ojdbc8:23.3.0.23.09 cannot be used under GraalVM Native Image and will not be considered. Reference https://github.com/oracle/graal/issues/7979 .
  • Wait https://issues.apache.org/jira/browse/LEGAL-663 .
• For Presto JDBC Driver com.facebook.presto:presto-jdbc:0.287, we need to wait for further clarification from https://github.com/testcontainers/testcontainers-java/issues/8657 , its testcontainers-java module is quite suspicious .
• For Doris, as affected by https://github.com/apache/doris/issues/9426 , we may have to use a third-party Docker Image from selectdb. Refer to https://hub.docker.com/r/selectdb/doris.be-ubuntu and https://hub.docker.com/r/selectdb/doris.fe-ubuntu .
  • For https://github.com/apache/doris, we lack a real usable Runtime Image.
  • Wait https://github.com/bitnami/containers/issues/69243 .
• ~For HiveServer2, we need to wait for further clarification from~
  • https://issues.apache.org/jira/browse/HIVE-28295
  • https://issues.apache.org/jira/browse/HIVE-28308
  • https://issues.apache.org/jira/browse/HIVE-28316
  • https://issues.apache.org/jira/browse/HIVE-28317
  • https://issues.apache.org/jira/browse/HIVE-28322
  • https://issues.apache.org/jira/browse/HIVE-28417
  • https://issues.apache.org/jira/browse/HIVE-28418
  • https://issues.apache.org/jira/browse/HIVE-28420
  • https://issues.apache.org/jira/browse/HIVE-28423
  • https://issues.apache.org/jira/browse/HIVE-28424
  • https://issues.apache.org/jira/browse/HIVE-28429
  • https://issues.apache.org/jira/browse/HIVE-28430
  • https://issues.apache.org/jira/browse/HIVE-28432
  • https://issues.apache.org/jira/browse/HIVE-28437
  • https://issues.apache.org/jira/browse/HIVE-28444
  • https://issues.apache.org/jira/browse/HIVE-28445
  • https://github.com/apache/spark/pull/30250
  • https://github.com/dbeaver/dbeaver/issues/22777
  • https://github.com/apache/logging-log4j2/issues/1539
  • https://github.com/grpc/grpc-java/issues/10601
  • https://github.com/oracle/graalvm-reachability-metadata/issues/377
  • https://github.com/apache/hive/pull/5313
  • https://github.com/apache/hive/pull/5187
  • https://github.com/apache/logging-log4j2/issues/2794
  • https://github.com/apache/logging-log4j2/issues/2791
  • https://github.com/apache/hive/pull/5375
  • https://github.com/apache/hive/pull/5381
  • https://github.com/apache/iceberg/pull/10518
  • https://github.com/apache/iceberg/issues/10940
  • https://github.com/apache/orc/pull/1627

[linghengqian]

Modes except Local:

• Etcd integration is affected by a bug in vertx-grpc. Refer to https://github.com/linghengqian/shardingsphere-master-b-etcd-test .

  • Fixed by apache/shardingsphere#29738 .

• Consul integration is blocked by:

  • apache/shardingsphere#29419
  • apache/shardingsphere#29509
  • apache/shardingsphere-plugin#33

[github-actions[bot]]

There hasn't been any activity on this issue recently, and in order to prioritize active issues, it will be marked as stale.

[linghengqian]

@zhfeng

• Since https://github.com/oracle/graal/pull/8815 has been merged, we can completely remove the use of --report-unsupported-elements-at-runtime on the shardingsphere side as long as we change the minimum GraalVM CE compilation version from JDK 22.0.1 to JDK 23 on the shardingsphere side.
• What do you think? I'm not sure how long Quarkus will have to maintain support for lower versions of GraalVM CE. GraalVM CE For JDK 23 will be released on September 17, 2024.