search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
9.03k stars 1.82k forks source link

Cannot initialize AwsClientFactory, missing no-arg constructor #51660

Open ismaelhamed opened 1 month ago

ismaelhamed commented 1 month ago

I've been testing StarRocks with Nessie (Iceberg REST) successfully up until Nessie v0.95, when I started getting this error whenever I try to query any of the tables:

[42000][1064] Cannot initialize AwsClientFactory, missing no-arg constructor: com.starrocks.connector.iceberg.IcebergAwsClientFactory

Steps to reproduce the behavior (Required)

  1. Create an iceberg table in the nessie catalog via PySpark.
  2. Make sure I can query the table in Spark SQL with no problem.
  3. Create an external catalog in StarRocks
CREATE EXTERNAL CATALOG 'iceberg'
PROPERTIES
(
  "type"="iceberg",
  "iceberg.catalog.type"="rest",
  "iceberg.catalog.uri"="http://nessie:19120/iceberg",
  "iceberg.catalog.warehouse"="ceph",
  "aws.s3.access_key"="admin",
  "aws.s3.secret_key"="password",
  "aws.s3.endpoint"="http://ceph:8080/",
  "aws.s3.enable_ssl" = "false",
  "aws.s3.enable_path_style_access"="true",
  "client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
);
  1. Set the iceberg catalog, move to the database and run the same sql statement from point 2 above.
StarRocks > SET CATALOG iceberg;
Query OK, 0 rows affected (0.00 sec)

StarRocks > SHOW databases IN iceberg;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| demo               |
+--------------------+
2 rows in set (0.02 sec)

StarRocks > USE demo;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
StarRocks > SELECT COUNT(*) AS total_users FROM users;
ERROR 2013 (HY000): Lost connection to MySQL server during query
No connection. Trying to reconnect...
ERROR 5501 (3F000): Unknown database 'demo'
ERROR:
Can't connect to the server

Expected behavior (Required)

I should see the result of the query (this was the case before Nessie v0.95)

Real behavior (Required)

Starting Nessie v0.95 I'm getting an exception. The following is the stacktrace from the fe.log:

2024-09-26 08:12:53.023Z WARN (starrocks-mysql-nio-pool-1|198) [ReadListener.lambda$handleEvent$0():81] Exception happened in one session(com.starrocks.mysql.nio.NConnectContext@4863cf4a).
java.lang.IllegalArgumentException: Cannot initialize AwsClientFactory, missing no-arg constructor: com.starrocks.connector.iceberg.IcebergAwsClientFactory
    at org.apache.iceberg.aws.AwsClientFactories.loadClientFactory(AwsClientFactories.java:74) ~[iceberg-aws-1.6.0.jar:?]
    at org.apache.iceberg.aws.AwsClientFactories.from(AwsClientFactories.java:61) ~[iceberg-aws-1.6.0.jar:?]
    at org.apache.iceberg.aws.S3FileIOAwsClientFactories.initialize(S3FileIOAwsClientFactories.java:45) ~[iceberg-aws-1.6.0.jar:?]
    at org.apache.iceberg.aws.s3.S3FileIO.initialize(S3FileIO.java:364) ~[iceberg-aws-1.6.0.jar:?]
    at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:351) ~[iceberg-core-1.6.0.jar:?]
    at org.apache.iceberg.rest.RESTSessionCatalog.newFileIO(RESTSessionCatalog.java:933) ~[iceberg-core-1.6.0.jar:?]
    at org.apache.iceberg.rest.RESTSessionCatalog.tableFileIO(RESTSessionCatalog.java:944) ~[iceberg-core-1.6.0.jar:?]
    at org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:449) ~[iceberg-core-1.6.0.jar:?]
    at org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:99) ~[iceberg-core-1.6.0.jar:?]
    at org.apache.iceberg.rest.RESTCatalog.loadTable(RESTCatalog.java:102) ~[iceberg-core-1.6.0.jar:?]
    at com.starrocks.connector.iceberg.rest.IcebergRESTCatalog.getTable(IcebergRESTCatalog.java:121) ~[starrocks-fe.jar:?]
    at com.starrocks.connector.iceberg.CachingIcebergCatalog.getTable(CachingIcebergCatalog.java:150) ~[starrocks-fe.jar:?]
    at com.starrocks.connector.iceberg.IcebergMetadata.getTable(IcebergMetadata.java:339) ~[starrocks-fe.jar:?]
    at com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:135) ~[starrocks-fe.jar:?]
    at com.starrocks.server.MetadataMgr.lambda$getTable$5(MetadataMgr.java:497) ~[starrocks-fe.jar:?]
    at java.util.Optional.map(Optional.java:265) ~[?:?]
    at com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:497) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.handleFieldList(ConnectProcessor.java:413) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:558) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:882) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.aws.AwsClientFactory
    Missing com.starrocks.connector.iceberg.IcebergAwsClientFactory [java.lang.ClassNotFoundException: com.starrocks.connector.iceberg.IcebergAwsClientFactory]
    at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:256) ~[iceberg-common-1.6.0.jar:?]
    at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32) ~[iceberg-common-1.6.0.jar:?]
    at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:226) ~[iceberg-common-1.6.0.jar:?]
    at org.apache.iceberg.aws.AwsClientFactories.loadClientFactory(AwsClientFactories.java:71) ~[iceberg-aws-1.6.0.jar:?]
    ... 23 more
    Suppressed: java.lang.ClassNotFoundException: com.starrocks.connector.iceberg.IcebergAwsClientFactory
        at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
        at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
        at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
        at java.lang.Class.forName0(Native Method) ~[?:?]
        at java.lang.Class.forName(Class.java:398) ~[?:?]
        at org.apache.iceberg.common.DynConstructors$Builder.hiddenImpl(DynConstructors.java:192) ~[iceberg-common-1.6.0.jar:?]
        at org.apache.iceberg.aws.AwsClientFactories.loadClientFactory(AwsClientFactories.java:70) ~[iceberg-aws-1.6.0.jar:?]
        at org.apache.iceberg.aws.AwsClientFactories.from(AwsClientFactories.java:61) ~[iceberg-aws-1.6.0.jar:?]
        at org.apache.iceberg.aws.S3FileIOAwsClientFactories.initialize(S3FileIOAwsClientFactories.java:45) ~[iceberg-aws-1.6.0.jar:?]
        at org.apache.iceberg.aws.s3.S3FileIO.initialize(S3FileIO.java:364) ~[iceberg-aws-1.6.0.jar:?]
        at org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:351) ~[iceberg-core-1.6.0.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.newFileIO(RESTSessionCatalog.java:933) ~[iceberg-core-1.6.0.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.tableFileIO(RESTSessionCatalog.java:944) ~[iceberg-core-1.6.0.jar:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.loadTable(RESTSessionCatalog.java:449) ~[iceberg-core-1.6.0.jar:?]
        at org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.loadTable(BaseSessionCatalog.java:99) ~[iceberg-core-1.6.0.jar:?]
        at org.apache.iceberg.rest.RESTCatalog.loadTable(RESTCatalog.java:102) ~[iceberg-core-1.6.0.jar:?]
        at com.starrocks.connector.iceberg.rest.IcebergRESTCatalog.getTable(IcebergRESTCatalog.java:121) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.iceberg.CachingIcebergCatalog.getTable(CachingIcebergCatalog.java:150) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.iceberg.IcebergMetadata.getTable(IcebergMetadata.java:339) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:135) ~[starrocks-fe.jar:?]
        at com.starrocks.server.MetadataMgr.lambda$getTable$5(MetadataMgr.java:497) ~[starrocks-fe.jar:?]
        at java.util.Optional.map(Optional.java:265) ~[?:?]
        at com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:497) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleFieldList(ConnectProcessor.java:413) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:558) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:882) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]

StarRocks version (Required)

3.3.4

alekhed commented 1 week ago

Same problem for 3.3.5. Can't use latest Starrocks with Nessie as REST Iceberg Catalog.

alekhed commented 1 week ago

Found solution for this problen. Use: "client.factory"="com.starrocks.connector.share.iceberg.IcebergAwsClientFactory". Documentation need to be updated according to changes

ismaelhamed commented 9 hours ago

Thanks @AlekHed , that worked!