IDE introspecting errors for external Glue catalog

[svdimchenko]

Steps to reproduce the behavior (Required)

create external catalog

CREATE EXTERNAL CATALOG `sd_iceberg`
PROPERTIES ("aws.glue.use_aws_sdk_default_behavior"  =  "true",
"aws.s3.region"  =  "eu-central-1",
"aws.glue.region"  =  "eu-central-1",
"type"  =  "iceberg",
"aws.s3.use_aws_sdk_default_behavior"  =  "true",
"iceberg.catalog.type"  =  "glue"
)

then following queries run normally

set catalog sd_iceberg;
show databases;

Expected behavior (Required)

IDE can introspect and show all external tables and their schemas.

Real behavior (Required)

when I connect to starrocks from Pycharm Pro or DBeaver and set database as sd_iceberg.aggregate (aggregate database actually exists in my glue catalog), I get the following errors:

this can be fixed with setting starrocks role as lf-admin

com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Insufficient Lake Formation permission(s) on information_schema

however after that the error is:

2023-10-18 10:46:00,788 ERROR (starrocks-mysql-nio-pool-29|4203) [IcebergMetadata.getDb():139] Database information_schema not found
org.apache.iceberg.exceptions.NoSuchNamespaceException: fail to find Glue database for namespace information_schema, error message: Database information_schema not found. (Service: Glue, Status Code: 400, Request ID: e9b0231a-7223-4c00-ad57-d2272ab755ea)

This is so because Glue does not have information_schema database.

StarRocks version (Required)

3.1.3-384ba23554

[DorianZheng]

Could you send us the entire fe.log

[svdimchenko]

2023-10-18 10:46:00,788 ERROR (starrocks-mysql-nio-pool-29|4203) [IcebergMetadata.getDb():139] Database information_schema not found
org.apache.iceberg.exceptions.NoSuchNamespaceException: fail to find Glue database for namespace information_schema, error message: Database information_schema not found. (Service: Glue, Status Code: 400, Request ID: e9b0231a-7223-4c00-ad57-d2272ab755ea)
    at org.apache.iceberg.aws.glue.GlueCatalog.loadNamespaceMetadata(GlueCatalog.java:523) ~[iceberg-aws-1.2.1.jar:?]
    at com.starrocks.connector.iceberg.glue.IcebergGlueCatalog.getDB(IcebergGlueCatalog.java:134) ~[starrocks-fe.jar:?]
    at com.starrocks.connector.iceberg.IcebergMetadata.getDb(IcebergMetadata.java:137) ~[starrocks-fe.jar:?]
    at com.starrocks.server.MetadataMgr.lambda$getDb$1(MetadataMgr.java:149) ~[starrocks-fe.jar:?]
    at java.util.Optional.map(Optional.java:265) ~[?:?]
    at com.starrocks.server.MetadataMgr.getDb(MetadataMgr.java:149) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer.resolveTable(QueryAnalyzer.java:919) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer.access$200(QueryAnalyzer.java:96) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.resolveTableRef(QueryAnalyzer.java:287) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:193) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitSelect(QueryAnalyzer.java:113) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.ast.SelectRelation.accept(SelectRelation.java:242) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:118) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryRelation(QueryAnalyzer.java:133) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:123) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.visitQueryStatement(QueryAnalyzer.java:113) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:56) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer$Visitor.process(QueryAnalyzer.java:118) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.QueryAnalyzer.analyze(QueryAnalyzer.java:106) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.AnalyzerVisitor.visitQueryStatement(AnalyzerVisitor.java:350) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.AnalyzerVisitor.visitQueryStatement(AnalyzerVisitor.java:133) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.ast.QueryStatement.accept(QueryStatement.java:56) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:55) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.AnalyzerVisitor.analyze(AnalyzerVisitor.java:135) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.analyzer.Analyzer.analyze(Analyzer.java:34) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:73) ~[starrocks-fe.jar:?]
    at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:57) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:448) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:364) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:478) ~[starrocks-fe.jar:?]
    at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:754) ~[starrocks-fe.jar:?]
    at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
    at java.lang.Thread.run(Thread.java:829) ~[?:?]

did you mean this log ?

[alberttwong]

More like all the files in https://github.com/StarRocks/starrocks/discussions/32661

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!