Closed alberttwong closed 11 months ago
Try to create catalog as:
create external catalog 'iceberg'
PROPERTIES
(
"type"="iceberg",
"iceberg.catalog.type"="rest",
"iceberg.catalog.uri"="http://iceberg-rest:8181",
"iceberg.catalog.warehouse"="starrocks",
"aws.s3.access_key"="admin",
"aws.s3.secret_key"="password",
"aws.s3.endpoint"="http://minio:9000",
"aws.s3.enable_path_style_access"="true",
"client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
);
Force using SR's AWS client factory in the rest catalog.
Because in order to seamlessly support Tabular, Starrocks used Iceberg's official AWS client factory in the rest catalog, which caused the parameters of aws.s3.xxx
to become invalid. So if we want aws.s3.xxx
parameters to be valid, we need to force specific "client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
validated the fix.
@Smith-Cruise Can you please help figure out what to write in the docs about client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory
? I don't know what to add to https://github.com/StarRocks/starrocks/edit/main/docs/en/data_source/catalog/iceberg_catalog.md
Is this property used only for Tabular?
@Smith-Cruise Can you please help figure out what to write in the docs about
client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory
? I don't know what to add to https://github.com/StarRocks/starrocks/edit/main/docs/en/data_source/catalog/iceberg_catalog.mdIs this property used only for Tabular?
No, this property only used by iceberg self-hosted rest catalog. If you are using tabular, you don't need to add this property.
OK, I am using the REST catalog container provided by Tabular (the same as Albert was using). I will try it with and without the property.
that rest catalog container is provided by apache iceberg.
that rest catalog container is provided by apache iceberg.
This one @alberttwong ?
image: tabulario/iceberg-rest
Here is what I am thinking for the section of the Lakehouse quickstart that describes the external catalog:
The external catalog is the configuration that allows StarRocks to operate on the Iceberg data as if it was in StarRocks databases and tables. The individual configuration properties will be detailed after the command.
CREATE EXTERNAL CATALOG 'iceberg'
PROPERTIES
(
"type"="iceberg",
"iceberg.catalog.type"="rest",
"iceberg.catalog.uri"="http://iceberg-rest:8181",
"iceberg.catalog.warehouse"="warehouse",
"aws.s3.access_key"="admin",
"aws.s3.secret_key"="password",
"aws.s3.endpoint"="http://minio:9000",
"aws.s3.enable_path_style_access"="true",
"client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
);
Property | Description |
---|---|
type |
In this example the type is iceberg . Other options include Hive, Hudi, Delta Lake, and JDBC. |
iceberg.catalog.type |
In this example rest is used. Tabular provides the Docker image used and Tabular uses REST. |
iceberg.catalog.uri |
The REST server endpoint. |
iceberg.catalog.warehouse |
The identifier of the Iceberg catalog. In this case the warehouse name specified in the compose file is warehouse . |
aws.s3.access_key |
The MinIO key. In this case the key and password are set in the compose file to admin |
aws.s3.secret_key |
and password . |
aws.s3.endpoint |
The MinIO endpoint. |
aws.s3.enable_path_style_access |
When using MinIO for Object Storage this is required. MinIO expects this format http://host:port/<bucket_name>/<key_name> |
client.factory |
By setting this property to use iceberg.IcebergAwsClientFactory the aws.s3.access_key and aws.s3.secret_key parameters are used for authentication. |
following https://iceberg.apache.org/spark-quickstart/ and https://github.com/StarRocks/starrocks/discussions/23427
docker-compose.yml
then we run
docker-compose up
.creating table and data.
FE error log