search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
919 stars 147 forks source link

Clarification on Sync Command and Required JAR Files #473

Closed soumilshah1995 closed 5 months ago

soumilshah1995 commented 5 months ago

Description: In your recent Medium post titled Using OneTable to Translate a HUDI Table to Iceberg Format and Sync with Glue Catalog, you discussed the process of converting a HUDI table to Iceberg format and syncing it with the Glue catalog. I found the content insightful, but I have a couple of questions regarding the provided instructions and required dependencies.

Usage of catalog.yaml and Sync Command:

You mentioned using a catalog.yaml file and running a sync command. Could you please clarify whether executing this sync command automatically creates a table in the Glue catalog?

Required JAR Files: In the command provided in your blog post:

java -cp utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar:/Users/sagarl/Downloads/iceberg-aws-1.3.1.jar:/Users/sagarl/Downloads/bundle-2.23.9.jar io.onetable.utilities.RunSync --datasetConfig config.yaml --icebergCatalogConfig catalog.yaml

You mentioned using three JAR files: iceberg-aws-1.3.1.jar, bundle-2.23.9.jar, and utilities-0.1.0-SNAPSHOT-bundled.jar. While the first and last JAR files seem self-explanatory, I'm unsure about the purpose of bundle-2.23.9.jar. Could you please provide more information about this JAR file and where I can find it?

im trying the labs as well catalog.yaml

catalogImpl: org.apache.iceberg.aws.glue.GlueCatalog
catalogName: onetable
catalogOptions:
    io-impl: org.apache.iceberg.aws.s3.S3FileIO
    warehouse: s3://soumilshah-dev-1995/warehouse

my_config.yaml

sourceFormat: HUDI

targetFormats:
  - ICEBERG
datasets:
  -
    tableBasePath: s3://soumilshah-dev-1995/xmldata/silver/table_name=invoices/
    tableName: invoices
    partitionSpec: destinationstate:VALUE
    namespace: icebergdb

Commands

java -jar utilities-0.1.0-beta1-bundled.jar --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml

Error Logs

java -jar utilities-0.1.0-beta1-bundled.jar --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml io.onetable.utilities.RunSync

SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x or earlier.
SLF4J: Ignoring binding found at [jar:file:/home/glue_user/workspace/jupyter_workspace/utilities-0.1.0-beta1-bundled.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an explanation.
2024-06-19 22:32:30 INFO  io.onetable.utilities.RunSync:141 - Running sync for basePath s3://soumilshah-dev-1995/xmldata/silver/table_name=invoices/ for following table formats [ICEBERG]
2024-06-19 22:32:32 ERROR io.onetable.utilities.RunSync:164 - Error running sync for s3://soumilshah-dev-1995/xmldata/silver/table_name=invoices/
java.lang.IllegalArgumentException: Cannot initialize Catalog implementation org.apache.iceberg.aws.glue.GlueCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
        Missing org.apache.iceberg.aws.glue.GlueCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
        at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:224) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.iceberg.IcebergTableManager.lambda$getCatalog$6(IcebergTableManager.java:116) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_392]
        at io.onetable.iceberg.IcebergTableManager.getCatalog(IcebergTableManager.java:113) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.iceberg.IcebergTableManager.tableExists(IcebergTableManager.java:63) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.iceberg.IcebergClient.<init>(IcebergClient.java:105) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.iceberg.IcebergClient.<init>(IcebergClient.java:66) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.client.TableFormatClientFactory.createForFormat(TableFormatClientFactory.java:45) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.client.OneTableClient.lambda$sync$0(OneTableClient.java:106) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) ~[?:1.8.0_392]
        at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_392]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_392]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_392]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_392]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_392]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_392]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_392]
        at io.onetable.client.OneTableClient.sync(OneTableClient.java:102) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at io.onetable.utilities.RunSync.main(RunSync.java:162) ~[utilities-0.1.0-beta1-bundled.jar:?]
Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
        Missing org.apache.iceberg.aws.glue.GlueCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
        at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221) ~[utilities-0.1.0-beta1-bundled.jar:?]
        ... 18 more
        Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog
                at java.net.URLClassLoader.findClass(URLClassLoader.java:387) ~[?:1.8.0_392]
                at java.lang.ClassLoader.loadClass(ClassLoader.java:418) ~[?:1.8.0_392]
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_392]
                at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ~[?:1.8.0_392]
                at java.lang.Class.forName0(Native Method) ~[?:1.8.0_392]
                at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_392]
                at org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.iceberg.IcebergTableManager.lambda$getCatalog$6(IcebergTableManager.java:116) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_392]
                at io.onetable.iceberg.IcebergTableManager.getCatalog(IcebergTableManager.java:113) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.iceberg.IcebergTableManager.tableExists(IcebergTableManager.java:63) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.iceberg.IcebergClient.<init>(IcebergClient.java:105) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.iceberg.IcebergClient.<init>(IcebergClient.java:66) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.client.TableFormatClientFactory.createForFormat(TableFormatClientFactory.java:45) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.client.OneTableClient.lambda$sync$0(OneTableClient.java:106) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321) ~[?:1.8.0_392]
                at java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169) ~[?:1.8.0_392]
                at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) ~[?:1.8.0_392]
                at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[?:1.8.0_392]
                at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_392]
                at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_392]
                at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_392]
                at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566) ~[?:1.8.0_392]
                at io.onetable.client.OneTableClient.sync(OneTableClient.java:102) ~[utilities-0.1.0-beta1-bundled.jar:?]
                at io.onetable.utilities.RunSync.main(RunSync.java:162) ~[utilities-0.1.0-beta1-bundled.jar:?]
sh-4.2$ 
sh-4.2$

Your insights on these matters would greatly help me understand the process better. Looking forward to your response.

Are you willing to submit PR?

Code of Conduct

sagarlakshmipathy commented 5 months ago

@soumilshah1995

  1. You mentioned using a catalog.yaml file and running a sync command. Could you please clarify whether executing this sync command automatically creates a table in the Glue catalog?

Pre requisites are that you need to create a database in Glue Catalog. XTable can create the table.

  1. You mentioned using three JAR files: iceberg-aws-1.3.1.jar, bundle-2.23.9.jar, and utilities-0.1.0-SNAPSHOT-bundled.jar. While the first and last JAR files seem self-explanatory, I'm unsure about the purpose of bundle-2.23.9.jar. Could you please provide more information about this JAR file and where I can find it?

Its an AWS jar file, look here

3.

Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog
        Missing org.apache.iceberg.aws.glue.GlueCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]
        at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220) ~[utilities-0.1.0-beta1-bundled.jar:?]
        at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221) ~[utilities-0.1.0-beta1-bundled.jar:?]
        ... 18 more
        Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog

You can get rid of this is by adding all the jars I used in the blog. LMK how it goes.

soumilshah1995 commented 5 months ago

Thanks a lot Sagar

your suggestion appear to fix issue

just adding steps for other I used 3 jar files

Jar Files iceberg-aws-1.3.1.jar https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws/1.3.1/iceberg-aws-1.3.1.jar bundle-2.23.9.jar https://mvnrepository.com/artifact/software.amazon.awssdk/bundle/2.23.9 utilities-0.1.0-beta1-bundled.jar https://github.com/apache/incubator-xtable/packages/1986830

RAN java -cp "utilities-0.1.0-beta1-bundled.jar:iceberg-aws-1.3.1.jar:bundle-2.23.9.jar" io.onetable.utilities.RunSync --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml

image

ambaricloud commented 4 months ago

java -cp bundle-2.23.1o.jar:iceberg-aws-1.5.1.jar:xtable-utilities-0.1.0-SNAPSHOT-bundled.jar -jar xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig config.yaml --icebergCatalogConfig catalog.yaml

WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features. 2024-07-10 13:42:17 INFO org.apache.xtable.utilities.RunSync:148 - Running sync for basePath s3://ambaricloudsatya/prod.db/orders for following table formats [DELTA] 2024-07-10 13:42:18 WARN org.apache.hadoop.util.NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/satyak/iceberg/demo/xtable/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar) to constructor java.nio.DirectByteBuffer(long,int) WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 2024-07-10 13:42:19 WARN org.apache.spark.util.Utils:72 - Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 2024-07-10 13:42:19 WARN org.apache.spark.util.Utils:72 - Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 2024-07-10 13:42:20 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:136 - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties 2024-07-10 13:42:21 WARN org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2 2024-07-10 13:42:21 INFO org.apache.spark.sql.delta.storage.DelegatingLogStore:60 - LogStore LogStoreAdapter(io.delta.storage.S3SingleDriverLogStore) is used for scheme s3 2024-07-10 13:42:22 INFO org.apache.spark.sql.delta.DeltaLog:60 - Creating initial snapshot without metadata, because the directory is empty 2024-07-10 13:42:23 INFO org.apache.spark.sql.delta.InitialSnapshot:60 - [tableId=9d2b9c69-23fa-43f3-939d-be28a9066043] Created snapshot InitialSnapshot(path=s3://<>/prod.db/orders/data/_delta_log, version=-1, metadata=Metadata(71b4d663-d8a2-4e51-a1ef-23d45fa1d70e,null,null,Format(parquet,Map()),null,List(),Map(),Some(1720636943963)), logSegment=LogSegment(s3://ambaricloudsatya/prod.db/orders/data/_delta_log,-1,List(),None,-1), checksumOpt=None) 2024-07-10 13:42:24 INFO org.apache.xtable.conversion.ConversionController:240 - No previous InternalTable sync for target. Falling back to snapshot sync. 2024-07-10 13:42:24 ERROR org.apache.xtable.utilities.RunSync:171 - Error running sync for s3://ambaricloudsatya/prod.db/orders java.lang.IllegalArgumentException: Cannot initialize Catalog implementation org.apache.iceberg.aws.glue.GlueCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog Missing org.apache.iceberg.aws.glue.GlueCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog] at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:224) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergTableManager.lambda$getCatalog$6(IcebergTableManager.java:116) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705) ~[?:?] at org.apache.xtable.iceberg.IcebergTableManager.getCatalog(IcebergTableManager.java:113) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergTableManager.getTable(IcebergTableManager.java:56) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionSource.initSourceTable(IcebergConversionSource.java:81) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionSource.getSourceTable(IcebergConversionSource.java:60) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:183) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.utilities.RunSync.main(RunSync.java:169) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] Caused by: java.lang.NoSuchMethodException: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog Missing org.apache.iceberg.aws.glue.GlueCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog] at org.apache.iceberg.common.DynConstructors.buildCheckedException(DynConstructors.java:250) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.iceberg.common.DynConstructors.access$200(DynConstructors.java:32) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.iceberg.common.DynConstructors$Builder.buildChecked(DynConstructors.java:220) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] ... 11 more Suppressed: java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?] at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?] at java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[?:?] at java.lang.Class.forName0(Native Method) ~[?:?] at java.lang.Class.forName(Class.java:398) ~[?:?] at org.apache.iceberg.common.DynConstructors$Builder.impl(DynConstructors.java:149) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.iceberg.CatalogUtil.loadCatalog(CatalogUtil.java:221) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergTableManager.lambda$getCatalog$6(IcebergTableManager.java:116) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705) ~[?:?] at org.apache.xtable.iceberg.IcebergTableManager.getCatalog(IcebergTableManager.java:113) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergTableManager.getTable(IcebergTableManager.java:56) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionSource.initSourceTable(IcebergConversionSource.java:81) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionSource.getSourceTable(IcebergConversionSource.java:60) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:183) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:121) ~[xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT] at org.apache.xtable.utilities.RunSync.main(RunSync.java:169) [xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]

sagarlakshmipathy commented 4 months ago

@ambaricloud As I referred to in the comment in a different issue, you need relevant jars to fix this issue.

java.lang.IllegalArgumentException: Cannot initialize Catalog implementation org.apache.iceberg.aws.glue.GlueCatalog: Cannot find constructor for interface org.apache.iceberg.catalog.Catalog Missing org.apache.iceberg.aws.glue.GlueCatalog [java.lang.ClassNotFoundException: org.apache.iceberg.aws.glue.GlueCatalog]

https://github.com/apache/incubator-xtable/issues/488#issuecomment-2221062642

ambaricloud commented 4 months ago

Thank you. Now I can convert Glue Iceberg to Delta.

couple of comments.

  1. In datasetConfig requires namespace.
  2. In Catalog.yaml "warehouse: s3://<>/prod" ---Not sure the purpose this.

java -cp "utilities-0.1.0-beta1-bundled.jar:iceberg-aws-1.3.1.jar:bundle-2.23.9.jar" io.onetable.utilities.RunSync --datasetConfig ice_to_delta_orders_config.yaml --icebergCatalogConfig ice_to_delta_orders_catalog.yam