alberttwong
commented
3 months ago running xtable again
root@spark:/opt/hudi/hudi-sync/hudi-hive-sync# cd /opt/xtable/jars/; java -jar xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig xtable_delta.yaml -p core-site.xml
WARNING: Runtime environment or build system does not support multi-release JARs. This will impact location-based features.
2024-06-04 23:40:55 INFO org.apache.xtable.utilities.RunSync:148 - Running sync for basePath s3a://warehouse/people for following table formats [HUDI, ICEBERG]
2024-06-04 23:40:55 WARN org.apache.hadoop.util.NativeCodeLoader:60 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/opt/xtable/jars/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 23:40:56 WARN org.apache.hadoop.metrics2.impl.MetricsConfig:136 - Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2024-06-04 23:40:56 WARN org.apache.hadoop.fs.s3a.SDKV2Upgrade:39 - Directly referencing AWS SDK V1 credential provider com.amazonaws.auth.DefaultAWSCredentialsProviderChain. AWS SDK V1 credential providers will be removed once S3A is upgraded to SDK V2
2024-06-04 23:40:57 INFO org.apache.spark.sql.delta.storage.DelegatingLogStore:60 - LogStore `LogStoreAdapter(io.delta.storage.S3SingleDriverLogStore)` is used for scheme `s3a`
2024-06-04 23:40:57 INFO org.apache.spark.sql.delta.DeltaLog:60 - Loading version 0.
2024-06-04 23:40:58 INFO org.apache.spark.sql.delta.DeltaLogFileIndex:60 - Created DeltaLogFileIndex(JSON, numFilesInSegment: 1, totalFileSize: 3848)
2024-06-04 23:40:59 INFO org.apache.spark.sql.delta.Snapshot:60 - [tableId=3ce0a1d7-8227-4adc-950d-4909c31ce4ec] Created snapshot Snapshot(path=s3a://warehouse/people/_delta_log, version=0, metadata=Metadata(f9b5140a-bd50-4095-9695-021c48c8e079,null,null,Format(parquet,Map()),{"type":"struct","fields":[{"name":"id","type":"integer","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"age","type":"integer","nullable":true,"metadata":{}},{"name":"city","type":"string","nullable":true,"metadata":{}},{"name":"create_ts","type":"string","nullable":true,"metadata":{}}]},List(city),Map(),Some(1717543438122)), logSegment=LogSegment(s3a://warehouse/people/_delta_log,0,WrappedArray(S3AFileStatus{path=s3a://warehouse/people/_delta_log/00000000000000000000.json; isDirectory=false; length=3848; replication=1; blocksize=33554432; modification_time=1717543440230; access_time=0; owner=root; group=root; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=b4ef5c900263aaf74ca626d27d79cc3c versionId=null),None,1717543440230), checksumOpt=None)
2024-06-04 23:40:59 INFO org.apache.hudi.common.table.HoodieTableMetaClient:133 - Loading HoodieTableMetaClient from s3a://warehouse/people
2024-06-04 23:40:59 INFO org.apache.hudi.common.table.HoodieTableConfig:276 - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:40:59 INFO org.apache.hudi.common.table.HoodieTableMetaClient:152 - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:40:59 INFO org.apache.iceberg.hadoop.HadoopTables:101 - Table location loaded: s3a://warehouse/people
2024-06-04 23:40:59 INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline:171 - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}running sync again
root@spark:/opt/xtable/jars# cd /opt/hudi/hudi-sync/hudi-hive-sync ; \
> ./run_sync_tool.sh \
> --metastore-uris 'thrift://hive-metastore:9083' \
> --partitioned-by city \
> --base-path 's3a://warehouse/people' \
> --database hudi_db \
> --table people \
> --sync-mode hms
ls: cannot access '/opt/hudi/hudi-sync/hudi-hive-sync/../../packaging/hudi-hive-sync-bundle/target/hudi-hive-sync-*.jar': No such file or directory
setting hadoop conf dir
Running Command : java -cp /hive/lib/hive-metastore-2.3.10.jar::/hive/lib/hive-service-2.3.10.jar::/hive/lib/hive-exec-2.3.10.jar::/hive/lib/hive-jdbc-2.3.10.jar:/hive/lib/hive-jdbc-handler-2.3.10.jar::/hive/lib/jackson-annotations-2.12.0.jar:/hive/lib/jackson-core-2.12.0.jar:/hive/lib/jackson-core-asl-1.9.13.jar:/hive/lib/jackson-databind-2.12.0.jar:/hive/lib/jackson-dataformat-smile-2.12.0.jar:/hive/lib/jackson-datatype-guava-2.12.0.jar:/hive/lib/jackson-datatype-joda-2.12.0.jar:/hive/lib/jackson-jaxrs-1.9.13.jar:/hive/lib/jackson-jaxrs-base-2.12.0.jar:/hive/lib/jackson-jaxrs-json-provider-2.12.0.jar:/hive/lib/jackson-jaxrs-smile-provider-2.12.0.jar:/hive/lib/jackson-mapper-asl-1.9.13.jar:/hive/lib/jackson-module-jaxb-annotations-2.12.0.jar:/hive/lib/jackson-module-scala_2.11-2.12.0.jar:/hive/lib/jackson-xc-1.9.13.jar::/hadoop/share/hadoop/common/*:/hadoop/share/hadoop/mapreduce/*:/hadoop/share/hadoop/hdfs/*:/hadoop/share/hadoop/common/lib/*:/hadoop/share/hadoop/hdfs/lib/*:/hadoop/etc/hadoop: org.apache.hudi.hive.HiveSyncTool --metastore-uris thrift://hive-metastore:9083 --partitioned-by city --base-path s3a://warehouse/people --database hudi_db --table people --sync-mode hms
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/.ivy2/jars/org.slf4j_slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop-2.10.2/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
2024-06-04 23:45:16,811 INFO [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file null
2024-06-04 23:45:17,016 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(133)) - Loading HoodieTableMetaClient from s3a://warehouse/people
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/root/.ivy2/jars/org.apache.hadoop_hadoop-auth-2.10.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024-06-04 23:45:17,050 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2024-06-04 23:45:17,513 INFO [main] Configuration.deprecation (Configuration.java:logDeprecation(1294)) - fs.s3a.server-side-encryption-key is deprecated. Instead, use fs.s3a.server-side-encryption.key
2024-06-04 23:45:17,555 INFO [main] table.HoodieTableConfig (HoodieTableConfig.java:<init>(276)) - Loading table properties from s3a://warehouse/people/.hoodie/hoodie.properties
2024-06-04 23:45:17,570 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(152)) - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from s3a://warehouse/people
2024-06-04 23:45:17,570 INFO [main] table.HoodieTableMetaClient (HoodieTableMetaClient.java:<init>(155)) - Loading Active commit timeline for s3a://warehouse/people
2024-06-04 23:45:17,589 INFO [main] timeline.HoodieActiveTimeline (HoodieActiveTimeline.java:<init>(172)) - Loaded instants upto : Option{val=[20240604232400230__replacecommit__COMPLETED__20240604232710563]}
2024-06-04 23:45:17,621 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(408)) - Trying to connect to metastore with URI thrift://hive-metastore:9083
2024-06-04 23:45:17,629 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(493)) - Opened a connection to metastore, current connections: 1
2024-06-04 23:45:17,639 INFO [main] hive.metastore (HiveMetaStoreClient.java:open(546)) - Connected to metastore.
2024-06-04 23:45:17,666 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(162)) - Syncing target hoodie table with hive table(hudi_db.people). Hive metastore URL from HiveConf:thrift://hive-metastore:9083). Hive metastore URL from HiveSyncConfig:thrift://hive-metastore:9083, basePath :s3a://warehouse/people
2024-06-04 23:45:17,666 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(224)) - Trying to sync hoodie table people with base path s3a://warehouse/people of type COPY_ON_WRITE
2024-06-04 23:45:17,828 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:readSchemaFromParquetBaseFile(329)) - Reading schema from s3a://warehouse/people/city=ORD/part-00003-6b980503-b6c2-42cb-aaa5-2d2feacda190.c000.snappy.parquet_20240604232400230_hudiext
2024-06-04 23:45:17,843 INFO [main] table.TableSchemaResolver (TableSchemaResolver.java:hasOperationField(470)) - Failed to read operation field from avro schema (Failed to read data schema)
2024-06-04 23:45:17,893 INFO [main] ddl.HMSDDLExecutor (HMSDDLExecutor.java:getTableSchema(182)) - Time taken to getTableSchema: 16 ms
2024-06-04 23:45:17,896 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncSchema(345)) - No Schema difference for people
MessageType: message struct {
optional binary _hoodie_commit_time (UTF8);
optional binary _hoodie_commit_seqno (UTF8);
optional binary _hoodie_record_key (UTF8);
optional binary _hoodie_partition_path (UTF8);
optional binary _hoodie_file_name (UTF8);
optional int32 id;
optional binary name (UTF8);
optional int32 age;
optional binary city (UTF8);
optional binary create_ts (UTF8);
}
2024-06-04 23:45:17,925 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(265)) - Last commit time synced was found to be 20240604232400230, last commit completion time is found to be 20240604232710563
2024-06-04 23:45:17,926 INFO [main] common.HoodieSyncClient (HoodieSyncClient.java:getWrittenPartitionsSince(140)) - Last commit time synced is 20240604232400230, Getting commits since then
2024-06-04 23:45:17,928 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(286)) - Storage partitions scan complete. Found 0
2024-06-04 23:45:17,968 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(298)) - Sync complete for people
2024-06-04 23:45:17,971 INFO [main] hive.metastore (HiveMetaStoreClient.java:close(576)) - Closed a connection to metastore, current connections: 0
vinishjail97
the-other-tim-brown
Search before asking
Please describe the bug 🐞
I wrote 6 rows into delta on s3 and then used xtable to covert the data to iceberg and hudi. Next I used run_sync_tool.sh to register the hudi files in HMS and then query it into Trino. No data in Trino.
I don't see any errors.
xtable looks good
hudi-hive-sync/run_sync_tool.sh looks good
trino should show rows
Are you willing to submit PR?
Code of Conduct