onehouseinc / LakeView

Monitoring and insights on your data lakehouse tables
Apache License 2.0
15 stars 3 forks source link

Changed the Wrong hoodie.table.name and ran lakeview extractor service but doesn't update the UI #83

Open alberttwong opened 1 month ago

alberttwong commented 1 month ago

https://cloud.onehouse.ai/c3eb3868-6979-41cd-9018-952d29a43337/data/lakes/iceberg/databases/taxis

reran extractor service

root@spark:/opt/lakeview# java -jar LakeView-1.0-SNAPSHOT-all.jar -p '/opt/lakeview/iceberg.yaml'
17:33:20.605 [main] INFO  com.onehouse.Main - Starting LakeView extractor service
17:33:20.762 [main] INFO  com.onehouse.RuntimeModule - Spinning up 70 threads
17:33:21.121 [main] INFO  com.onehouse.metrics.MetricsServer - Starting metrics server
17:33:21.133 [main] INFO  c.o.m.TableDiscoveryAndUploadJob - Running metadata-extractor one time
17:33:21.134 [main] INFO  c.o.m.TableDiscoveryService - Starting table discover service, excluding []
17:33:21.134 [main] INFO  c.o.m.TableDiscoveryService - Discovering tables in s3://warehouse/people
17:33:21.213 [main] INFO  c.o.m.TableDiscoveryService - Discovering tables in s3://warehouse/taxis
17:33:21.347 [metadata-extractor-3] INFO  c.o.m.TableMetadataUploaderService - Uploading metadata of following tables: [Table(absoluteTableUri=s3://warehouse/taxis, databaseName=taxis, lakeName=iceberg, tableId=null)]
17:33:21.349 [metadata-extractor-3] INFO  c.o.m.TableMetadataUploaderService - Fetching checkpoint for tables: [Table(absoluteTableUri=s3://warehouse/taxis, databaseName=taxis, lakeName=iceberg, tableId=1fbfcc05-408b-3e43-b95e-af8bb335e97d)]
17:33:21.626 [metadata-extractor-2] INFO  c.o.m.TimelineCommitInstantsUploader - uploading instants in table: Table(absoluteTableUri=s3://warehouse/taxis, databaseName=taxis, lakeName=iceberg, tableId=1fbfcc05-408b-3e43-b95e-af8bb335e97d) timeline: COMMIT_TIMELINE_TYPE_ACTIVE
17:33:21.640 [metadata-extractor-2] INFO  c.o.m.TimelineCommitInstantsUploader - Reached end of instants in COMMIT_TIMELINE_TYPE_ACTIVE for table Table(absoluteTableUri=s3://warehouse/taxis, databaseName=taxis, lakeName=iceberg, tableId=1fbfcc05-408b-3e43-b95e-af8bb335e97d)
17:33:21.640 [main] INFO  c.o.m.TableDiscoveryAndUploadJob - Run Completed
17:33:21.640 [main] INFO  com.onehouse.metrics.MetricsServer - Shutting down metrics server

modified fixed file

albert@Alberts-MBP Downloads % cat hoodie.properties
#Updated at 2024-07-16T17:09:04.443893Z
#Tue Jul 16 17:09:04 UTC 2024
hoodie.table.type=COPY_ON_WRITE
hoodie.table.metadata.partitions=column_stats,files
hoodie.table.partition.fields=vendor_id
hoodie.archivelog.folder=archived
hoodie.timeline.layout.version=1
hoodie.table.checksum=1914023381
hoodie.datasource.write.drop.partition.columns=false
hoodie.table.timeline.timezone=UTC
hoodie.table.recordkey.fields=
hoodie.table.name=taxis
hoodie.datasource.write.hive_style_partitioning=true
hoodie.table.metadata.partitions.inflight=
hoodie.populate.meta.fields=false
hoodie.table.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator
hoodie.table.version=6
alberttwong commented 1 month ago

related to https://github.com/onehouseinc/LakeView/issues/82

alberttwong commented 1 month ago

suggested to delete and recreate table.

Unfortunately I have a new error. hoodie.properties file seems invalid. Please check for left over .updated files if any, manually copy it to hoodie.properties and retry

alberttwong commented 1 month ago

advice was to delete the .hoodie table.

alberttwong commented 1 month ago

removing partitionSpec in xtable config, deleted the .hoodie table in S3, deleted the table in onehouse, ran xtable again and lakeview extractor. Same hoodie.table.name