apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
919 stars 147 forks source link

[Bug] Fix UnknownHostException: host.docker.internal in docker demo #429

Closed daragu closed 6 months ago

daragu commented 6 months ago

Throw UnknownHostException when try the demo in docker under Linux OS.

MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.UnknownHostException: host.docker.internal
    at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:478)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMeth

图片

vinishjail97 commented 6 months ago

I'm not able to re-produce the issue and the demo works for me in local.

namespace: String = "demo"
namespaceArray: Array[String] = Array("demo")
catalogOptions: HashMap[String, String] = {uri=thrift://host.docker.internal:9083}
res8_3: String = null
icebergCatalogConfig: IcebergCatalogConfig = IcebergCatalogConfig(catalogImpl=org.apache.iceberg.hive.HiveCatalog, catalogName=iceberg, catalogOptions={uri=thrift://host.docker.internal:9083})
perTableConfig: PerTableConfigImpl = PerTableConfigImpl(tableBasePath=file:///home/data/db/hudi_dimCustomer, tableDataPath=file:/home/data/db/hudi_dimCustomer, tableName=hudi_dimCustomer, namespace=[demo], hudiSourceConfig=HudiSourceConfigImpl(partitionSpecExtractorClass=org.apache.xtable.hudi.ConfigurationBasedPartitionSpecExtractor, partitionFieldSpecs=[]), targetTableFormats=[ICEBERG], icebergCatalogConfig=IcebergCatalogConfig(catalogImpl=org.apache.iceberg.hive.HiveCatalog, catalogName=iceberg, catalogOptions={uri=thrift://host.docker.internal:9083}), syncMode=INCREMENTAL, targetMetadataRetentionInHours=168)
hudiConversionSourceProvider: HudiConversionSourceProvider = org.apache.xtable.hudi.HudiConversionSourceProvider@76964a4
conversionController: ConversionController = org.apache.xtable.conversion.ConversionController@7841cb4e
res8_9: Map[String, SyncResult] = {ICEBERG=SyncResult(mode=FULL, lastInstantSynced=2023-11-05T17:21:36.406Z, syncStartTime=2024-05-11T17:48:09.633Z, syncDuration=PT2.756S, status=SyncResult.SyncStatus(statusCode=SUCCESS, errorMessage=null, errorDescription=null, canRetryOnFailure=false), recommendedSyncMode=null)}

I don't have any special configuration in /etc/hosts as well for mapping host.docker.internal. Can you upgrade the docker version you are using and see if it resolves the issue ? This is the version I'm using.

image