search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
919 stars 147 forks source link

[Demo] Fix UnknownHostException: host.docker.internal in docker demo #430

Closed daragu closed 6 months ago

daragu commented 6 months ago

Important Read

Fixed #429

What is the purpose of the pull request

Fix UnknownHostException: host.docker.internal in docker demo under Linux OS.

MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.UnknownHostException: host.docker.internal
    at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:478)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:245)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1740)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:83)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:97)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMeth

Brief change log

We can use docker container hostname.

Verify this pull request

locally.

ghost commented 6 months ago

CI report:

Bot commands @xtable-bot supports the following commands: - `@xtable-bot run azure` re-run the last Azure build
daragu commented 6 months ago

hi @the-other-tim-brown can you take a look, please?

daragu commented 6 months ago

hi @vinishjail97 can you help to take a look? thanks

the-other-tim-brown commented 6 months ago

I have 127.0.0.1 kubernetes.docker.internal added to /etc/hosts, I'm guessing that is how I'm able to make this work without these changes. I will try this out though on my machine and validate it works with the updates

the-other-tim-brown commented 6 months ago

I'm running into java.io.FileNotFoundException: File file:/home/data/db/hudi_dimCustomer/metadata/00001-64637ad8-218d-496d-aa33-52db8188727b.metadata.json does not exist right now. @daragu did you ever see something similar?

vinishjail97 commented 6 months ago

I'm not able to re-produce the issue from my side, added a comment on the issue. https://github.com/apache/incubator-xtable/issues/429#issuecomment-2105976390

daragu commented 6 months ago

I'm not able to re-produce the issue from my side, added a comment on the issue. #429 (comment)

hi @vinishjail97, I think host.docker.internal is effective in Docker Desktop environment, but not in systems such as Linux.

https://docs.docker.com/desktop/networking/#i-want-to-connect-from-a-container-to-a-service-on-the-host

daragu commented 6 months ago

@daragu can you squash your commits? I was able to confirm this is working after resetting some things on my local env.

Done, thanks.