Error: DNS resolution Failed for metadata-grpc-service.kubeflow:8080

[Manvi07]

/kind bug

What steps did you take and what happened: On running the example metadata notebook demo.ipynb, the step of workspace creation fails, with the following logs:

ERROR:absl:mlmd client UnavailableError: DNS resolution failed
ERROR:absl:mlmd client UnavailableError: DNS resolution failed
ERROR:absl:mlmd client UnavailableError: DNS resolution failed
ERROR:absl:mlmd client UnavailableError: DNS resolution failed
Traceback (most recent call last):
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 157, in _call_method
    response.CopyFrom(grpc_method(request))
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.UNAVAILABLE
    details = "DNS resolution failed"
    debug_error_string = "{"created":"@1594040425.818042354","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1594040424.362512978","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":262,"referenced_errors":[{"created":"@1594040424.362508705","description":"DNS resolution failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":370,"grpc_status":14,"referenced_errors":[{"created":"@1594040424.362442103","description":"C-ares status is not ARES_SUCCESS: Domain name not found","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[{"created":"@1594040424.362424448","description":"C-ares status is not ARES_SUCCESS: Domain name not found","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}]}]}]}]}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "metadata/example.py", line 14, in <module>
    labels={"n1": "v1"})
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/kubeflow/metadata/metadata.py", line 131, in __init__
    self.context_id = self._get_context_id(reuse_workspace_if_exists)
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/kubeflow/metadata/metadata.py", line 188, in _get_context_id
    ctx = self._get_existing_context()
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/kubeflow/metadata/metadata.py", line 220, in _get_existing_context
    lambda: self.store.get_contexts_by_type(self.CONTEXT_TYPE_NAME))
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/retrying.py", line 212, in call
    raise attempt.get()
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/kubeflow/metadata/metadata.py", line 756, in _retry
    return f()
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/kubeflow/metadata/metadata.py", line 220, in <lambda>
    lambda: self.store.get_contexts_by_type(self.CONTEXT_TYPE_NAME))
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 817, in get_contexts_by_type
    self._call('GetContextsByType', request, response)
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 131, in _call
    return self._call_method(method_name, request, response)
  File "/home/manvi/miniconda2/envs/tf-gpu/lib/python3.7/site-packages/ml_metadata/metadata_store/metadata_store.py", line 162, in _call_method
    raise _make_exception(e.details(), e.code().value[0])
tensorflow.python.framework.errors_impl.UnavailableError: DNS resolution failed

What did you expect to happen: Successful creation of workspace.

Environment:

• Kubeflow version:1.0.2
• Kubernetes version: (use kubectl version): 1.18.2
• OS (e.g. from /etc/os-release):Ubuntu 18.04

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.

[Manvi07]

Closing. Unable to connect when running on local, should use Kubeflow Jupyter Notebooks.

[sagarshrestha24]

i got same issue how can we slove this issue