Open cpereir1 opened 1 year ago
szczeles
commented
1 year ago Hey @cpereir1! That's interesting issue, let's investigate together. Could you please provide me a bit more of a context?
kedro kubeflow compile and attach the generated yaml -> I will check it on my testbed.pip freeze so I can ensure I use the same libraries versions as you.
cpereir1
commented
1 year ago Hi @szczeles ! Thank you so much for your reply :)
Running kedro kubeflow compile does not generate a yaml for me? It generates the following warning in the terminal:
[02/08/23 10:33:50] WARNING /opt/homebrew/lib/python3.10/site-packages/kedro_viz/models warnings.py:109
/experiment_tracking.py:18: MovedIn20Warning:
[31mDeprecated API features detected! These feature(s) are
not compatible with SQLAlchemy 2.0. [32mTo prevent
incompatible upgrades prior to updating applications,
ensure requirements files are pinned to "sqlalchemy<2.0".
[36mSet environment variable SQLALCHEMY_WARN_20=1 to show
all deprecation warnings. Set environment variable
SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this
message.[0m (Background on SQLAlchemy 2.0 at:
https://sqlalche.me/e/b8d9)
Base = declarative_base()
[02/08/23 10:34:03] WARNING Failed to send data to Heap. Exception of type 'SSLError' was plugin.py:213
raised.
[02/08/23 10:34:05] WARNING Failed to send data to Heap. Exception of type 'SSLError' was plugin.py:213
raised.
[02/08/23 10:34:06] WARNING /opt/homebrew/lib/python3.10/site-packages/kfp/components/_ warnings.py:109
data_passing.py:227: UserWarning: Missing type name was
inferred as "Float" based on the value "0.8".
warnings.warn(
WARNING /opt/homebrew/lib/python3.10/site-packages/kfp/components/_ warnings.py:109
data_passing.py:227: UserWarning: Missing type name was
inferred as "Integer" based on the value "3".
warnings.warn( Running kedro kubeflow init <host> does generate a yaml (attached in .txt so it is allowed by github). Running kedro kubeflow compile after does not change the yaml file.
szczeles
commented
1 year ago If you use the latest plugin version, kedro kubeflow compile should generate file pipeline.yml in the root directory: https://github.com/getindata/kedro-kubeflow/blob/0.7.3/kedro_kubeflow/cli.py#L162
cpereir1
commented
1 year ago Oops, apologies. Yes it does. Here it goes attached. pipeline_yml.txt
szczeles
commented
1 year ago Hm, all looks good, both pipeline yaml and the packages you have installed. Maybe the exception is a result of bad handling of connectivity issue. Can you please run the following?
$ kedro kubeflow list-pipelinesHmm, I guess that is the issue. How could I correctly configure the connectivity or authenticate to the Kubeflow instance from kedro/kedro kubeflow?
This is the output kedro kubeflow list-pipelines
[02/08/23 16:11:29] WARNING /opt/homebrew/lib/python3.10/site-packages/kedro_vi warnings.py:109
z/models/experiment_tracking.py:18:
MovedIn20Warning: [31mDeprecated API features
detected! These feature(s) are not compatible with
SQLAlchemy 2.0. [32mTo prevent incompatible
upgrades prior to updating applications, ensure
requirements files are pinned to "sqlalchemy<2.0".
[36mSet environment variable SQLALCHEMY_WARN_20=1
to show all deprecation warnings. Set environment
variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to
silence this message.[0m (Background on SQLAlchemy
2.0 at: https://sqlalche.me/e/b8d9)
Base = declarative_base()
[02/08/23 16:11:35] WARNING Failed to send data to Heap. Exception of type plugin.py:213
'SSLError' was raised.
WARNING Failed to send data to Heap. Exception of type plugin.py:213
'SSLError' was raised.
WARNING Retrying (Retry(total=2, connect=None, connectionpool.py:812
read=None, redirect=None, status=None)) after
connection broken by
'NewConnectionError('<urllib3.connection.HTTP
SConnection object at 0x12f125090>: Failed to
establish a new connection: [Errno 8]
nodename nor servname provided, or not
known')': /apis/v1beta1/healthz
WARNING Retrying (Retry(total=1, connect=None, connectionpool.py:812
read=None, redirect=None, status=None)) after
connection broken by
'NewConnectionError('<urllib3.connection.HTTP
SConnection object at 0x12f124e50>: Failed to
establish a new connection: [Errno 8]
nodename nor servname provided, or not
known')': /apis/v1beta1/healthz
WARNING Retrying (Retry(total=0, connect=None, connectionpool.py:812
read=None, redirect=None, status=None)) after
connection broken by
'NewConnectionError('<urllib3.connection.HTTP
SConnection object at 0x12f124f40>: Failed to
establish a new connection: [Errno 8]
nodename nor servname provided, or not
known')': /apis/v1beta1/healthz
╭───────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connection.py:174 in _new_conn │
│ │
│ 171 │ │ │ extra_kw["socket_options"] = self.socket_options │
│ 172 │ │ │
│ 173 │ │ try: │
│ ❱ 174 │ │ │ conn = connection.create_connection( │
│ 175 │ │ │ │ (self._dns_host, self.port), self.timeout, **extra_kw │
│ 176 │ │ │ ) │
│ 177 │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/util/connection.py:72 in │
│ create_connection │
│ │
│ 69 │ │ │ LocationParseError(u"'%s', label empty or too long" % host), None │
│ 70 │ │ ) │
│ 71 │ │
│ ❱ 72 │ for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM): │
│ 73 │ │ af, socktype, proto, canonname, sa = res │
│ 74 │ │ sock = None │
│ 75 │ │ try: │
│ │
│ /opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/lib/python │
│ 3.10/socket.py:955 in getaddrinfo │
│ │
│ 952 │ # We override this function since we want to translate the numeric family │
│ 953 │ # and socket type values to enum constants. │
│ 954 │ addrlist = [] │
│ ❱ 955 │ for res in _socket.getaddrinfo(host, port, family, type, proto, flags): │
│ 956 │ │ af, socktype, proto, canonname, sa = res │
│ 957 │ │ addrlist.append((_intenum_converter(af, AddressFamily), │
│ 958 │ │ │ │ │ │ _intenum_converter(socktype, SocketKind), │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
gaierror: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
╭───────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:703 in urlopen │
│ │
│ 700 │ │ │ │ self._prepare_proxy(conn) │
│ 701 │ │ │ │
│ 702 │ │ │ # Make the request on the httplib connection object. │
│ ❱ 703 │ │ │ httplib_response = self._make_request( │
│ 704 │ │ │ │ conn, │
│ 705 │ │ │ │ method, │
│ 706 │ │ │ │ url, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:386 in _make_request │
│ │
│ 383 │ │ │
│ 384 │ │ # Trigger any extra validation we need to do. │
│ 385 │ │ try: │
│ ❱ 386 │ │ │ self._validate_conn(conn) │
│ 387 │ │ except (SocketTimeout, BaseSSLError) as e: │
│ 388 │ │ │ # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout. │
│ 389 │ │ │ self._raise_timeout(err=e, url=url, timeout_value=conn.timeout) │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:1042 in _validate_conn │
│ │
│ 1039 │ │ │
│ 1040 │ │ # Force connect early to allow us to validate the connection. │
│ 1041 │ │ if not getattr(conn, "sock", None): # AppEngine might not have `.sock` │
│ ❱ 1042 │ │ │ conn.connect() │
│ 1043 │ │ │
│ 1044 │ │ if not conn.is_verified: │
│ 1045 │ │ │ warnings.warn( │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connection.py:358 in connect │
│ │
│ 355 │ │
│ 356 │ def connect(self): │
│ 357 │ │ # Add certificate verification │
│ ❱ 358 │ │ self.sock = conn = self._new_conn() │
│ 359 │ │ hostname = self.host │
│ 360 │ │ tls_in_tls = False │
│ 361 │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connection.py:186 in _new_conn │
│ │
│ 183 │ │ │ ) │
│ 184 │ │ │
│ 185 │ │ except SocketError as e: │
│ ❱ 186 │ │ │ raise NewConnectionError( │
│ 187 │ │ │ │ self, "Failed to establish a new connection: %s" % e │
│ 188 │ │ │ ) │
│ 189 │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x12f125db0>: Failed to
establish a new connection: [Errno 8] nodename nor servname provided, or not known
During handling of the above exception, another exception occurred:
╭───────────────────────────── Traceback (most recent call last) ──────────────────────────────╮
│ /opt/homebrew/bin/kedro:8 in <module> │
│ │
│ 5 from kedro.framework.cli import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kedro/framework/cli/cli.py:211 in main │
│ │
│ 208 │ """ │
│ 209 │ _init_plugins() │
│ 210 │ cli_collection = KedroCLI(project_path=Path.cwd()) │
│ ❱ 211 │ cli_collection() │
│ 212 │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1130 in __call__ │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kedro/framework/cli/cli.py:139 in main │
│ │
│ 136 │ │ ) │
│ 137 │ │ │
│ 138 │ │ try: │
│ ❱ 139 │ │ │ super().main( │
│ 140 │ │ │ │ args=args, │
│ 141 │ │ │ │ prog_name=prog_name, │
│ 142 │ │ │ │ complete_var=complete_var, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1055 in main │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1657 in invoke │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1657 in invoke │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:1404 in invoke │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/core.py:760 in invoke │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/click/decorators.py:26 in new_func │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/cli.py:53 in list_pipelines │
│ │
│ 50 def list_pipelines(ctx): │
│ 51 │ """List deployed pipeline definitions""" │
│ 52 │ context_helper = ctx.obj["context_helper"] │
│ ❱ 53 │ click.echo(context_helper.kfp_client.list_pipelines()) │
│ 54 │
│ 55 │
│ 56 @kubeflow_group.command() │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/context_helper.py:91 in kfp_client │
│ │
│ 88 │ def kfp_client(self): │
│ 89 │ │ from .kfpclient import KubeflowClient │
│ 90 │ │ │
│ ❱ 91 │ │ return KubeflowClient( │
│ 92 │ │ │ self.config, │
│ 93 │ │ │ self.project_name, │
│ 94 │ │ │ self.context, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kedro_kubeflow/kfpclient.py:39 in __init__ │
│ │
│ 36 │ │ │ │ "cookies": f"authservice_session={dex_authservice_session}" │
│ 37 │ │ │ } │
│ 38 │ │ self.host = config.host │
│ ❱ 39 │ │ self.client = Client(host=self.host, **client_params) │
│ 40 │ │ │
│ 41 │ │ self.project_name = project_name │
│ 42 │ │ self.pipeline_description = config.run_config.description │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp/_client.py:196 in __init__ │
│ │
│ 193 │ │ │ api_client) │
│ 194 │ │ self._healthz_api = kfp_server_api.api.healthz_service_api.HealthzServiceApi( │
│ 195 │ │ │ api_client) │
│ ❱ 196 │ │ if not self._context_setting['namespace'] and self.get_kfp_healthz( │
│ 197 │ │ ).multi_user is True: │
│ 198 │ │ │ try: │
│ 199 │ │ │ │ with open(Client.NAMESPACE_PATH, 'r') as f: │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp/_client.py:410 in get_kfp_healthz │
│ │
│ 407 │ │ │ │ │ 'Failed getting healthz endpoint after {} attempts.'.format( │
│ 408 │ │ │ │ │ │ max_attempts)) │
│ 409 │ │ │ try: │
│ ❱ 410 │ │ │ │ response = self._healthz_api.get_healthz() │
│ 411 │ │ │ │ return response │
│ 412 │ │ │ # ApiException, including network errors, is the only type that may │
│ 413 │ │ │ # recover after retry. │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api/healthz_service_api.py:63 in │
│ get_healthz │
│ │
│ 60 │ │ :rtype: ApiGetHealthzResponse │
│ 61 │ │ """ │
│ 62 │ │ kwargs['_return_http_data_only'] = True │
│ ❱ 63 │ │ return self.get_healthz_with_http_info(**kwargs) # noqa: E501 │
│ 64 │ │
│ 65 │ def get_healthz_with_http_info(self, **kwargs): # noqa: E501 │
│ 66 │ │ """Get healthz data. # noqa: E501 │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api/healthz_service_api.py:134 in │
│ get_healthz_with_http_info │
│ │
│ 131 │ │ # Authentication setting │
│ 132 │ │ auth_settings = ['Bearer'] # noqa: E501 │
│ 133 │ │ │
│ ❱ 134 │ │ return self.api_client.call_api( │
│ 135 │ │ │ '/apis/v1beta1/healthz', 'GET', │
│ 136 │ │ │ path_params, │
│ 137 │ │ │ query_params, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api_client.py:364 in call_api │
│ │
│ 361 │ │ │ then the method will return the response directly. │
│ 362 │ │ """ │
│ 363 │ │ if not async_req: │
│ ❱ 364 │ │ │ return self.__call_api(resource_path, method, │
│ 365 │ │ │ │ │ │ │ │ path_params, query_params, header_params, │
│ 366 │ │ │ │ │ │ │ │ body, post_params, files, │
│ 367 │ │ │ │ │ │ │ │ response_type, auth_settings, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api_client.py:181 in __call_api │
│ │
│ 178 │ │ │
│ 179 │ │ try: │
│ 180 │ │ │ # perform request and return response │
│ ❱ 181 │ │ │ response_data = self.request( │
│ 182 │ │ │ │ method, url, query_params=query_params, headers=header_params, │
│ 183 │ │ │ │ post_params=post_params, body=body, │
│ 184 │ │ │ │ _preload_content=_preload_content, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/api_client.py:389 in request │
│ │
│ 386 │ │ │ │ _request_timeout=None): │
│ 387 │ │ """Makes the HTTP request using RESTClient.""" │
│ 388 │ │ if method == "GET": │
│ ❱ 389 │ │ │ return self.rest_client.GET(url, │
│ 390 │ │ │ │ │ │ │ │ │ │ query_params=query_params, │
│ 391 │ │ │ │ │ │ │ │ │ │ _preload_content=_preload_content, │
│ 392 │ │ │ │ │ │ │ │ │ │ _request_timeout=_request_timeout, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/rest.py:230 in GET │
│ │
│ 227 │ │
│ 228 │ def GET(self, url, headers=None, query_params=None, _preload_content=True, │
│ 229 │ │ │ _request_timeout=None): │
│ ❱ 230 │ │ return self.request("GET", url, │
│ 231 │ │ │ │ │ │ │ headers=headers, │
│ 232 │ │ │ │ │ │ │ _preload_content=_preload_content, │
│ 233 │ │ │ │ │ │ │ _request_timeout=_request_timeout, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/kfp_server_api/rest.py:208 in request │
│ │
│ 205 │ │ │ │ │ raise ApiException(status=0, reason=msg) │
│ 206 │ │ │ # For `GET`, `HEAD` │
│ 207 │ │ │ else: │
│ ❱ 208 │ │ │ │ r = self.pool_manager.request(method, url, │
│ 209 │ │ │ │ │ │ │ │ │ │ │ fields=query_params, │
│ 210 │ │ │ │ │ │ │ │ │ │ │ preload_content=_preload_content, │
│ 211 │ │ │ │ │ │ │ │ │ │ │ timeout=timeout, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/request.py:74 in request │
│ │
│ 71 │ │ urlopen_kw["request_url"] = url │
│ 72 │ │ │
│ 73 │ │ if method in self._encode_url_methods: │
│ ❱ 74 │ │ │ return self.request_encode_url( │
│ 75 │ │ │ │ method, url, fields=fields, headers=headers, **urlopen_kw │
│ 76 │ │ │ ) │
│ 77 │ │ else: │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/request.py:96 in request_encode_url │
│ │
│ 93 │ │ if fields: │
│ 94 │ │ │ url += "?" + urlencode(fields) │
│ 95 │ │ │
│ ❱ 96 │ │ return self.urlopen(method, url, **extra_kw) │
│ 97 │ │
│ 98 │ def request_encode_body( │
│ 99 │ │ self, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/poolmanager.py:376 in urlopen │
│ │
│ 373 │ │ if self._proxy_requires_url_absolute_form(u): │
│ 374 │ │ │ response = conn.urlopen(method, url, **kw) │
│ 375 │ │ else: │
│ ❱ 376 │ │ │ response = conn.urlopen(method, u.request_uri, **kw) │
│ 377 │ │ │
│ 378 │ │ redirect_location = redirect and response.get_redirect_location() │
│ 379 │ │ if not redirect_location: │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:815 in urlopen │
│ │
│ 812 │ │ │ log.warning( │
│ 813 │ │ │ │ "Retrying (%r) after connection broken by '%r': %s", retries, err, ur │
│ 814 │ │ │ ) │
│ ❱ 815 │ │ │ return self.urlopen( │
│ 816 │ │ │ │ method, │
│ 817 │ │ │ │ url, │
│ 818 │ │ │ │ body, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:815 in urlopen │
│ │
│ 812 │ │ │ log.warning( │
│ 813 │ │ │ │ "Retrying (%r) after connection broken by '%r': %s", retries, err, ur │
│ 814 │ │ │ ) │
│ ❱ 815 │ │ │ return self.urlopen( │
│ 816 │ │ │ │ method, │
│ 817 │ │ │ │ url, │
│ 818 │ │ │ │ body, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:815 in urlopen │
│ │
│ 812 │ │ │ log.warning( │
│ 813 │ │ │ │ "Retrying (%r) after connection broken by '%r': %s", retries, err, ur │
│ 814 │ │ │ ) │
│ ❱ 815 │ │ │ return self.urlopen( │
│ 816 │ │ │ │ method, │
│ 817 │ │ │ │ url, │
│ 818 │ │ │ │ body, │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/connectionpool.py:787 in urlopen │
│ │
│ 784 │ │ │ elif isinstance(e, (SocketError, HTTPException)): │
│ 785 │ │ │ │ e = ProtocolError("Connection aborted.", e) │
│ 786 │ │ │ │
│ ❱ 787 │ │ │ retries = retries.increment( │
│ 788 │ │ │ │ method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2] │
│ 789 │ │ │ ) │
│ 790 │ │ │ retries.sleep() │
│ │
│ /opt/homebrew/lib/python3.10/site-packages/urllib3/util/retry.py:592 in increment │
│ │
│ 589 │ │ ) │
│ 590 │ │ │
│ 591 │ │ if new_retry.is_exhausted(): │
│ ❱ 592 │ │ │ raise MaxRetryError(_pool, url, error or ResponseError(cause)) │
│ 593 │ │ │
│ 594 │ │ log.debug("Incremented Retry for (url='%s'): %r", url, new_retry) │
│ 595 │
╰──────────────────────────────────────────────────────────────────────────────────────────────╯
MaxRetryError: HTTPSConnectionPool(host='<host>', port=443): Max retries exceeded with url:
/apis/v1beta1/healthz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object
at 0x12f125db0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided,
or not known'))It looks you didn't provide a valid Kubeflow Pipelines URL when doing kedro kubeflow init, it can be fixed by editing the generated kubeflow.yml file in conf directory. It should look like this:
host: https://<your-kubeflow-base-url>/pipelineAlso, if you use Dex for authentication, you need additional setup, as described in the docs
cpereir1
commented
1 year ago Hi @szczeles thank you. Regarding the Dex authentication additional setup: we would like to avoid hardcoding a static user & password, and we disabled this by design choice. Would there be other, more dynamic ways, of authenticating a kedro kubeflow plugin user (from local environment) to the kubeflow pipelines (living in a cloud kubernetes cluter behind dex)?
cpereir1
commented
1 year ago Also, could you confirm that, if the kubeflow pipelines url is correct, but the error persists, the reason should be inability to authenticate? Thank you!
szczeles
commented
1 year ago @cpereir1 Unfortunately, at the time, Dex doesn't allow programmatic authentication, see: https://github.com/kubeflow/kfctl/issues/140. Ideally, we would just add ID Token in the headers and Dex would check its validity and return a local token (exchanged for session cookie) for kubeflow services interaction. Unfortunately, the feature request is still open since 2019...
And yes, if kubeflow pipelines URL is correct but you still see an exception, it's likely due to an authentication issue (with no DEX_* env vars set the plugin doesn't try to authenticate at all).
Hi, I am trying to turn the kedro iris sample project into a kubeflow pipelines deployment.
I have the following in place: 1) Local installation of kedro and successful execution of the iris project 2) Cloud deployment of kubeflow (with pipelines) to which I want to push the kedro-iris project 3) The kedro project containerized and available at a remote container registry following the instructions here. All steps seem to work fine until:
Running kedro kubeflow upload-pipeline which gives me the following error:
This is my kubeflow.yaml:
What could I be doing wrong?
Local Kedro version: 0.18.4
Thank you!