Connect to minikf via sdk

[NicoG94]

What steps did you take:

import kfp

host="http://10.10.10.10:8080"
client = kfp.Client(host=host)
client.list_experiments()

What happened:

gets me this error:

Traceback (most recent call last):
  File "C:\Users\nicog\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3319, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-d6d42b2f4c92>", line 3, in <module>
    client.list_experiments()
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp\_client.py", line 307, in list_experiments
    resource_reference_key_id=namespace)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\api\experiment_service_api.py", line 350, in list_experiment
    (data) = self.list_experiment_with_http_info(**kwargs)  # noqa: E501
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\api\experiment_service_api.py", line 438, in list_experiment_with_http_info
    collection_formats=collection_formats)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\api_client.py", line 330, in call_api
    _preload_content, _request_timeout)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\api_client.py", line 161, in __call_api
    _request_timeout=_request_timeout)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\api_client.py", line 351, in request
    headers=headers)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\rest.py", line 238, in GET
    query_params=query_params)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\kfp_server_api\rest.py", line 211, in request
    headers=headers)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\urllib3\request.py", line 76, in request
    method, url, fields=fields, headers=headers, **urlopen_kw
  File "C:\Users\nicog\Anaconda3\lib\site-packages\urllib3\request.py", line 97, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\urllib3\poolmanager.py", line 378, in urlopen
    return self.urlopen(method, redirect_location, **kw)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\urllib3\poolmanager.py", line 325, in urlopen
    conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
  File "C:\Users\nicog\Anaconda3\lib\site-packages\urllib3\poolmanager.py", line 231, in connection_from_host
    raise LocationValueError("No host specified.")
urllib3.exceptions.LocationValueError: No host specified.

What did you expect to happen:

Connect to minikf and list the experiments i ran in the minikf UI

Environment:

Windows 10 64-Bit Python 3.7.4

How did you deploy Kubeflow Pipelines (KFP)? minikf locally on computer with virtual box and vagrant

KFP version: build version v1.0-56-g5ae311f8

KFP SDK version: kfp 0.4.0 kfp-server-api 0.3.0

Anything else you would like to add:

/kind bug

[Ark-kun]

Does it work if you remove "http://"? Does it work if you add a "/"?

[NicoG94]

No I still get the "urllib3.exceptions.LocationValueError: No host specified." - Error.

[numerology]

I think this is due to the fact that we're consulting the format of endpoint to determine the authentication style, see https://github.com/kubeflow/pipelines/blob/62e33c711b12ebeb60d5fb393e9484f2b297bcac/sdk/python/kfp/_client.py#L189 and other similar methods.

This is okay once before when we did not have that many deployment/authentication options, but I think it becomes non-extensible and hard to maintain now.

I am wondering if we want to be more explicit by let users pass in the authentication style as a parameter, instead of inferring from host format. For the current issue, perhaps the only workaround on top of my head is to directly use the KFP server API package.

WDYT? @rmgogogo @Ark-kun @IronPan

[NicoG94]

@numerology can you maybe tell me how the workaround with the KFP server API package would work?

[numerology]

Actually this seems to be more complicated after taking another look. I cannot get the same error using my test environment, and _load_config() will only erase the http:// but not the whole host address.

Perhaps, do you mind adding some debugging message to show what host became in _load_config()?

[NicoG94]

config.host became the host it was supposed to be, so '10.10.10.10:8080'.

Also the client._job_api.api_client.configuration.host, client._upload_api.api_client.configuration.host and other host-attributes of client are 10.10.10.10:8080.

While debuggin I realised where the host is gone missing:

urlopen() is called without the url (which contains the host) at the end of urlopen(). In line 381 in urllib3\poolmanager.py: return self.urlopen(method, redirect_location, **kw)

The url value is: '/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTU4ODQxMzE3MHxFd3dBRUhkU1F6VlZTRXN5UzNGQmRIaHFVREk9fMvJePBWFVwgl2VOXMUXSwz-YsV8FTfEdK0A1pNXYIdI'

So the host and port can not be extracted from url in line 327&328 of urllib3\poolmanager.py:

u = parse_url(url) 
conn = self.connection_from_host(u.host, port=u.port, scheme=u.scheme)

u.host and u.port are None

[NicoG94]

Also if I hardcode the host and port into urllib3\poolmanager.py and try this:

import kfp
host="http://10.10.10.10:8080/"
client = kfp.Client(host=host)#other_client_id for IAP auf GCP
print(client.list_experiments())
experiment = client.create_experiment(name="test_exp")
print(experiment)

I get this output:

{'experiments': None, 'next_page_token': None, 'total_size': None}
<IPython.core.display.HTML object>
{'created_at': None,
 'description': None,
 'id': None,
 'name': None,
 'resource_references': None}

Even though I already did a test-experiment in the UI.

[NicoG94]

btw I use PyCharm 2019.3.1 Community Edition, don't know if that might be the problem

[Ark-kun]

File "C:\Users\nicog\Anaconda3\lib\site-packages\urllib3\request.py", line 76, in request

Can you please try dumping the HTTP request. If the request looks sane, then you can use Fiddler or command-line tools like curl or wget to investigate how to tweak the request to make it work.

[rmgogogo]

I don't think "host" parameter is supported with minikf as no context for authentication.

Or you may simply change the SDK client to allow no-authentication to directly hit the API server.

[yanniszark]

Hi everyone, just wanted to shed some light into MiniKF's auth configuration in case it helps with this issue. In MiniKF, every request to the Istio Gateway (10.10.10.10:8080) is checked for authentication credentials. Currently, this credential is a cookie in the user's browser, support for programmatic authentication isn't there yet.

AFAIK, the available solutions are the following:

1. Copy the session value from the user header and put it inside the X-Auth-Token header for every request. This is a hack because there isn't support for programmatic authentication yet (we are planning for it soon though).
2. Proxy requests through the K8s API Server, provided you have a valid kubeconfig. I believe the KFP client supports this (@Ark-kun can you confirm?).

Based on these, what is the KFP client configuration needed for (2)?

[CRSilkworth]

I'm currently trying to fumble my way through this.

@yanniszark What exactly would be required to implement either of these work arounds? Do you have an example of what (1) would look like? or (2) or that matter?

[rmgogogo]

Here are few tips.

1. please check whether you can open the KFP UI via the IP
2. please check whether the UI can list the pipelines (use Chorme etc. dev tool to check the network request/response). If can't work, please check your minikf etc. setup.
3. In previous step, you are expected to see "/apis/v1beta1/pipelines" request.
4. Directly http get http://your-kfp-fe-host-port/apis/v1beta1/pipelines to see if you can get response. Expected yes.
5. Check/debug the code logics here to see if need special changes. Welcome contribution.

[yanniszark]

@CRSilkworth sure, For (1), see https://github.com/kubeflow/kfserving/tree/master/docs/samples/istio-dex#authentication. For (2), afaik you only need a valid kubeconfig file, but not sure how to force the client to connect that way, the pipelines team should know more.

[CRSilkworth]

OK, thanks guys. I'll take a look at it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[nroberts1]

@yanniszark Do you know if there is an issueI can monitor to see the progress of "there isn't support for programmatic authentication yet (we are planning for it soon though)"? This would have been great to have, I'm assuming none of the security related params for kfp.Client will currently help me? _kfp.Client(host=None, client_id=None, namespace='kubeflow', other_client_id=None, other_client_secret=None, existing_token=None, cookies=None, proxy=None, ssl_ca_cert=None, kubecontext=None, credentials=None)