Closed jlewi closed 1 year ago
@jlewi @yanniszark Here's an approach: Using Dex's Cross-client trust and authorized party to generate tokens for the SDK from oidc-authservice app.
Haven't tested this approach yet.
Yeah it's more complex than you'd expect as dex doesn't support the password grant flow. We (Seldon) have so far only been able to do it in flows that include the browser.
FWIW, argocd uses dex and lets you login just from the CLI. But I believe it's doing that by actually running a browser in the background.
The argocd CLI code to perform the login starts at https://github.com/argoproj/argo-cd/blob/master/cmd/argocd/commands/login.go
I was able to authenticate following this flow. Not sure if it's the correct path but it worked for me using this installation https://www.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/#add-static-users-for-basic-auth.
0) curl -v http://$SERVICE:$PORT
Response:
>> <a href="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=STATE_VALUE">Found</a>.
STATE=STATE_VALUE
1) curl -v "http://$SERVICE:$PORT/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=$STATE_VALUE"
Response:
>> <a href="/dex/auth/local?req=REQ_VALUE">Found</a>
REQ=REQ_VALUE
2) curl -v 'http://$SERVICE:$PORT/dex/auth/local?req=REQ_VALUE' -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234'
3) curl -v 'http://$SERVICE:$PORT/dex/approval?req=$REQ_VALUE'
Response:
>> <a href="/login/oidc?code=CODE_VALUE&state=STATE_VALUE">See Other</a>.
CODE=CODE_VALUE
4) curl -v 'http://$SERVICE:$PORT/login/oidc?code=$CODE_VALUE&state=$STATE_VALUE'
Response:
>> set cookie authservice_session=SESSION
curl -v 'http://$SERVICE:$PORT/pipeline/apis/v1beta1/pipelines' -H 'Cookie: authservice_session=SESSION'
Response:
>> 200 OK { ... }
@jlewi If IAP supports programmatic authentication, that means KFServing can use same istio ingress gateway. kfserving-ingressgateway without auth for IAP is not for authentication purpose but only knative probe issue?
@Jeffwan I don't know the answer to that question; I had have to defer to the kfserving folks.
Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.
Issue Label Bot is not confident enough to auto-label this issue. See dashboard for more details.
I'm having issues with programatically authenticating using dex for an on-prem install. Is there any progress on this or some standard way of authenticating using dex?
Running into this issue as well, which is unfortunately preventing us from upgrading to 1.1.
@rafaelbarreto87 For our purposes, this was solved. See: https://github.com/kubeflow/kubeflow/issues/5345
Issue-Label Bot is automatically applying the labels:
Label | Probability |
---|---|
area/istio | 0.56 |
Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback! Links: app homepage, dashboard and code for this bot.
@rafaelbarreto87 For our purposes, this was solved. See: kubeflow/kubeflow#5345
@ConverJens Thanks a lot! This solves for some of our use cases as well (i.e. deploying pipelines from shell)! Really appreciated.
Unfortunately, I think it would be great to have a more permanent and general solution for this (e.g. OAuth2) because some processes are not interactive and this method depend on the UI, which is something that could easily change. Either way, it's a workable solution for now, so thanks a lot again!
I agree
@rafaelbarreto87 For our purposes, this was solved. See: kubeflow/kubeflow#5345
@ConverJens Thanks a lot! This solves for some of our use cases as well (i.e. deploying pipelines from shell)! Really appreciated.
@rafaelbarreto87 Happy to help!
Unfortunately, I think it would be great to have a more permanent and general solution for this (e.g. OAuth2) because some processes are not interactive and this method depend on the UI, which is something that could easily change. Either way, it's a workable solution for now, so thanks a lot again!
I agree. A more stable solution in necessary for the long term!
Here is a workaround to authenticate for kfp.Client()
in Python:
import requests
import kfp
HOST = "http://localhost:8080/"
USERNAME = "admin@kubeflow.org"
PASSWORD = "12341234"
NAMESPACE = "your-kf-ui-namespace"
session = requests.Session()
response = session.get(HOST)
headers = {
"Content-Type": "application/x-www-form-urlencoded",
}
data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
session_cookie = session.cookies.get_dict()["authservice_session"]
client = kfp.Client(
host=f"{HOST}/pipeline",
cookies=f"authservice_session={session_cookie}",
namespace=NAMESPACE,
)
print(client.list_pipelines())
/cc @yanniszark @cventes
@yanp Thanks for this snippet. I've tested it, but I cannot get it to work yet. As I am using kfp from a remote client (mac -> ki-server3), I set kubectl edit svc authservice -n istio-system
to NodePort on 30080. Thus I am able to reach the Pod, but no cookies are fetched (assuming NAMESPACE = 'kubeflow' for default installation).
Any ideas what I am doing wrong?
USERNAME = "admin@kubeflow.org"
PASSWORD = "12341234"
#NAMESPACE = "your-kf-ui-namespace"
NAMESPACE = "kubeflow" # "admin" does not work either
HOST = 'http://ki-server3:30080/'
session = requests.Session()
response = session.get(HOST)
headers = {
"Content-Type": "application/x-www-form-urlencoded",
}
data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
print(session.cookies.get_dict())
session_cookie = session.cookies.get_dict()["authservice_session"]
client = kfp.Client(
host=f"{HOST}/pipeline",
cookies=f"authservice_session={session_cookie}",
namespace=NAMESPACE,
)
print(client.list_pipelines())
Debugging Info:
# response
...
response.OK = True
response.reason = 'OK'
response.status_code = 200
...
# session
session.auth = None
session.cert = None
cookies = <RequestCookieJar[]>
In short RequestCookieJar[]
is empty: {}
http://ki-server3:30080/
Hi @tomalbrecht, were you able to authenticate the Web UI on browser using url http://ki-server3:30080/
and USERNAME
/PASSWORD
above?
@yanp
Well, no. Not with USERNAME /PASSWORD. Port 30080 points to istio-system/authservice. When opening http://ki-server3:30080/
I get OK
as an result. While opening https://ki-server3:31390
will point to dex and will ask for PASSWORD/USERNAME.
kubectl get svc -A | grep 8080
istio-system authservice NodePort 10.152.183.145 <none> 8080:30080/TCP 6d12h
But this pointed me to the right direction. Thank you. I have to use the service istio-system/dex for authentication (https://ki-server3:31390/
in my setup).
Now at least I get the cookie, but struggling with SSL certification in kfp.Client connecting from client -> server.
Traceback (most recent call last):
File "/Users/tom/workspaces/ws-trevisto/stack/tests/wip_pipeline.py", line 144, in <module>
print(client.list_pipelines())
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp/_client.py", line 458, in list_pipelines
return self._pipelines_api.list_pipelines(page_token=page_token, page_size=page_size, sort_by=sort_by)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api/pipeline_service_api.py", line 1222, in list_pipelines
return self.list_pipelines_with_http_info(**kwargs) # noqa: E501
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api/pipeline_service_api.py", line 1327, in list_pipelines_with_http_info
collection_formats=collection_formats)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 383, in call_api
_preload_content, _request_timeout, _host)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 199, in __call_api
_request_timeout=_request_timeout)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/api_client.py", line 407, in request
headers=headers)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/rest.py", line 248, in GET
query_params=query_params)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/kfp_server_api/rest.py", line 226, in request
headers=headers)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/request.py", line 76, in request
method, url, fields=fields, headers=headers, **urlopen_kw
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/request.py", line 97, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/poolmanager.py", line 330, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 760, in urlopen
**response_kw
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
File "/Users/tom/miniconda3/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ki-server3', port=31390): Max retries exceeded with url: //pipeline/apis/v1beta1/pipelines?page_token=&page_size=10&sort_by= (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Seems to be related to https://github.com/kubeflow/pipelines/issues/3095
@tomalbrecht I'm not sure about the ssl issue, but I had my port point to svc/istio-ingressgateway
instead of dex
, following the doc: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#accessing-kubeflow Hope it helps!
@yanp My fault. Port 31390 refers to svc/istio-ingressgateway
and will forward to dex.
Did you change your gateway specs to use httpsRedirect: https://www.kubeflow.org/docs/started/k8s/kfctl-istio-dex/#secure-with-https?
@tomalbrecht No I didn’t do any customization. I’d recommend asking for more help on the slack channel https://www.kubeflow.org/docs/about/community/#slack-community-and-channels
@tomalbrecht BTW namespace for client should be the one showing up on web UI, not necessarily kubeflow
For the records. Using http instead of ssl does work as a workaround on my microk8s manifest installation.
httpsRedirect has to be disabled:
kubectl edit -n kubeflow gateways.networking.istio.io kubeflow-gateway
# set spec.hosts.tls.httpsRedirect: false
Then, the following code will work for me.
USERNAME = "admin@kubeflow.org"
PASSWORD = "12341234"
NAMESPACE = "admin"
HOST = 'http://ki-server3:31380/'
session = requests.Session()
response = session.get(HOST)
headers = {
"Content-Type": "application/x-www-form-urlencoded",
}
data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
print(session.cookies.get_dict())
session_cookie = session.cookies.get_dict()["authservice_session"]
print(session_cookie)
client = kfp.Client(
host=f"{HOST}/pipeline",
cookies=f"authservice_session={session_cookie}",
namespace=NAMESPACE
)
print(client.list_pipelines())
Thanks for your help @yanp
+1 for all the helpful solutions to auth, thank you.
Any recommendations on how to do this in the reverse? Meaning once we have a cookie generated, is there anyway to validate that it is a valid cookie? For multi-user KfP I am able to verify this but only for a specific user or namespace, and am looking for general way to verify a cookie. Might be that this is flawed approach but wanted to put it in here.
I was able to authenticate following this flow. Not sure if it's the correct path but it worked for me using this installation https://www.kubeflow.org/docs/started/k8s/kfctl-existing-arrikto/#add-static-users-for-basic-auth.
0) curl -v http://$SERVICE:$PORT Response: >> <a href="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=STATE_VALUE">Found</a>. STATE=STATE_VALUE 1) curl -v "http://$SERVICE:$PORT/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=$STATE_VALUE" Response: >> <a href="/dex/auth/local?req=REQ_VALUE">Found</a> REQ=REQ_VALUE 2) curl -v 'http://$SERVICE:$PORT/dex/auth/local?req=REQ_VALUE' -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234' 3) curl -v 'http://$SERVICE:$PORT/dex/approval?req=$REQ_VALUE' Response: >> <a href="/login/oidc?code=CODE_VALUE&state=STATE_VALUE">See Other</a>. CODE=CODE_VALUE 4) curl -v 'http://$SERVICE:$PORT/login/oidc?code=$CODE_VALUE&state=$STATE_VALUE' Response: >> set cookie authservice_session=SESSION curl -v 'http://$SERVICE:$PORT/pipeline/apis/v1beta1/pipelines' -H 'Cookie: authservice_session=SESSION' Response: >> 200 OK { ... }
Thanks @maurogonzalez for saving my day. this worked perfectly with my local kubeflow + dex in wsl2 converted into a nice little function to extract SESSION in my curl calls: If your endpoint is not localhost:8080 you can call the function with host and port args.
dex_session () {
if [ $# -eq 0 ]; then HOST="localhost"; PORT="8080"; else HOST="$1"; PORT="$2";fi
STATE=$(curl -s http://${HOST}:${PORT} | grep -oP '(?<=state=)[^ ]*"' | cut -d \" -f1)
REQ=$(curl -s "http://${HOST}:${PORT}/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=$STATE" | grep -oP '(?<=req=)\w+')
curl -s "http://${HOST}:${PORT}/dex/auth/local?req=$REQ" -H 'Content-Type: application/x-www-form-urlencoded' --data 'login=admin%40kubeflow.org&password=12341234'
CODE=$(curl -s "http://${HOST}:${PORT}/dex/approval?req=$REQ" | grep -oP '(?<=code=)\w+')
curl -s --cookie-jar - "http://${HOST}:${PORT}/login/oidc?code=$CODE&state=$STATE" > .dex_session
echo $(cat .dex_session | grep 'authservice_session' | awk '{print $NF}')
}
Here's something that I came up with:
import logging
import os
import requests
from urllib.parse import parse_qs
host = 'https://your.inference.service'
def get_auth_session_token(host):
response = requests.get(host, verify=False, allow_redirects=False)
location = response.headers['location']
state = parse_qs(location)['state'][0]
location_prefix, _ = location.split('state=')
location = location_prefix + f'state={state}'
url = host + location
response = requests.get(url, verify=False, allow_redirects=False)
location = response.headers['location']
req_value = location.split('=')[1]
with requests.Session() as s:
data = {
'login': os.getenv('KUBEFLOW_USERNAME', 'EMAIL-GOES-HERE'),
'password': os.getenv('KUBEFLOW_PASSWORD', 'PASSWORD-GOES-HERE'),
}
url = f'{host}/dex/auth/local?req={req_value}'
logging.debug(url)
response = requests.post(url, data=data, verify=False, allow_redirects=False)
location = response.headers['location']
req_value = location.split('=')[1]
url = f'{host}/dex/approval?req={req_value}'
logging.debug(url)
response = requests.get(url, verify=False, allow_redirects=False)
location = response.headers['location']
url = host + location
response = requests.get(url, verify=False, allow_redirects=False)
logging.debug(url)
return response.headers['set-cookie']
如何通过java进行dex认证,访问kubeflow集群呢?
You can translate the Python code and do it in Java too. There's nothing specific to Python in the code. :)
On Thu, Nov 11, 2021 at 3:11 PM LiuZhiYuan123456 @.***> wrote:
如何通过java进行dex认证,访问kubeflow集群呢?
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubeflow/kfctl/issues/140#issuecomment-966046151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGSINAZC43RZKSPHV3GRELULNUAZANCNFSM4J3DGADQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Thank you for help @benjamintanweihao I have idea
This worked for me for authentication with static user credentials on DEX
import os
from urllib.parse import urlparse, parse_qs
import kfp
from requests_html import HTMLSession
KF_SESSION_COOKIE_NAME = "authservice_session"
KF_UI_HOST = os.getenv("KF_PIPELINES_UI_ENDPOINT", "https://kubeflow.example.com")
USERNAME = os.environ["KF_USER_EMAIL"]
PASSWORD = os.environ["KF_USER_PASSWORD"]
NAMESPACE = os.environ["KF_USER_NAMESPACE"]
def get_auth_cookie_from_dex(url, username, password):
"""Get session cookie from Kubeflow login."""
dex_local_auth_path = "/dex/auth/local"
url = url.rstrip("/")
with HTMLSession() as session:
# First redirect to Dex Login Page https://kubeflow.example.com/dex/auth/
response = session.get(url=url, allow_redirects=True)
dex_login_paths = response.html.xpath("//a/@href")
dex_local_path = [link for link in dex_login_paths if dex_local_auth_path in link]
error_msg = "Only one local auth link is expected from DEX login page."
assert len(dex_local_path) == 1, error_msg
# Login on DEX with User credentials
dex_login_url = f"{url}{dex_local_path[0]}"
dex_request_token = parse_qs(urlparse(dex_login_url).query)["req"]
response = session.post(
url=dex_login_url,
headers={"Content-Type": "application/x-www-form-urlencoded"},
data=dict(login=username, password=password),
allow_redirects=False
)
assert response.status_code == 303, "Login Failure. Please check credentials."
# Access DEX approval path to get OIDC code and state
dex_approval_url = f"{url}/dex/approval"
response = session.get(
url=dex_approval_url,
params=dict(req=dex_request_token),
allow_redirects=False
)
assert response.status_code == 303, "DEX Approval failed after successful login."
# Access DEX OIDC path to generate session cookie
dex_oidc_url = response.html.xpath("//a/@href")[0]
response = session.get(
url=dex_oidc_url,
allow_redirects=False
)
assert response.status_code == 302, "DEX Auth session creation failed."
# Extract session cookie
cookie_jar = response.cookies.get_dict()
session_cookie = cookie_jar[KF_SESSION_COOKIE_NAME]
return session_cookie
auth_session_cookie = get_auth_cookie_from_dex(KF_UI_HOST, USERNAME, PASSWORD)
client = kfp.Client(
host=f"{KF_UI_HOST}/pipeline",
cookies=f"{KF_SESSION_COOKIE_NAME}={auth_session_cookie}",
)
print("Pipelines:\n" + client.list_pipelines().to_str() + "\n")
print("Experiments:\n" + client.list_experiments(namespace=NAMESPACE).to_str() + "\n")
Here is a workaround to authenticate for
kfp.Client()
in Python:import requests import kfp HOST = "http://localhost:8080/" USERNAME = "admin@kubeflow.org" PASSWORD = "12341234" NAMESPACE = "your-kf-ui-namespace" session = requests.Session() response = session.get(HOST) headers = { "Content-Type": "application/x-www-form-urlencoded", } data = {"login": USERNAME, "password": PASSWORD} session.post(response.url, headers=headers, data=data) session_cookie = session.cookies.get_dict()["authservice_session"] client = kfp.Client( host=f"{HOST}/pipeline", cookies=f"authservice_session={session_cookie}", namespace=NAMESPACE, ) print(client.list_pipelines())
Hello all, I am trying this on-prem on Docker image. The localhost://8080 works separately but I can't access connection:
Tried:`
import requests
import kfp
HOST = "http://locahost:8080/"
USERNAME = "user@example.com"
PASSWORD = "12341234"
NAMESPACE = "kubeflow-user-example-com"
session = requests.Session()
response = session.get(HOST)
headers = {
"Content-Type": "application/x-www-form-urlencoded",
}
data = {"login": USERNAME, "password": PASSWORD}
session.post(response.url, headers=headers, data=data)
session_cookie = session.cookies.get_dict()["authservice_session"]
client = kfp.Client(
host=f"{HOST}/pipeline",
cookies=f"authservice_session={session_cookie}",
namespace=NAMESPACE
)
Error:
NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f4104615df0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution
followed by
MaxRetryError: HTTPConnectionPool(host='pipelines-api.kubeflow.svc.cluster.local', port=8888): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f40fbf6e6d0>: Failed to establish a new connection: [Errno -2] Name or service not known')) `
Tried quite some combinations of hosts :
Also tried User:
user@example.com
Same error message.
Kindly help me initiate a Pipeline with Kubeflow. Thanks in advance.
The Kubeflow docs explain how to do programmatic authentication with dex and Kubeflow Pipelines.
You can find the information and code examples on the "Connect the Pipelines SDK to Kubeflow Pipelines" page under the "Full Kubeflow (from outside cluster)"
heading.
My only other tip is that you should catch errors when calling methods like kfp.Client.list_experiments()
to handle the case that your session lasts long enough for your cookie to expire (and re-auth in that case).
I am going to close this issue for now, as those docs fully explain how to achieve what this issue is asking for.
/close
@thesuperzapper: Closing this issue.
@yanniszark @krishnadurai How do folks do programmatic authentication using DEX?
Here's the use case; suppose you are running code outside the cluster and you want to programmatically access a service running inside the cluster e.g.
My expectation would be that the client code can obtain a credential and add it as an Auth Header in order to pass the AuthZ check performed at the ISTIO ingress-gateway.
For reference we follow this pattern on GCP.
Here are the instructions for doing programmatic authentication with IAP https://cloud.google.com/iap/docs/authentication-howto
And for example here's the code in the KFP SDK for doing auth using IAP. https://github.com/kubeflow/pipelines/blob/9ad7d7dd9776ce75a83712f5723db2ef93ba5c26/sdk/python/kfp/_auth.py#L41
@animeshsingh @jinchihe We might want to think about building a Kubeflow Auth library as part of a Kubeflow SDK so that each application (KFP, KFServing, etc...) doesn't have to implement it themselves.