YetiCuzMountain commented 1 year ago

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Search before asking

[X] I have searched in the issues and found no similar issues.

Describe the bug

i used curl to post rest request for creating a sessions , but both http and https didn't work.

curl -v debug info show as follows:

# curl -v -X 'POST' --negotiate -u : 'https://myhostname:10099/api/v1/sessions'

About to connect() to myhostname port 10099 (#0)
Trying 10.30.17.237...
Connected to myhostname (10.30.17.237) port 10099 (#0)
Initializing NSS with certpath: sql:/etc/pki/nssdb
CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none
NSS error -12263 (SSL_ERROR_RX_RECORD_TOO_LONG)
SSL received a record that exceeded the maximum permissible length.
Closing connection 0 curl: (35) SSL received a record that exceeded the maximum permissible length.

# curl -v -X 'POST' --negotiate -u : 'http://myhostname:10099/api/v1/sessions'

About to connect() to myhostname port 10099 (#0)
Trying 10.30.17.237...
Connected to myhostname (10.30.17.237) port 10099 (#0)

POST /api/v1/sessions HTTP/1.1 User-Agent: curl/7.29.0 Host: myhostname:10099 Accept: /

< HTTP/1.1 500 Internal Server Error < Date: Wed, 18 Oct 2023 01:49:37 GMT < Content-Type: application/json < Content-Length: 16 < Server: Jetty(9.4.52.v20230823) <
Connection #0 to host myhostnameleft intact {"message":null}

It seems like that jetty server is not configed with ssl on, but i don't know how to fix it.

Thanks very much for your helps!

Affects Version(s)

master/1.7.0

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

kyuubi.authentication=KERBEROS
kyuubi.kinit.principal=hive/myhostname@BIGDATA.CTY.CN
kyuubi.kinit.keytab=/etc/security/keytabs/hive.keytab

kyuubi.backend.server.event.loggers=JSON
kyuubi.backend.server.event.json.log.path=/var/log/kyuubi/event

kyuubi.ha.addresses=nm-bigdata-030017237.ctc.local:2181,nm-bigdata-030017238.ctc.local:2181,nm-bigdata-030017239.ctc.local:2181
kyuubi.ha.namespace=kyuubi_root

kyuubi.frontend.rest.bind.host=10.30.17.237
kyuubi.frontend.bind.host=10.30.17.237
kyuubi.frontend.protocols=REST,THRIFT_BINARY

kyuubi.spnego.keytab=/etc/security/keytabs/HTTP.keytab
kyuubi.spnego.principal=HTTP/myhostname@BIGDATA.CTY.CN

kyuubi.server.administrators=anonymous

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

[x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
[ ] No. I cannot submit a PR at this time.

github-actions[bot] commented 1 year ago

Hello @YetiCuzMountain, Thanks for finding the time to report the issue! We really appreciate the community's efforts to improve Apache Kyuubi.

YetiCuzMountain commented 1 year ago

When I traced the related code of the kyuubi-admin script, I found the configuration “kyuubi.ctl.rest.base.url” & “ kyuubi.ctl.rest.auth.schema”

it seems that the scipt kyuubi-ctl is used to add a http request header of 'Autehtication : Negotiate : ${token}'.

And i get error as follows :

Exception in thread "main" org.apache.kyuubi.client.exception.KyuubiRestException: Failed to generate spnego auth header for 10.10.37.137
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:50)
        at org.apache.kyuubi.client.KyuubiRestClient.getAuthHeader(KyuubiRestClient.java:137)
        at org.apache.kyuubi.client.SessionRestApi.listSessions(SessionRestApi.java:41)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.$anonfun$doRun$1(ListSessionCommand.scala:36)
        at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:34)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:29)
        at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47)
        at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46)
        at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79)
        at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87)
        at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala)
Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateToken(SpnegoAuthHeaderGenerator.java:80)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:46)
        ... 11 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.doGenerateToken(SpnegoAuthHeaderGenerator.java:97)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.lambda$generateToken$0(SpnegoAuthHeaderGenerator.java:80)
        ... 15 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:466)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
        ... 19 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 25 more

YetiCuzMountain commented 1 year ago

When I traced the related code of the kyuubi-admin script, I found the configuration “kyuubi.ctl.rest.base.url” & “ kyuubi.ctl.rest.auth.schema”

it seems that the scipt kyuubi-ctl is used to add a http request header of 'Autehtication : Negotiate : ${token}'.

And i get error as follows :

Exception in thread "main" org.apache.kyuubi.client.exception.KyuubiRestException: Failed to generate spnego auth header for 10.10.37.137
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:50)
        at org.apache.kyuubi.client.KyuubiRestClient.getAuthHeader(KyuubiRestClient.java:137)
        at org.apache.kyuubi.client.SessionRestApi.listSessions(SessionRestApi.java:41)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.$anonfun$doRun$1(ListSessionCommand.scala:36)
        at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:34)
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:29)
        at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47)
        at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46)
        at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79)
        at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87)
        at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala)
Caused by: java.security.PrivilegedActionException: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateToken(SpnegoAuthHeaderGenerator.java:80)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.generateAuthHeader(SpnegoAuthHeaderGenerator.java:46)
        ... 11 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - LOOKING_UP_SERVER)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:772)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
        at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.doGenerateToken(SpnegoAuthHeaderGenerator.java:97)
        at org.apache.kyuubi.client.auth.SpnegoAuthHeaderGenerator.lambda$generateToken$0(SpnegoAuthHeaderGenerator.java:80)
        ... 15 more
Caused by: KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:73)
        at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:251)
        at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:262)
        at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)
        at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)
        at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:466)
        at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)
        ... 19 more
Caused by: KrbException: Identifier doesn't match expected value (906)
        at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
        at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
        at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
        at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
        ... 25 more

Perhaps i have to add --spnegoHost during using the script kyuubi-ctl so as to get a correct kerberos principal.

YetiCuzMountain commented 1 year ago

I used the command

bin/kyuubi-ctl list session -b --hostUrl http://myhostname:10099/api --version v1 --authSchema spnego  --spnegoHost  myhostname

to list session, and the same error occured even though the autication http header was attached to the http request.

javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
        at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) ~[?:1.8.0_251]
        at sun.security.ssl.InputRecord.read(InputRecord.java:527) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416) ~[?:1.8.0_251]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400) ~[?:1.8.0_251]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:221) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:165) ~[httpclient-4.5.14.jar:4.5.14]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:140) ~[httpclient-4.5.14.jar:4.5.14]
        at **org.apache.kyuubi.client.RestClient.doRequest**(RestClient.java:181) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.client.RestClient.get(RestClient.java:80) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.client.RestClient.get(RestClient.java:74) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_251]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_251]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_251]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_251]
        at org.apache.kyuubi.client.RetryableRestClient.invoke(RetryableRestClient.java:82) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at com.sun.proxy.$Proxy22.get(Unknown Source) ~[?:?]
        at org.apache.kyuubi.client.SessionRestApi.listSessions(SessionRestApi.java:41) ~[kyuubi-rest-client-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.$anonfun$doRun$1(ListSessionCommand.scala:36) ~[kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.RestClientFactory$.withKyuubiRestClient(RestClientFactory.scala:40) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:34) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.list.ListSessionCommand.doRun(ListSessionCommand.scala:29) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cmd.Command.run(Command.scala:47) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli.doAction(ControlCli.scala:46) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli$$anon$1.doAction(ControlCli.scala:79) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli$.main(ControlCli.scala:87) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]
        at org.apache.kyuubi.ctl.cli.ControlCli.main(ControlCli.scala) [kyuubi-ctl_2.12-1.9.0-SNAPSHOT.jar:1.9.0-SNAPSHOT]

YetiCuzMountain commented 1 year ago

After my exploration, I guess, some kyuubi fe services , such as KyuubiTHttpFrontendService and TBinaryFrontedService, support SSL, but others does not seem to support SSL, e.g KyuubiRestFrontendService.

The existing configuration are all about KyuubiTHttpFrontendService and KyuubiTHttpFrontendService , such as 'kyuubi.frontend.ssl.keystore.path' & 'kyuubi.frontend.thrift.http.ssl.keystore.password'.

pan3793 commented 1 year ago

Perhaps i have to add --spnegoHost during using the script kyuubi-ctl so as to get a correct kerberos principal.

The server principal name, a.k.a. SPN, is definitely required for Kerberos/SPNego authentication mechanism.

pan3793 commented 1 year ago

Usually, we deploy Kyuubi with plain HTTP behind the Nginx which uses HTTPs, do you plan to expose Kyuubi directly to the end users with HTTPs protocols?

YetiCuzMountain commented 1 year ago

Usually, we deploy Kyuubi with plain HTTP behind the Nginx which uses HTTPs, do you plan to expose Kyuubi directly to the end users with HTTPs protocols?

yes, no nginx , directly depolyed the 10099

YetiCuzMountain commented 1 year ago

Thanks for your instruction. i use this command : "

myhostname:10099--version v1 --authSchema spnego --spnegoHost myhostname"

and get correct response.

kyuubi-ctl --hostUrl , no "/api"

YetiCuzMountain commented 1 year ago

It seems that it will not be automatically added -H 'Authentication : Negotiate : token ' when using the 'curl' command , unless use kyuubi-ctl script.

pan3793 commented 1 year ago

It seems that it will not be automatically added -H 'Authentication : Negotiate : token ' when using the 'curl' command

it's unexpected behavior, but I have no idea what happened inside curl

YetiCuzMountain commented 1 year ago

I fixed the 'kyuubi-ctl' script for rest submission in condition of SSL + SPNEGO.

What should I do next？

pan3793 commented 1 year ago

How did you fix that? is it a usage issue or requires code changes?

YetiCuzMountain commented 1 year ago

How did you fix that? is it a usage issue or requires code changes?

Perhaps it requires code changes, in which i add some keystore Configs during initializing a JettyServer object.

YetiCuzMountain commented 1 year ago

I have fixed this problem in #5548. Thanks greatly for your instructions!

jiaoqingbo commented 6 months ago

when use curl like curl -X 'GET' --negotiate -u : "http://XXX:10099/api/v1/sessions" ，you must confirm curl feature support GSS-API/SSPI Or SPNEGO Older versions of curl that support GSS-Negotiate cannot work properly.

apache / kyuubi

[Bug] get errors when i use "curl -v -X 'POST' --negotiate -u : 'https://hostname:10099/api/v1/sessions' to start a session #5460

Code of Conduct

Search before asking

Describe the bug

Affects Version(s)

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

Are you willing to submit PR?

bin/kyuubi-ctl list session -b --hostUrl http://myhostname:10099--version v1 --authSchema spnego --spnegoHost myhostname"