After upgrading to 1.10.1 from 1.8 kibana login is not working

[sushovan23]

Describe the bug performed upgrade to 1.10.1 from 1.8. All elasticsearch nodes are up and cluster is green. Before upgrade cluster was configured with openid/sso. we were able to login correctly. After upgrade when I try to login to kibana its giving following error:

{"statusCode":400,"error":"Bad Request","message":"[request query.session_state]: definition for this key is missing"}

To Reproduce Steps to reproduce the behavior:

1. upgrade to 1.10.1 from 1.8 (elasticsearch should be configured
2. upgrade opendistro kibana
3. try to login
4. See error {"statusCode":400,"error":"Bad Request","message":"[request query.session_state]: definition for this key is missing"} Expected behavior Able to login to kibana using sso.

Configuration (please complete the following information):

• ODFE/Kibana version [1.10.1]
• Distribution [RPM]
• Host Machine [Centos7]

Did something change in the way security plugin is configured? Because I have it working correctly in 1.6 and 1.8.

[lukasmrtvy]

I have same problem, I am using openid_auth_domain ,Azure AD and Opendistro 1.10.1.

[lukasmrtvy]

Related: https://stackoverflow.com/questions/53650185/azure-active-directory-v2-0-doesnt-seem-to-support-oauth-session-management. Problem ( at least for me and Azure AD ) is that v2 endpoint https://login.microsoftonline.com/<tenantid>/v2.0/.well-known/openid-configuration does not support session_state.

[sushovan23]

We are using v2.0 in opendistro 1.6 and 1.8. Those are working perfectly. So something must have changed in 1.10 which has caused it to break.

[gaiksaya]

Hi @sushovan23 ,

Sorry for the issues faced by upgrading the ODFE to v1.10.1. We have added the instructions to address this. Please check https://opendistro.github.io/for-elasticsearch-docs/docs/upgrade/1-10-1/ Please let us know if you continue to face the issue. Thanks!

[chandu484]

Hello @gaiksaya Thanks for your reply! I'm following this thread as I'm also facing same issue with Okta OIDC connector. Even with that setting its not working! Seeing same error {“statusCode”:403,“error”:“Forbidden”,“message”:“Request failed authentication.”}

[lukasmrtvy]

@gaiksaya setting opendistro_security.cookie.secure: false to kibana.yml does not work for me. Still getting:

{
"statusCode": 400,
"error": "Bad Request",
"message": "[request query.session_state]: definition for this key is missing"
}

[gaiksaya]

Hi @lukasmrtvy ,

Did you restart Kibana after adding that setting?

[lukasmrtvy]

@gaiksaya yes, of course. I am using helm chart v1.10.1.

[peterzhuamazon]

@vrozov @zengyan-amazon Seems like we have multiple community members with security plugin related issues. Could you guys help with this? Thanks.

[sushovan23]

Hi @gaiksaya: I have added setting and restarted kibana but still same error. please let me know what to do

[zengyan-amazon]

@sushovan23 can you share your kibana.yml configuration so that we can try to reproduce the issue on our side?

[sushovan23]

@zengyan-amazon here is the kibana.yml:

# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# or in the "license" file accompanying this file. This file is distributed
# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
# express or implied. See the License for the specific language governing
# permissions and limitations under the License.

# Description:
# Default Kibana configuration for Open Distro.

elasticsearch.hosts: https://localhost:9200
elasticsearch.ssl.verificationMode: none
elasticsearch.username: <user>
elasticsearch.password: <pwd>
elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization", "security_tenant"]

opendistro_security.multitenancy.enabled: true
opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]
opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://login.microsoftonline.com/<secret>/v2.0/.well-known/openid-configuration"
opendistro_security.openid.client_id: "<client_id>"
opendistro_security.openid.client_secret: "<secret>"
opendistro_security.openid.base_redirect_url: "https://fqdn/"
opendistro_security.cookie.secure: false

these are rpms installed on server:

rpm -qa | grep -i opendistr
opendistro-security-1.10.1.0-0.noarch
opendistro-knnlib-1.10.1.0-1.x86_64
opendistroforelasticsearch-kibana-1.10.1-1.x86_64
opendistro-anomaly-detection-1.10.1.0-1.noarch
opendistro-job-scheduler-1.10.1.0-1.noarch
opendistro-performance-analyzer-1.10.1.0-1.noarch
opendistro-alerting-1.10.1.2-1.noarch
opendistro-knn-1.10.1.0-1.noarch
opendistro-sql-1.10.1.1-1.noarch
opendistro-index-management-1.10.1.1-1.noarch
opendistroforelasticsearch-1.10.1-1.noarch

elasticsearch version elasticsearch-oss-7.9.1-1.x86_64

Let me know if anything else is needed

[zengyan-amazon]

I am suspecting this is due to our new Kibana security plugin OpenIDConnect implementation issue, which doesn't allow session_state as query parameter.

can you try updating the <kibana_installation_dir>/plugins/opendistro_security/server/auth/types/openid/routes.js, update line 43 - 48

                  query: config_schema_1.schema.object({
                      code: config_schema_1.schema.maybe(config_schema_1.schema.string()),
                      nextUrl: config_schema_1.schema.maybe(config_schema_1.schema.string()),
                      state: config_schema_1.schema.maybe(config_schema_1.schema.string()),
                      refresh: config_schema_1.schema.maybe(config_schema_1.schema.string()),
                  }),

to

query: config_schema_1.schema.any(),

and restart Kibana to see if it works?

[sushovan23]

That did it. However first time I logged in after reboot I saw following error:

Error: Internal Server Error
    at Fetch._callee3$ (https://fqdn/33912/bundles/core/core.entry.js:34:109213)
    at l (https://fqdn/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155323)
    at Generator._invoke (https://fqdn/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155076)
    at Generator.forEach.e.<computed> [as next] (https://fqdn/33912/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:368:155680)
    at fetch_asyncGeneratorStep (https://fqdn/33912/bundles/core/core.entry.js:34:102354)
    at _next (https://fqdn/33912/bundles/core/core.entry.js:34:102670)

Also noticed that Global tenant was not available for non admin users? Is that a bug & should be reported separately?

[sushovan23]

[sushovan23]

As you can see Global tenant is grayed out.

[zengyan-amazon]

The error message seems not related to the OpenIDConnect issue, we will also try to reproduce it on our side.

And I would suggest to open a separate issue for the global tenant issue. please create the issue in Kibana security plugin repo https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues

[sushovan23]

yes I only noticed it first time i logged in. Havent seen it since.

I have opened separate issue for global tenant grayed out: https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/509

[zengyan-amazon]

@chandu484 can you try clear your browser cookies, or try logging in using Incognito mode? if it still not working, please share your kibana.yml to help us diagnose. Thanks!

[sushovan23]

Will we get patch released or minor release with this fix as we want to use 1.10.x?

[zengyan-amazon]

@sushovan23 We are working on creating a 1.10.1.2 release of security kibana plugin, so that you can download and replace the plugin. Hope this works for you.

[sushovan23]

Great we are presently installing opendistroforelasticsearch-kibana rpm and I mention the version number 1.10.1 so I’m hoping all I need to do is to update version 1.10.1.2 for that rpm and elastic search related remains same at 1.10.1 or will that change. I’ll wait for release n install/upgrade documentation to figure it out. Thank you

[chandu484]

@chandu484 can you try clear your browser cookies, or try logging in using Incognito mode? if it still not working, please share your kibana.yml to help us diagnose. Thanks!

Hi @zengyan-amazon I have followed the steps you mentioned to modify routes.js in kibana node and after that I'm not seeing any issue with iss in kibana. Looks good. Thanks for that. I'm using amazon/opendistro kibana image 1.10.1. How can we update the patch for kibana plugin (1.10.1.2) you are taking about in the container as its using base image?

[lukasmrtvy]

Is there any private version of v1.10.1.2 kibana image to test ? Thanks

[arnitolog]

for some reason code substitution didn't help me. I run everything on docker, so I deleted the original plugin first, and then install the new one, which has the above changes. And I got an error in the browser: "ERR_TOO_MANY_REDIRECTS"

[zengyan-amazon]

@arnitolog can you clear your browser cookies or try in incognito mode?

[arnitolog]

tried in incognito and even in a different browser. But got the same result - the error above.

[zengyan-amazon]

@arnitolog can you share your configurations in kibana.yml and your ES security OIDC config? it will help us to understand your setup

[arnitolog]

sure. Here is kibana.yml:

`

server.name: kibana
server.host: "0"
## Replace with Elasticsearch DNS name picked during Service deployment
elasticsearch.hosts: https://es.local:9200

## Kibana TLS Config
# server.ssl.enabled: true
# server.ssl.key: /usr/share/kibana/certs/kibana-key.pem
# server.ssl.certificate: /usr/share/kibana/certs/kibana-crt.pem
elasticsearch.ssl.verificationMode: none
elasticsearch.ssl.certificate: /new/certificates/tls.crt
elasticsearch.ssl.key: /new/certificates/tls.key
elasticsearch.ssl.certificateAuthorities: /new/certificates/ca.crt
elasticsearch.username: kibanaserver
elasticsearch.password: xxxxxxxxxxxx
elasticsearch.requestTimeout: 150000

logging.dest: stdout
logging.json: true
logging.timezone: UTC
logging.verbose: false

elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
opendistro_security.allow_client_certificates: true
opendistro_security.multitenancy.enabled: false
opendistro_security.multitenancy.tenants.preferred: ["Global","Private"]
opendistro_security.readonly_mode.roles: ["kibana_read_only"]
opendistro_security.auth.type: "openid"
opendistro_security.openid.connect_url: "https://keycloak.company.com/auth/realms/master/.well-known/openid-configuration"
opendistro_security.openid.base_redirect_url: "https://logs.dev.company.com"
opendistro_security.openid.client_id: "kibana-dev"
opendistro_security.openid.client_secret: "zzzzzzzzzzzzzzzz"
opendistro_security.cookie.password: XXXXXXXXXXXXXXX
opendistro_security.cookie.ttl: 3600000
opendistro_security.session.ttl: 86400000
opendistro_security.session.keepalive: true
opendistro_security.cookie.secure: false`

Here is security config:

       dynamic:
        http:
          anonymous_auth_enabled: false
          xff:
            enabled: false
        authc:
          basic_internal_auth_domain:
            http_enabled: true
            order: 0
            http_authenticator:
              type: basic
              challenge: false
            authentication_backend:
              type: internal
          openid_auth_domain:
            http_enabled: true
            order: 1
            http_authenticator:
              type: openid
              challenge: true
              config:
                subject_key: preferred_username
                roles_key: roles
                openid_connect_url: https://keycloak.company.com/auth/realms/master/.well-known/openid-configuration
            authentication_backend:
              type: noop

[plnordquist]

I had the {"statusCode":400,"error":"Bad Request","message":"[request query.session_state]: definition for this key is missing"} error when I connected my Kibana to Keycloak and was able to work around it by configuring the client registration in Keycloak to turn on the Exclude Session State From Authentication Response option. Is there a point where I will need to reverse this configuration? Keycloak alludes to this option being useful for maintaining compatibility with older ODIC/OAuth 2 adapters.

[zengyan-amazon]

@arnitolog your configuration looks correct, I have tried to reproduced using similar config on my side, and it works ok.

Usually infinite redirection loop is caused by cookie not properly passed between server and browser. Can you check the requests in your browser developer tool and see if the set-cookie headers returned by kibana server are honored by the browser?

[zengyan-amazon]

@plnordquist we had a fix to allow the session_state parameter https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/pull/510 , we plan to release a patch version of security kibana plugin soon in next few days, then you should be able to turn the session management option back on

[zengyan-amazon]

Is there any private version of v1.10.1.2 kibana image to test ? Thanks

@lukasmrtvy you can download a snapshot version of v1.10.1.2 security kibana plugin at https://d3g5vo6xdbdb9a.cloudfront.net/downloads/kibana-plugins/opendistro-security/opendistro_security_kibana_plugin-1.10.1.2.zip

[lukasmrtvy]

@zengyan-amazon I am looking for a docker image ( I am using helm chart, its not so easy to mount & install plugin at deploy time .. )

I modified official image https://github.com/lukasmrtvy/opendistro-for-elasticsearch-kibana-fix/blob/main/Dockerfile#L3

Helm args: --set kibana.image=lukasmrtvy/opendistro-for-elasticsearch-kibana-fix & --set kibana.imageTag=latest

Will try asap..

EDIT: Works!

[arnitolog]

@zengyan-amazon I checked set-cookie header and found an error: Set-Cookie header is ignored in response from url: https://logs-test.us.dev.kube.company.com/auth/openid/login?state=abKJ_B2F8UGxIw7HWbhWIp&code=a950100f-19d1-4bb6-8b50-0557362be14b.1e86c530-392c-4bd2-81fa-ca31a25db5f2.208d33ca-1735-423c-8075-8860fa0a8c34. Cookie length should be less than or equal to 4096 characters. Are there any changed introduced in ODFE 1.10.1 how cookies are formed. Exactly the same config is working fine with ODFE 1.9.0

[zengyan-amazon]

@arnitolog yes, we do have changes regarding cookie in 1.10.1.

In 1.9.0 and older versions, we only save id_token in cookie, then you have to refresh the page when the id_token expires.

In 1.10.1 openID connect auth, we save both id_token and refresh_token in cookie, to support token refresh, thus our plugin can take care of the token renewal without requiring user to refresh the page manually.

this might be the reason of increased cookie size. So if you have a lot of attributes from your IDP, then your id_token might be already pretty big and adding the refresh token might make it exceeded the 4096 byte limit.

Can you create a new issue in Kibana security plugin repo ? We can track this cookie size problem separately using that issue.

[arnitolog]

thank you @zengyan-amazon. I've created https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/issues/516

[zengyan-amazon]

we have released v1.10.1.2 of Kibana security plugin, which fixed a few issues including this OpenID parameter problem. You can download the plugin zip bundle from the release and install it using kibana-plugin install command. We are not going to release a new ODFE bundle or image for this patch version.

[sushovan23]

So this would be special handling for 1.10.1 and in later version it would be included in rpm?

Also what are the steps to install it? Copy over zip , unzip it and install using kibana-plugin install command? We have our provisioning/upgrade via ansible playbooks so asking as it would require changing our playbooks to handle this version.

[zengyan-amazon]

@sushovan23 yes, this is a patch release of Kibana security plugin for 1.10.1. We will definitely include the fixes in future ODFE versions in their rpm/docker image etc. According to ODFE roadmap, the next ODFE version should be 1.11.0.

you can use the kibana-plugin command to install the plugin. e.g.

<kibana_dir>/bin/kibana-plugin install https://github.com/opendistro-for-elasticsearch/security-kibana-plugin/releases/download/v1.10.1.2/opendistro_security_kibana_plugin-1.10.1.2.zip

if you have the 1.10.1.1 version installed, you may want to uninstall it first by executing

<kibana_dir>/bin/kibana-plugin remove opendistro_security

[darkdragn]

@zengyan-amazon AWESOME! It worked perfectly. The biggest thing was that I'm using the helm chart and you guys don't have a docker image on it right now, so I just rebuild the 1.10.1 from your repo. if anyone wants to use the fixed kibana image before they get one up, or if you want to retag mine and push it: darkdragn/opendistro-for-elasticsearch-kibana:1.10.1.2

Thanks again @zengyan-amazon , you're awesome, man.

[zengyan-amazon]

@darkdragn Thank you! As you can see from our roadmap we are currently working on 1.11 release which is still on Elasticsearch 7.9. It includes several new features or feature enhancements. This fix will definitely be included in the 1.11 release with the image.

[sushant-pradhan-tm]

@zengyan-amazon AWESOME! It worked perfectly. The biggest thing was that I'm using the helm chart and you guys don't have a docker image on it right now, so I just rebuild the 1.10.1 from your repo. if anyone wants to use the fixed kibana image before they get one up, or if you want to retag mine and push it: darkdragn/opendistro-for-elasticsearch-kibana:1.10.1.2

Thanks again @zengyan-amazon , you're awesome, man.

Hey @darkdragn I have the same setup as yours and faced similar error as in this thread. I took the image published by you: darkdragn/opendistro-for-elasticsearch-kibana:1.10.1.2 and changed kibana.yml to opendistro_security.cookie.secure: false. However, I am still getting error with Kibana login. Can you share your configurations. I guess I am missing some additional change

[darkdragn]

@zengyan-amazon AWESOME! It worked perfectly. The biggest thing was that I'm using the helm chart and you guys don't have a docker image on it right now, so I just rebuild the 1.10.1 from your repo. if anyone wants to use the fixed kibana image before they get one up, or if you want to retag mine and push it: darkdragn/opendistro-for-elasticsearch-kibana:1.10.1.2 Thanks again @zengyan-amazon , you're awesome, man.

Hey @darkdragn I have the same setup as yours and faced similar error as in this thread. I took the image published by you: darkdragn/opendistro-for-elasticsearch-kibana:1.10.1.2 and changed kibana.yml to opendistro_security.cookie.secure: false. However, I am still getting error with Kibana login. Can you share your configurations. I guess I am missing some additional change

Here you go, man. I removed my keycloak stuff and my domain names, but other than that everything is the same. Also, I'm using a manual secret for config.yaml to be the security config. I found that the "easier" method in there does a directory mount which drops everything and I didn't want to recreate all of the base configs in my values.yaml. It also has some of the keys(Filenames) wrong in the base values.yaml that gave me a migraine, so using the single secret way was easier for me. (Ref https://gist.github.com/darkdragn/e19982150306ebd4368c6fdb890a7303#file-values-yaml-L166)

https://gist.github.com/darkdragn/e19982150306ebd4368c6fdb890a7303

[sushant-pradhan-tm]

Thank you @darkdragn I am now getting a different error: redirected you too many times. (ERR_TOO_MANY_REDIRECTS). I suspect similar issue to the one faced by @arnitolog . @zengyan-amazon is it because #516 is still open and there is no patch yet? Or may be its not part of arkdragn/opendistro-for-elasticsearch-kibana:1.10.1.2?

[sushovan23]

@zengyan-amazon Did this make it to release 1.11.0 which was released yesterday?

[sushant-pradhan-tm]

@zengyan-amazon Did this make it to release 1.11.0 which was released yesterday?

It doesn't look like. I still get TOO_MANY_ATTEMPTS error after the upgrade. @zengyan-amazon can confirm the latest on this though. Kind of blocked and doesn't seem to have a working solution/workaround

[zengyan-amazon]

@sushant-pradhan if you get ERR_TOO_MANY_REDIRECTS , it looks more like the same issue as #516 , can you check the cookie size and your JWT token size?

We don't have a patch for #516 yet, our team is still discussing how to solve the root cause of this issue.

[sushant-pradhan-tm]

@zengyan-amazon yes after the upgrade, the cookie shows invalid (i assume because of it exceeds allowed length). I can't downgrade to a lower version too. Kind of a limbo state.

[ch3lo]

@zengyan-amazon yes after the upgrade, the cookie shows invalid (i assume because of it exceeds allowed length). I can't downgrade to a lower version too. Kind of a limbo state.

I had the same problem using opendistro with Keycloak SSO. As a workaround I removed some flields mapped to JWT (claim). For example, remove displayName, user_prefered, phone, etc. I think you don't need those fields to assing roles and permissions.

Regards