SSO-Auth users using oauth2 proxy are not able to start graylog input or view node information

Problem description

I'm having an issue with getting permissions to work properly with users authenticating via the SSO-Auth plugin. I have 3 Graylog nodes running behind an Oauth2 proxy and a Loadbalancer. My setup is like

user -->> LB --> Nginx -->> Oauth2_proxy —>> Graylog Cluster.

I get a lot of warning shown in the logs repeatedly and I cannot start Graylog input

Oct 07 20:10:00 node01.graylog. docker[2616115]: 2020-10-07 18:10:00,421 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://private-ip-of-the-vm:9000/api/system on node , result: Unauthorized Oct 07 20:10:02 node01.graylog. docker[2616115]: 2020-10-07 18:10:02,151 WARN : org.graylog2.shared.rest.resources.ProxiedResource - Unable to call http://private-ip-of-the-vm:9000/api/system/metrics/multiple on node , result: Unauthorized

I also get the following errors when accessing nodes. The system information is also not available in the logs:

Could not get JVM information Getting JVM information for node ‘d686b638-3175-4d61-b529-6d4e0025bca1’ failed: Error: cannot GET https://Public-URL/api/cluster/d686b638-3175-4d61-b529-6d4e0025bca1/jvm (502)

I'm running graylog 3.3.8 version and here is the graylog configuration: GRAYLOG_IS_MASTER=true GRAYLOG_HTTP_PUBLISH_URI=http://privateip-of-graylog:9000/ GRAYLOG_HTTP_EXTERNAL_URI=https://public-url-pointing-LB/ GRAYLOG_ROOT_PASSWORD_SHA2=************ GRAYLOG_PASSWORD_SECRET=************** GRAYLOG_ELASTICSEARCH_HOSTS=http://private-ip-of-the-vm:9200/ GRAYLOG_MONGODB_URI=mongodb://********************27017/graylog?replicaSet=graylog-rs0 GRAYLOG_WEB_ENABLE=true GRAYLOG_OUTPUT_BATCH_SIZE=5000 GRAYLOG_TRUSTED_PROXIES=\"127.0.0.1/32, 172.17.0.0/16, 10.0.0.0/16\" \ GRAYLOG_SERVER_JAVA_OPTS=\"-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Xms6g -Xmx6g\""
oauth2-proxy configuration:

Environment="OAUTH2_PROXY_OPTS=--email-domain=igniti.de \ --http-address=0.0.0.0:4180 \ --upstream=http://privateip-of-graylog:9000 \ --provider=azure \ --cookie-secret="*" \ --client-id="" \ --client-secret="****" \ --redirect-url="https://public-url-pointing-LB/oauth2/callback" \ --pass-user-headers=true \ --set-xauthrequest=true \ --pass-authorization-header=true \ --pass-basic-auth=false \ --cookie-secure=true \ --cookie-domain=public-url-pointing-LB \ --whitelist-domain=public-url-pointing-LB \ --prefer-email-to-user=true \ --cookie-expire=0h30m0s"

nginx configiration:

location / { proxy_buffers 4 256k; proxy_buffer_size 128k; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; proxy_connect_timeout 1; proxy_send_timeout 30; proxy_read_timeout 30; proxy_pass http://privateip-of-graylog:4180; }

Steps to reproduce the problem

Logs in oauth2-proxy when login is successful:

Nov 04 16:29:30 node01.graylog docker[2029130]: 10.0.0.7:48638 - jerald@example.com [2020/11/04 15:29:30] [AuthSuccess] Authenticated via OAuth2: Session{email:jerald@example.com user: PreferredUsername: token:true id_token:true created:2020-11-04 15:29:30.228503419 +0000 UTC m=+4782.819723682 expires:2020-11-04 16:29:29 +0000 UTC refresh_token:true} Nov 04 16:29:30 node01.graylog docker[2029130]: [2020/11/04 15:29:30] [logger.go:508] WARNING: Multiple cookies are required for this session as it exceeds the 4kb cookie limit. Please use server side session storage (eg. Redis) instead. Nov 04 16:29:30 node01.graylog docker[2029130]: 10.0.0.7:48638 - - [2020/11/04 15:29:29] graylog-public-web-url GET - "/oauth2/callback?code=0.AAAA5q5NjEGcNUmsC8ekoVFyIABTnS82JYNOqOmFRC43HUEAAAA.AQABAAIAAAB2UyzwtQEKR7-rWbgdcBZIOOtoPtoTqs2l_LThn-G4j4o1gDGdZgkxHcuDAmSFnap9w8lte3mw3vp32ACnW2YUWoPob-cxoqc4kP65Bq9AZ6TWCHqcEmmOXDYiueftMlvortv1TJMJq4E0JqdHXkE7n37WdEzCn2XVsnoE9T4zPII_BXrKU_PlQFhq5p1KRdu5HjkFy_Z6L8pXw9wQjxlxy-q8j6Z1RbGcdMl3L_1nh8WJBHIzFQ81tJHQDQG9XFTjbNR-PVvcrnMP6xXO_beyi3RsyuNx2eEnEhK3oQ-BcBzv7SAURZyd3g_djl9G1-qFhm9GxMZ1bf2vljH-OLYaLWq3u_EsgB0RzhByvj2iURUlofByuCfuMWZlov_kQNUO2eYHKnXqLGaMq6JXvazZ6msjZpj6atxN5nWqjAfmo_ynZ-Oq4nfZLZEaImxwLHf3qxlR2HWEugNhcSYrPCvQgTRCFt0ECLzS8BP1FDSUpfigZa5ikJ2_VYvy83-7cYH5sGZgC0M55s46TrqJ5-T6gNLC35BnATujxp6yDt3b4_7sOI8JF_4lWTuPwJd1aD7Ssx6JX1V2xln0mljEo_8cZcKAtudcpq1jC0xfyixZnCAA&state=928411ccd15194ea7e4a131302a0f9fe%3a%2f&session_state=221c4564-428d-4f2d-b1d4-73b627768599" HTTP/1.0 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36" 302 24 0.433

Go to System > inputs, and try to start the input but it fails:
Go to system nodes and view node.

From the above screenshot, it can be seen that the node cannot be viewed and there are errors.

Environment

Graylog Version: 3.3.8
Plugin Version: graylog-plugin-auth-sso:3.3.0
Elasticsearch Version: 6.8.12
MongoDB Version: 4.0.20
Operating System: uibuntu 20.x
Browser version: Google Chrome 86.0.x

Additional information Single Sign-On Configuration Username Header: X-Forwarded-User

A similar problem seems to be reported in the community below, but , I didn't see a bug report or possible solution for this issue. https://community.graylog.org/t/sso-auth-users-with-admin-role-are-unauthorized-for-some-calls/15615

PS: Everything works, without using oauth2-proxy or just disabling single sign-on (this will cause double authentication)

Any suggestions are highly appreciated.

Graylog2 / graylog-plugin-auth-sso