Forward OAuth Identity does not work for queries

[anemyte]

What happened: This plugin does not include Authorization header when it makes queries to an opensearch server despite Forward OAuth Identity being toggled on. This is indicated by this error:

Post "https://opensearch.example.com:9200/_msearch?max_concurrent_shard_requests=5": downstream error: 401 Unauthorized

What you expected to happen: I expect the plugin to actually Forward OAuth Identity as it was configured to do. I suppose it is done by supplementing query requests with current user's access_token via Authorization HTTP header.

How to reproduce it (as minimally and precisely as possible):

1. Setup an Opensearch server
2. Login to your Grafana instance using an OpenID-Connect identity provider
3. Configure this plugin to use the opensearch server from p1 and toggle Forward OAuth Identity on
4. Use tcpdump to capture Grafana requests
5. Try executing a query via explore or a dashboard panel.
6. Use Wireshark or an alternative to analyze packet dump from p.4 and notice missing Authorization header in query requests.

Anything else we need to know?: The request to get Opensearch version from plugin configuration page does actually come with the user's identity. Here is how this request looks in the packet capture:

And this is how a query request looks like:

As you can see, the second request comes without the header and thus it gets the 401 error from opensearch.

Another thing that I want to point out is that the Elasticsearch datasource plugin configured to use the same opensearch server in the same environment does actually work, I can see query results and no error. Here's how the Elasticsearch plugin performs the request:

Environment:

• Grafana version: v11.3.0 (d9455ff7db)
• OpenSearch version: 2.16.0
• Plugin version: 2.20.0, 2.21.0 (tested both)
• Identity Provider: Keycloak v25.0.2

[idastambuk]

Hi @anemyte can you try updating your plugin to the latest version, 2.21.1 and see if that changes if the headers are being passed in get_version and queryData? Thanks!

[anemyte]

@idastambuk Well, I don't see any changes with 2.21.1. "get_version" still passes the header and works, while querying still doesn't. Here's the query request once again:

[anemyte]

@idastambuk Do you need any help in setting up the environment to work on this? I understand it can be a little too hard to launch and configure all the services. I could write a docker-compose file for you to spin up the environment locally (if you have docker installed, of course).

[iwysiu]

Hi @anemyte , I'm currently taking a look at this, but to help me figure out what path to investigate, do you have the feature flag openSearchBackendFlowEnabled set to true? If you're on hosted grafana it should be on by default. (For context we've been migrating Opensearch to backend queries, and are planning on deprecating the frontend path). If you're not sure, if you send me a screenshot of network information for the frontend request and I can figure it out.

[iwysiu]

Also, a docker file to spin up the test environment would be very helpful, thanks!