search

grafana / opensearch-datasource

Apache License 2.0
27 stars 21 forks source link

Unable to use Alerting for OpenSearch in AMG #167

Closed ericayosh closed 1 year ago

ericayosh commented 1 year ago

What happened: When using the open search plugin in AWS Managed Grafana credentials are provided by specifying the Authorization Provider on the data source configuration as Workspace IAM Role.

Customers are able to successfully connect to and query the open search datasource.

However when attempting to create alerts, they are getting an error:

request hanler error: failed to query data: Failed to query data: rpc error: code = Unknown desc = post \"https://vpc-....": EC2RoleRequestError: no EC2 instance role found 

caused by: RequestError: send request failed

This error is only occurring when customers are trying to create alerts.

What you expected to happen: The open search plugin should use the same workspace iam role code path for getting credentials when evaluating alerts and querying the data source.

How to reproduce it (as minimally and precisely as possible):

  1. Create an open search cluster
  2. Create an AWS Managed Grafana workspace (use grafana version 8.4.7)
  3. Add the open search cluster as a data source in the AWS Managed Grafana workspace. Set the Authentication Provider to Workspace IAM Role
  4. Query the data source (should succeed)
  5. Attempt to create an alert from a dashboard panel. Save & Test should show the error mentioned above.

Anything else we need to know?:

Environment: AWS Managed Grafana workspace

sarahzinger commented 1 year ago

I believe unlike Grafana's other aws datasources, the opensearch datasource does not use https://github.com/grafana/grafana-aws-sdk for backend queries. I'm not sure if this was done intentionally, or as an oversight whenever it was written, but it's something we plan to revisit.

I think for us to support using Workspace IAM Roles in backend queries (such as those used in alerting) we need to use the aws-sdk ideally reusing the grafana-aws-sdk.

Unfortunately I don't have a recommended quick fix or workaround just yet but we'll be sure to prioritize this refactor.

fridgepoet commented 1 year ago

Hi @ericayosh would you be able to test this on the latest version of the plugin? v2.6.1

grglnd commented 1 year ago

I was seeing similar issues with the alerts and can verify the fix solved our issues. Thank you!!

sarahzinger commented 1 year ago

fixed by https://github.com/grafana/opensearch-datasource/pull/183