Pagination problem for HA grafana deployment

[dmazhar-cogniance]

Hi all. We have a grafana deployment with 2 instances and PostgreSQL database for the storage. Grafana session is stored in the DB too. Trying to get paginated response AWS Timestream results in error: ValidationException: Invalid pagination token. Reducing the number of grafana instances to 1 or enabling the sticky sessions on the load balancer resolves the problem, so I think that problem is that pagination requests are hitting different grafana instances. Instances will use different source IP to reach AWS API, if this is important. Please let me know if I should provide more details, I can reproduce the problem. Grafana version: 7.3.7 Timestream datasource plugin version: 1.1.2 Setup: running official dockerhub grafana containers in AWS ECS, 2 instances behind the application load balancer. PostgreSQL's used as a grafana DB.

[dmazhar-cogniance]

It looks like the root cause is not different source IPs, but different IAM credentials. IAM Roles for Tasks are used in our deployment, two instances of grafana use same IAM role, but different IAM tokens obtained from STS, and this is where the things broke :(

[ryantxu]

For HA deployments like this, you will need to make sure that everything in a cluster is configured the same. Let us know if there is something more concrete we should dig into

[dmazhar-cogniance]

@ryantxu , the issue here is that this is the set-up that you'll use with AWS usually. Example for the ECS:

• you create an IAM role with permissions to access the AWS timestream
• you create an ECS service with grafana docker container and use the IAM role created previously
• grafana starts and AWS SDK for Go will automatically use IAM role provided. To use the role SDK makes a request to AWS STS to acquire temporary credentials.
• 2 instances of grafana will use the same IAM role, but AWS SDK for Go will get different temporary credentials from IAM.
• During the pagination user requests will hit different grafana instances that will try to do pagination via AWS API using different temporary IAM credentials. The flow fails here

I see two options to workaround this:

1. Use IAM access/secret keys instead of IAM role, which is not a good practice
2. Use sticky sessions on ALB, so all the pagination requests from user will hit the same grafana instance. We're using this as a workaround right now, but sticky sessions are smelly...

As for me the issue here is that AWS pagination token is used across different instances of grafana.