AWS Elasticache redis: Attempt to modify a readonly table

[SiahGD]

Version

Using chart 14.1.0 AWS EKS (1.21)

AWS Elasticache redis (6.2.6), non clustered, 2 replicas (3 nodes). Using main endpoint.

Steps to Reproduce

Deploy sentry with terraform (helm_release) Sentry worker crashing with

│ 10:55:40 [ERROR] sentry.digests: Failed to perform scheduling for partition 0 due to error: Error running script (call to f_094f8735408af273da283389ad983f63977831e8): @user_script:11: user_script:11: Attempt to modi │
│ Traceback (most recent call last):                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/sentry/utils/safe.py", line 28, in safe_execute                                                                                                                          │
│     result = func(*args, **kwargs)                                                                                                                                                                                      │
│   File "/usr/local/lib/python3.8/site-packages/sentry/similarity/__init__.py", line 130, in inner                                                                                                                       │
│     v1_method(*args, **kwargs)                                                                                                                                                                                          │
│   File "/usr/local/lib/python3.8/site-packages/sentry/similarity/features.py", line 142, in record                                                                                                                      │
│     return self.index.record(scope, key, items, timestamp=int(to_timestamp(event.datetime)))                                                                                                                            │
│   File "/usr/local/lib/python3.8/site-packages/sentry/similarity/backends/metrics.py", line 23, in record                                                                                                               │
│     return self.__instrumented_method_call("record", *args, **kwargs)                                                                                                                                                   │
│   File "/usr/local/lib/python3.8/site-packages/sentry/similarity/backends/metrics.py", line 20, in __instrumented_method_call                                                                                           │
│     return getattr(self.backend, method)(scope, *args, **kwargs)                                                                                                                                                        │
│   File "/usr/local/lib/python3.8/site-packages/sentry/similarity/backends/redis.py", line 144, in record                                                                                                                │
│     return self.__index(scope, arguments)                                                                                                                                                                               │
│   File "/usr/local/lib/python3.8/site-packages/sentry/similarity/backends/redis.py", line 49, in __index                                                                                                                │
│     return index(self.cluster, [scope], args)                                                                                                                                                                           │
│   File "/usr/local/lib/python3.8/site-packages/sentry/utils/redis.py", line 390, in call_script                                                                                                                         │
│     return script[0](keys, args, client)                                                                                                                                                                                │
│   File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 4021, in __call__                                                                                                                                 │
│     return client.evalsha(self.sha, len(keys), *args)                                                                                                                                                                   │
│   File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 3091, in evalsha                                                                                                                                  │
│     return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)                                                                                                                                                │
│   File "/usr/local/lib/python3.8/site-packages/sentry_sdk/integrations/redis.py", line 104, in sentry_patched_execute_command                                                                                           │
│     return old_execute_command(self, name, *args, **kwargs)                                                                                                                                                             │
│   File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 878, in execute_command                                                                                                                           │
│     return self.parse_response(conn, command_name, **options)                                                                                                                                                           │
│   File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 892, in parse_response                                                                                                                            │
│     response = connection.read_response()                                                                                                                                                                               │
│   File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 752, in read_response                                                                                                                         │
│     raise response                                                                                                                                                                                                      │
│ redis.exceptions.ResponseError: Error running script (call to f_4716e0f4a993856809343ed6798054c556dccdbf): @user_script:54: user_script:54: Attempt to modify a readonly table                                          │
│ 10:57:42 [ERROR] sentry.safe.function: record.process_error

values

``` prefix: user: create: true email: ${sentry_email} password: ${sentry_password} nginx: enabled: false rabbitmq: enabled: true redis: enabled: true master: disableCommands: [] sentry: web: service: annotations: alb.ingress.kubernetes.io/healthcheck-path: /_health/ alb.ingress.kubernetes.io/healthcheck-port: traffic-port relay: service: annotations: alb.ingress.kubernetes.io/healthcheck-path: /api/relay/healthcheck/ready/ alb.ingress.kubernetes.io/healthcheck-port: traffic-port postgresql: enabled: false externalPostgresql: host: ${postgres_db_host} password: ${postgres_password} username: ${postgres_username} database: ${postgres_db_name} ingress: enabled: true hostname: ${sentry_dns_name} regexPathStyle: aws-alb annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip alb.ingress.kubernetes.io/tags: ${tags} alb.ingress.kubernetes.io/inbound-cidrs: ${allowed_cidr_blocks_str} alb.ingress.kubernetes.io/subnets: ${public_subnet_ids_str} alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS": 443}]' alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/certificate-arn: ${subdomain_cert_arn} external-dns.alpha.kubernetes.io/hostname: ${sentry_dns_name} filestore: backend: s3 s3: accessKey: ${s3_access_key} secretKey: ${s3_secret_key} bucketName: ${s3_bucket_name} endpointUrl: ${s3_endpoint_url} region_name: ${region} clickhouse: enabled: true clickhouse: imageVersion: "20.8.19.4" configmap: remote_servers: internal_replication: true replica: backup: enabled: true mail: # For example: smtp useTls: true username: ${ses_user} password: ${ses_password} port: 587 host: "email-smtp.us-east-1.amazonaws.com" from: "${ses_from}" backend: "django.core.mail.backends.smtp.EmailBackend" ```

Expected Result

workers should process events

Actual Result

see log above

[SiahGD]

p.s.: same with built in redis on chart version 15.0.0

[enterdv]

Same for me

[enterdv]

On chart version 14.1.0 same error

[mazzz1y]

Looks like pinning redis version to 6.2.4 fixes this issue:

kubectl set image sts/sentry-sentry-redis-master redis=docker.io/bitnami/redis:6.2.4

[enterdv]

Looks like pinning redis version to 6.2.4 fixes this issue:

kubectl set image sts/sentry-sentry-redis-master redis=docker.io/bitnami/redis:6.2.4

6.2.4 version works for me, thank you

[SiahGD]

As for AWS Elasticache - it working with 5.0.6

[bumarcell]

6.2.4 version works for me in chart v15 as well

[mrsrvman]

https://github.com/getsentry/sentry/pull/34416

[bumarcell]

@mrsrvman Thanx for tha update! How does this affect the helm chart?

[mrsrvman]

need to upgrade sentry to version 22.7.0. https://github.com/sentry-kubernetes/charts/blob/develop/sentry/Chart.yaml#L6 appVersion: 22.7.0

[SiahGD]

appVersion: 22.7.0 helped, or downgrade redis to 5.0.6

[bumarcell]

@SiahGD This breaks snuba-migrate job for me because some GENERIC_METRICS entry isn't in some clickhouse table definition