store-gateway fatal error, signal SIGSEGV: segmentation violation

agardiman commented 1 year ago

Describe the bug

Some store-gateway recently crashed because of a segmentation fault.

To Reproduce

Few instances crashed but they stopped. I don't know what event made them fail.

Expected behavior

To not crash for segmentation fault.

Environment

Infrastructure: Kubernetes
Deployment tool: jsonnet
Mimir version: 2.2.0

Additional Context

store-gateway replicas: 110
in memory active series 720M (mostly one tenant)

store-gateway pod

➜  ~ kubectl describe pod store-gateway-47
Name:             store-gateway-47
Namespace:        cortex
Priority:         0
Service Account:  cortex
Node:            REDACTED
Start Time:       Wed, 23 Nov 2022 15:21:51 +0000
Labels:           controller-revision-hash=store-gateway-6f66f64dc
                  gossip_ring_member=true
                  name=store-gateway
                  statefulset.kubernetes.io/pod-name=store-gateway-47
Annotations:      kubernetes.io/psp: eks.privileged
Status:           Running
IP:               REDACTED
IPs:
  IP:           REDACTED
Controlled By:  StatefulSet/store-gateway
Containers:
  store-gateway:
    Container ID:  docker://d9153744b298810375c29be084a9652730c8305e5f10fea991c371322b2f06be
    Image:         REDACTED/mimir:2.2.0
    Image ID:      REDACTED
    Ports:         80/TCP, 9095/TCP, 7946/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      -auth.multitenancy-enabled=true
      -blocks-storage.backend=s3
      -blocks-storage.bucket-store.chunks-cache.backend=memcached
      -blocks-storage.bucket-store.chunks-cache.memcached.addresses=dnssrvnoa+memcached.cortex.svc.cluster.local:11211
      -blocks-storage.bucket-store.chunks-cache.memcached.max-async-concurrency=50
      -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-batch-size=5000
      -blocks-storage.bucket-store.chunks-cache.memcached.max-get-multi-concurrency=100
      -blocks-storage.bucket-store.chunks-cache.memcached.max-idle-connections=100
      -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size=1048576
      -blocks-storage.bucket-store.chunks-cache.memcached.timeout=4s
      -blocks-storage.bucket-store.ignore-blocks-within=7h
      -blocks-storage.bucket-store.index-cache.backend=memcached
      -blocks-storage.bucket-store.index-cache.memcached.addresses=dnssrvnoa+memcached-index-queries.cortex.svc.cluster.local:11211
      -blocks-storage.bucket-store.index-cache.memcached.max-async-concurrency=50
      -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-batch-size=5000
      -blocks-storage.bucket-store.index-cache.memcached.max-get-multi-concurrency=100
      -blocks-storage.bucket-store.index-cache.memcached.max-idle-connections=100
      -blocks-storage.bucket-store.index-cache.memcached.max-item-size=5242880
      -blocks-storage.bucket-store.index-cache.memcached.timeout=4s
      -blocks-storage.bucket-store.index-header-lazy-loading-enabled=true
      -blocks-storage.bucket-store.index-header-lazy-loading-idle-timeout=60m
      -blocks-storage.bucket-store.max-chunk-pool-bytes=59055800320
      -blocks-storage.bucket-store.metadata-cache.backend=memcached
      -blocks-storage.bucket-store.metadata-cache.memcached.addresses=dnssrvnoa+memcached-metadata.cortex.svc.cluster.local:11211
      -blocks-storage.bucket-store.metadata-cache.memcached.max-async-concurrency=50
      -blocks-storage.bucket-store.metadata-cache.memcached.max-get-multi-concurrency=100
      -blocks-storage.bucket-store.metadata-cache.memcached.max-idle-connections=100
      -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size=1048576
      -blocks-storage.bucket-store.metadata-cache.memcached.timeout=4s
      -blocks-storage.bucket-store.sync-dir=/data/tsdb
      -blocks-storage.bucket-store.sync-interval=15m
      -blocks-storage.s3.bucket-name=REDACTED
      -blocks-storage.s3.endpoint=s3.us-west-2.amazonaws.com
      -memberlist.bind-port=7946
      -memberlist.join=gossip-ring.cortex.svc.cluster.local:7946
      -runtime-config.file=/etc/mimir/overrides.yaml
      -server.grpc.keepalive.min-time-between-pings=10s
      -server.grpc.keepalive.ping-without-stream-allowed=true
      -server.http-listen-port=80
      -server.http-read-timeout=5m
      -server.http-write-timeout=5m
      -store-gateway.sharding-ring.prefix=
      -store-gateway.sharding-ring.replication-factor=3
      -store-gateway.sharding-ring.store=memberlist
      -store-gateway.sharding-ring.tokens-file-path=/data/tokens
      -store-gateway.sharding-ring.wait-stability-min-duration=1m
      -target=store-gateway
      -tenant-federation.enabled=true
    State:          Running
      Started:      Wed, 23 Nov 2022 15:32:32 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 23 Nov 2022 15:25:44 +0000
      Finished:     Wed, 23 Nov 2022 15:32:20 +0000
    Ready:          True
    Restart Count:  2
    Limits:
      memory:  55Gi
    Requests:
      cpu:      5
      memory:   40Gi
    Readiness:  http-get http://:80/ready delay=15s timeout=5s period=10s #success=1 #failure=3
    Environment:
      AWS_DEFAULT_REGION:           us-west-2
      AWS_REGION:                   us-west-2
      AWS_ROLE_ARN:                 REDACTED
      AWS_WEB_IDENTITY_TOKEN_FILE:  /var/run/secrets/eks.amazonaws.com/serviceaccount/token
    Mounts:
      /data from store-gateway-data (rw)
      /etc/mimir from overrides (rw)
      /var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-226mx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  aws-iam-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  86400
  store-gateway-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  store-gateway-data-store-gateway-47
    ReadOnly:   false
  overrides:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      overrides
    Optional:  false
  kube-api-access-226mx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

Config (only values that differ from default)

blocks_storage:
    backend: s3
    bucket_store:
        chunks_cache:
            backend: memcached
            memcached:
                addresses: dnssrvnoa+memcached.cortex.svc.cluster.local:11211
                max_get_multi_batch_size: 5000
                timeout: 4s
        ignore_blocks_within: 7h0m0s
        index_cache:
            backend: memcached
            memcached:
                addresses: dnssrvnoa+memcached-index-queries.cortex.svc.cluster.local:11211
                max_get_multi_batch_size: 5000
                max_item_size: 5242880
                timeout: 4s
        max_chunk_pool_bytes: 59055800320
        metadata_cache:
            backend: memcached
            memcached:
                addresses: dnssrvnoa+memcached-metadata.cortex.svc.cluster.local:11211
                timeout: 4s
        sync_dir: /data/tsdb
    s3:
        bucket_name: REDACTED
        endpoint: s3.us-west-2.amazonaws.com
memberlist:
    join_members:
        - gossip-ring.cortex.svc.cluster.local:7946
runtime_config:
    file: /etc/mimir/overrides.yaml
server:
    http_listen_port: 80
    http_server_read_timeout: 5m0s
    http_server_write_timeout: 5m0s
store_gateway:
    sharding_ring:
        kvstore:
            prefix: ""
        tokens_file_path: /data/tokens
        wait_stability_min_duration: 1m0s
target: store-gateway
tenant_federation:
    enabled: true

Runtime config (diff from default)

ingester_limits:
    max_inflight_push_requests: 50000
    max_ingestion_rate: 0
    max_series: 0
    max_tenants: 0
overrides:
    wd_user:
        compactor_split_and_merge_shards: 28
        compactor_split_groups: 21
        ingestion_burst_size: 90000000
        ingestion_rate: 4e+07
        max_global_series_per_metric: 160000000
        max_global_series_per_user: 1600000000
        ruler_max_rule_groups_per_tenant: 500

store-gateway-47.log