panic: runtime error: invalid memory address or nil pointer dereference

[MalteMagnussen]

Running it in Kubernetes, using an on-prem S3 as storage.

Went fine, until I restarted the statefulset, and it starting crashing with this error:

mhm@mhm:~/Documents/gokapi$ k logs pods/wip1-gokapi-deployment-0 init-litestream 
{"time":"2024-06-18T12:01:12.434760932Z","level":"INFO","msg":"restoring snapshot","db":"/var/lib/gokapi/gokapi.sqlite","replica":"s3","generation":"4aed9fd1507e58f2","index":0,"path":"/var/lib/gokapi/gokapi.sqlite.tmp"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xea1898]

goroutine 1 [running]:
github.com/benbjohnson/litestream/s3.(*ReplicaClient).SnapshotReader(0xc00098aa80, {0x17b4468, 0x20f37a0}, {0xc0000463aa, 0x10}, 0xc0000463aa?)
    /src/litestream/s3/replica_client.go:295 +0x378
github.com/benbjohnson/litestream.(*Replica).restoreSnapshot(0xc00033e0f0, {0x17b4468, 0x20f37a0}, {0xc0000463aa, 0x10}, 0x12?, {0xc000046ff0, 0x21})
    /src/litestream/replica.go:1333 +0x179
github.com/benbjohnson/litestream.(*Replica).Restore(0xc00033e0f0, {0x17b4468, 0x20f37a0}, {{0x7ffc7b455c57, 0x1d}, {0x0, 0x0}, {0xc0000463aa, 0x10}, 0x7fffffff, ...})
    /src/litestream/replica.go:1102 +0x905
main.(*RestoreCommand).Run(0x20f37a0, {0x17b4468, 0x20f37a0}, {0xc00013c020, 0x3, 0x3})
    /src/litestream/cmd/litestream/restore.go:83 +0x78a
main.(*Main).Run(0xc0000061a0?, {0x17b4468, 0x20f37a0}, {0xc00013c010, 0x4, 0x4})
    /src/litestream/cmd/litestream/main.go:121 +0x13f
main.main()
    /src/litestream/cmd/litestream/main.go:41 +0x68

Here is the relevant yaml.

      initContainers:
        - name: init-litestream  # https://litestream.io/guides/kubernetes/#automatic-recovery
          image: litestream/litestream:0.3.13
          args: ['restore', '-if-db-not-exists', '-if-replica-exists', '/var/lib/gokapi/gokapi.sqlite']
          volumeMounts:
          - name: data-volume
            mountPath: /var/lib/gokapi
          - name: litestream-config
            mountPath: /etc/litestream.yml
            subPath: litestream.yml
          env:
          - name: LITESTREAM_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: litestream-secret
                key: LITESTREAM_ACCESS_KEY_ID
          - name: LITESTREAM_SECRET_ACCESS_KEY
            valueFrom:
              secretKeyRef:
                name: litestream-secret
                key: LITESTREAM_SECRET_ACCESS_KEY
          resources:
            limits:
              cpu: 2000m
              memory: 2048Mi
            requests:
              cpu: 1000m
              memory: 1024Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
            privileged: false

# Configuration of Litestream
# Litestream is a tool for replicating SQLite databases in real-time.
# https://litestream.io/reference/config/

# DATABASE SETTINGS -----------------------------------------------------------
# https://litestream.io/reference/config/#database-settings
dbs:
  # https://litestream.io/reference/config/#s3-replica
  - path: /var/lib/gokapi/gokapi.sqlite
    replicas:
      - type:   s3
        endpoint: private.s3.instance  # Specifies the endpoint URL of the S3-compatible service. Only required for non-AWS services.
        bucket: oc-2ndlevel-gokapi  # Specifies the name of the remote bucket to replicate to.
        path:   gokapi.sqlite  # Specifies the path to use within the bucket.
        region: us-east-1
        # access-key-id: AKIAxxxxxxxxxxxxxxxx  # Replica-specific authentication key. If not specified, the global key or the LITESTREAM_ACCESS_KEY_ID environment variable will be used instead. The AWS_ACCESS_KEY_ID variable can also be used.
        # secret-access-key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/xxxxxxxxx  # Replica-specific secret key. If not specified, the global secret or the LITESTREAM_SECRET_ACCESS_KEY environment variable will be used instead. The AWS_SECRET_ACCESS_KEY variable can also be used.
        # skip-verify: "false"  # Disables TLS verification. This is useful when testing against a local node such as MinIO and you are using self-signed certificates.
        # validation-interval: 6h  # https://litestream.io/reference/config/#validation-interval

# LOGGING ---------------------------------------------------------------------
logging:
  level: info  # Logging level can be set to “debug”, “info”, “warn” or “error”. 
  type: json  # Log type can be set to either “text” or “json”. 
  stderr: false  # By setting stderr to true logs will be written to stderr instead of stdout.

# PROMETHEUS METRICS ----------------------------------------------------------
# Litestream produces a continuous stream of metrics that are exported as a Prometheus endpoint. 
# These are disabled by default as it requires an HTTP server to start. 
# When you start Litestream with this setting enabled, you’ll see metrics at http://localhost:9090/metrics
# You can enable it by setting a bind address in the config:
# addr: ":9090"

Followed this guide: https://litestream.io/guides/kubernetes/

Trying to add persistance to the metadata sqlite in our deployment of this app: https://github.com/Forceu/Gokapi

[hifi]

Your private S3 implementation doesn't return the Content-Length header that Litestream expects for metrics. It could be fixed downstream in Litestream not to expect that information to exist but most S3 implementations do include it.

[MalteMagnussen]

We are using NetApp StorageGrid S3 just for info :)

Thanks for the rapid response.

If we don't expose the Prometheus Metrics, is it still expecting that header? Or do you mean something else, when you say "metrics"? @hifi

EDIT: https://kb.netapp.com/hybrid/StorageGRID/Protocols/Missing_content-length_header_issue_in_S3_get_requests_on_StorageGRID