search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.98k stars 3.46k forks source link

TSDB migration error #14054

Open surekha3 opened 2 months ago

surekha3 commented 2 months ago

Describe the bug We have loki running in simple scalable mode in AWS EKS cluster with 2.8.2 version. Storage format taken is boltDB shipper form. I want to migrate it to TSDB form with object storage as s3. Though it's working well with existing s3 bucket config, when configured TSDB, it's not picking up the same bucket even tried to modified the configuration in many ways

To Reproduce Steps to reproduce the behavior:

  1. Started Loki 2.8.2 version with simple scalable mode in EKS cluster

  2. Below is the config used before migration loki: auth_enabled: false externalConfigSecretName: "loki" readinessProbe: null

    -- Limits config

    limits_config: retention_period: 4392h # Adjust as needed - Default retention to 1 month enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h max_cache_freshness_per_query: 10m split_queries_by_interval: 15m

    -- Storage config. Providing this will automatically populate all necessary storage configs in the templated config. storage: bucketNames: chunks: ruler: type: s3 s3: region: us-east-1 http_config: idle_conn_timeout: 300s response_header_timeout: 300s

    -- Additional storage config

    storage_config: aws: sse_encryption: true

  3. Below is the config used for migrationloki: config: | schema_config: configs:

    • from: 2022-01-11 store: boltdb-shipper object_store: s3 schema: v12 index: prefix: lokiindex period: 24h
    • from: 2024-09-07 store: tsdb object_store: s3 schema: v12 index: prefix: lokiindex period: 24h

    -- Limits config

    limits_config: retention_period: 4392h # Adjust as needed - Default retention to 1 month enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h max_cache_freshness_per_query: 10m split_queries_by_interval: 15m

    -- Additional storage config

    storage_config: aws: s3: s3://s3.us-east-1.amazonaws.com bucketnames: chunks: "" ruler: "" region: us-east-1 s3forcepathstyle: true sse_encryption: true tsdb_shipper: active_index_directory: /var/loki/tsdb-index cache_location: /var/loki/tsdb-cache cache_ttl: 168h shared_store: s3

Expected behavior Data storing in boltDB shipper form should be migrated to TSDB format

Environment:

Screenshots, Promtail config, or terminal output

level=warn ts=2024-09-05T05:28:21.066645268Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")." level=info ts=2024-09-05T05:28:21.068204901Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.2, branch=HEAD, revision=9f809eda7)" level=info ts=2024-09-05T05:28:21.068699895Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses" level=warn ts=2024-09-05T05:28:21.069391196Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - embedded-cache" level=warn ts=2024-09-05T05:28:21.069528094Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead" level=warn ts=2024-09-05T05:28:21.069543989Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache" level=error ts=2024-09-05T05:28:21.069758759Z caller=log.go:171 msg="error running loki" err="at least one bucket name must be specified\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.buckets\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:332\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.NewS3ObjectClient\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:152\ngithub.com/grafana/loki/pkg/storage.NewObjectClient\n\t/src/loki/pkg/storage/factory.go:483\ngithub.com/grafana/loki/pkg/storage.NewChunkClient\n\t/src/loki/pkg/storage/factory.go:325\ngithub.com/grafana/loki/pkg/storage.(store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:185\ngithub.com/grafana/loki/pkg/storage.(store).init\n\t/src/loki/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:147\ngithub.com/grafana/loki/pkg/loki.(Loki).initStore\n\t/src/loki/pkg/loki/modules.go:655\ngithub.com/grafana/dskit/modules.(Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror creating object client\ngithub.com/grafana/loki/pkg/storage.(store).chunkClientForPeriod\n\t/src/loki/pkg/storage/store.go:187\ngithub.com/grafana/loki/pkg/storage.(store).init\n\t/src/loki/pkg/storage/store.go:155\ngithub.com/grafana/loki/pkg/storage.NewStore\n\t/src/loki/pkg/storage/store.go:147\ngithub.com/grafana/loki/pkg/loki.(Loki).initStore\n\t/src/loki/pkg/loki/modules.go:655\ngithub.com/grafana/dskit/modules.(Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

surekha3 commented 2 months ago

Can anyone help me with this please