grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.81k stars 3.43k forks source link

How to use migrate command to migrate loki data from one loki to the other new loki #12221

Open ruiruizeng opened 7 months ago

ruiruizeng commented 7 months ago

this is the migrate cmd url from loki repo this is my cmd,i want to use this command migrate old loki data to new loki and dont stop the old loki service,but i was failed

migrate -source.config.file=/opt/source.loki.config.yaml \
-dest.config.file=/opt/dest.loki.config.yaml \
-source.tenant=1124 -dest.tenant=1124 \
-from=2020-06-16T14:00:00-00:00 \
-to=2021-01-01T00:00:00-00:00

and have this error image

i use this config,its my loki service config ,i think this config not used for migrate cmd but i dont find the correct configuration file

common:
  instance_addr: 10.1.xx.xx
chunk_store_config:
  max_look_back_period: 0s
compactor:
  shared_store: filesystem
  working_directory: /data/loki/boltdb-shipper-compactor
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      replication_factor: 1
  max_transfer_retries: 0
  wal:
    dir: /data/loki/wal
limits_config:
  enforce_metric_name: false
  max_entries_limit_per_query: 20000
  max_line_size: 1024kb
  max_line_size_truncate: true
  max_query_parallelism: 2
  max_query_series: 100000
  reject_old_samples: true
  reject_old_samples_max_age: 168h
memberlist:
  join_members:
  - 'loki-memberlist'
querier:
  multi_tenant_queries_enabled: true
ruler:
  alertmanager_url: http://xxxx
  enable_api: true
  ring:
    kvstore:
      store: inmemory
  rule_path: /tmp
  storage:
    local:
      directory: /rules
    type: local
schema_config:
  configs:
  - from: "2020-10-24"
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v11
    store: boltdb-shipper
server:
  grpc_listen_port: 9095
  http_listen_port: 30214
  grpc_server_max_recv_msg_size: 16777216
  grpc_server_max_send_msg_size: 16777216
storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: filesystem
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
ruiruizeng commented 7 months ago

now i edit my config ,the command run is ok ,but no data migrate

common:
  instance_addr: 10.1.x.x
schema_config:
  configs:
  - from: "2020-10-24"
    index:
      period: 24h
      prefix: index_
    object_store: filesystem
    schema: v11
    store: boltdb-shipper
server:
  grpc_listen_port: 9095
  http_listen_port: xxxx
  grpc_server_max_recv_msg_size: 16777216
  grpc_server_max_send_msg_size: 16777216
storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h
  filesystem:
    directory: /data/loki/chunks

image

rgroothuijsen commented 7 months ago

Is /data/loki/chunks a reachable directory from the location where you're executing the migrate command? From what I can tell, it transfers files using the local file system.

ruiruizeng commented 6 months ago

Is /data/loki/chunks a reachable directory from the location where you're executing the migrate command? From what I can tell, it transfers files using the local file system.

I used to run migrate in docker to expose the interface of loki. My loki was deployed on k8s, so it was difficult to install migrate locally.

mehransaeed7810 commented 2 months ago

I am trying to use this migration tool. what would the source and dest config file look like? anyone got any luck with it