search

infinilabs / gateway

🚀 A High-performance gateway designed for search scenarios. Good friend of Elasticsearch/Opensearch/Easysearch.
https://www.infinilabs.com/en/docs/latest/gateway/
28 stars 3 forks source link

Many ERROR log [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found #38

Closed dyrnq closed 7 months ago

dyrnq commented 7 months ago

Description

Many ERROR log [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found

Steps to reproduce

  1. systemctl start gateway
  2. journalctl -exu gateway --no-pager 
    
Jan 16 08:55:59 m178 gateway[3188]: [01-16 08:55:59] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:55:59 m178 gateway[3188]: [01-16 08:55:59] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:00 m178 gateway[3188]: [01-16 08:56:00] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:00 m178 gateway[3188]: [01-16 08:56:00] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:01 m178 gateway[3188]: [01-16 08:56:01] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:01 m178 gateway[3188]: [01-16 08:56:01] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:02 m178 gateway[3188]: [01-16 08:56:02] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:02 m178 gateway[3188]: [01-16 08:56:02] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:03 m178 gateway[3188]: [01-16 08:56:03] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:03 m178 gateway[3188]: [01-16 08:56:03] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:04 m178 gateway[3188]: [01-16 08:56:04] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:04 m178 gateway[3188]: [01-16 08:56:04] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:05 m178 gateway[3188]: [01-16 08:56:05] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:05 m178 gateway[3188]: [01-16 08:56:05] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:06 m178 gateway[3188]: [01-16 08:56:06] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:06 m178 gateway[3188]: [01-16 08:56:06] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:07 m178 gateway[3188]: [01-16 08:56:07] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:07 m178 gateway[3188]: [01-16 08:56:07] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:08 m178 gateway[3188]: [01-16 08:56:08] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:08 m178 gateway[3188]: [01-16 08:56:08] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:09 m178 gateway[3188]: [01-16 08:56:09] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:09 m178 gateway[3188]: [01-16 08:56:09] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:10 m178 gateway[3188]: [01-16 08:56:10] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:10 m178 gateway[3188]: [01-16 08:56:10] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:11 m178 gateway[3188]: [01-16 08:56:11] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:11 m178 gateway[3188]: [01-16 08:56:11] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:12 m178 gateway[3188]: [01-16 08:56:12] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:12 m178 gateway[3188]: [01-16 08:56:12] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:13 m178 gateway[3188]: [01-16 08:56:13] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:13 m178 gateway[3188]: [01-16 08:56:13] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:14 m178 gateway[3188]: [01-16 08:56:14] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:14 m178 gateway[3188]: [01-16 08:56:14] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:15 m178 gateway[3188]: [01-16 08:56:15] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:15 m178 gateway[3188]: [01-16 08:56:15] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:16 m178 gateway[3188]: [01-16 08:56:16] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:16 m178 gateway[3188]: [01-16 08:56:16] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:17 m178 gateway[3188]: [01-16 08:56:17] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:17 m178 gateway[3188]: [01-16 08:56:17] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:18 m178 gateway[3188]: [01-16 08:56:18] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found
Jan 16 08:56:18 m178 gateway[3188]: [01-16 08:56:18] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:19 m178 gateway[3188]: [01-16 08:56:19] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/aba6336762a30ebff588b0882bab0d49/000000000.dat not found
Jan 16 08:56:19 m178 gateway[3188]: [01-16 08:56:19] [ERR] [diskqueue_consumer.go:133] error in json indexing worker,/data/es/gateway/data/gateway/nodes/cmi7g1unu94gu7fkg1r0/queue/bd364f1916e904bfa31a653df826f936/000000000.dat not found

>  /lib/systemd/system/gateway.service

`systemctl cat gateway`

```bash
# /lib/systemd/system/gateway.service
[Unit]
Description=cubefs
After=network-online.target local-fs.target time-sync.target
Wants=network-online.target local-fs.target time-sync.target

[Service]
Type=simple
ExecStart=/usr/local/bin/gateway -config /data/es/gateway/config/gateway.yml
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-failure
RestartSec=10s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target

/data/es/gateway/config/gateway.yml

allow_multi_instance: true
path.data: /data/es/gateway/data
path.logs: /data/es/gateway/log

elasticsearch:
- name: cluster01
  enabled: true
  endpoint: http://192.168.6.175:9200
  basic_auth:
    username: elastic
    password: test123
  traffic_control.max_bytes_per_node: 1010485760
  metadata_cache_enabled: false # Whether to cache the cluster info in memory cache
- name: cluster02
  enabled: true
  endpoint: http://192.168.6.176:9200
  basic_auth:
    username: elastic
    password: test123
  discovery:
    enabled: false
entry:
  - name: my_es_entry
    enabled: true
    router: my_router
    max_concurrency: 200000
    network:
      binding: 0.0.0.0:8000
      reuse_port: true

flow:
 # 写请求优先发给主集群, 当主集群不可用时发给备用集群
 # 当主集群数据写入成功时，记录到队列中，异步消费写入备用集群
  - name: write-flow
    filter:
      - if:
      # 当主集群可用时
          cluster_available: ["cluster01"]
        then:
          # 先将数据写入主集群
          - elasticsearch:
              elasticsearch: "cluster01"
            # 写入消息队列,等待 pipeline 异步消费到备用集群
          - queue:
              queue_name: "cluster02-queue"
        else:
          - elasticsearch:
              elasticsearch: "cluster02"
          - queue:
              queue_name: "cluster01-queue"
  # 读请求优先发给主集群, 当主集群不可用时发给备用集群
  - name: read-flow
    filter:
      - if:
          cluster_available: ["cluster01"]
        then:
          - elasticsearch:
              elasticsearch: "cluster01"
        else:
          - elasticsearch:
              elasticsearch: "cluster02"

router:
  - name: my_router
    # 默认路由
    default_flow: write-flow
    # 读请求路由
    rules:
      - method:
          - "GET"
          - "HEAD"
        pattern:
          - "/{any:*}"
        flow:
          - read-flow
      - method:
          - "POST"
          - "GET"
        pattern:
          - "/_refresh"
          - "/_count"
          - "/_search"
          - "/_msearch"
          - "/_mget"
          - "/{any_index}/_count"
          - "/{any_index}/_search"
          - "/{any_index}/_msearch"
          - "/{any_index}/_mget"
        flow:
          - read-flow

pipeline:
  - name: cluster01-consumer
    auto_start: true
    keep_running: true
    processor:
      - queue_consumer:
          input_queue: "cluster01-queue" 
          elasticsearch: "cluster01"
          when:
            cluster_available: ["cluster01"] # 当集群可用时，才消费队列中的数据
  - name: cluster02-consumer
    auto_start: true
    keep_running: true
    processor:
      - queue_consumer:
          input_queue: "cluster02-queue"
          elasticsearch: "cluster02"
          when:
            cluster_available: ["cluster02"]

disk_queue:
  prepare_files_to_read: true
  #max_bytes_per_file: 20971520
  eof_retry_delay_in_ms: 500
  cleanup_files_on_init: false
  retention:
    max_num_of_local_files: 20 # automatically cleanup consumed files
  compress:
    segment:
      enabled: true
    delete_after_compress: true # trigger cleanup after compression.
    idle_threshold: 20 # max number of uncompressed consumed files to preserve.

badger:
  enabled: true
  single_bucket_mode: true
  path: '' # defaults to {path.data}/gateway/node/{nodeID}/badger/
  memory_mode: false # don't persist data to disk
  sync_writes: false # flush to disk on every write
  mem_table_size: 10485760
  num_mem_tables: 1
  # lsm tuning options
  value_log_max_entries: 1000000
  value_log_file_size: 536870912
  value_threshold: 1048576
  num_level0_tables: 1
  num_level0_tables_stall: 2

Expected behavior

No [ERR] [diskqueue_consumer.go:133] error in ?

Actual behavior

As mentioned above

Environment

[GATEWAY] A light-weight, powerful and high-performance search gateway. [GATEWAY] 1.21.0#1229, 2023-12-28 08:22:36, 2024-12-31 10:10:10, 5ca75b9539303a3ceb62e37e72811b580e7e004f gateway 1.21.0 1229 2023-12-28 08:22:36 +0000 UTC 2024-12-31 10:10:10 +0000 UTC 5ca75b9539303a3ceb62e37e72811b580e7e004f


* Operating system and version: 
```bash
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
medcl commented 7 months ago

queue_consumer is not supposed to use any more, please use consumer or bulk_indexing instead.

dyrnq commented 7 months ago

tks reply, I will try.