search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.57k stars 3.53k forks source link

InfluxDB crash with error [tcp] 2020/10/29 02:34:46 tcp.Mux: cannot read header byte: EOF timeout reached before the port went into state "inuse" #19849

Open minhhoangvn opened 3 years ago

minhhoangvn commented 3 years ago

Steps to reproduce: List the minimal actions needed to reproduce the behavior.

  1. Run K6.IO for 8h to seed data to InfluxDB
  2. Use Grafana to query data last 24 hours

Expected behavior: InfluxDB should return the query data

Actual behavior: InfluxDB crash with the error message 

[tcp] 2020/10/29 02:34:46 tcp.Mux: cannot read header byte: EOF
timeout reached before the port went into state "inuse"

image

Environment info:

Config:

  influxdb:
    configuration: |-
      reporting-disabled = false
      [meta]
        dir = "/bitnami/influxdb/meta"
      [data]
        dir = "/var/lib/influxdb/data"
        wal-dir = "/var/lib/influxdb/wal"
        index-version = "tsi1"
        trace-logging-enabled = true
        query-log-enabled = true
        max-index-log-file-size = "5m"
        max-series-per-database = 0
        max-values-per-tag = 0
      [coordinator]
        log-queries-after = "30s"
        write-timeout = "60s"
        query-timeout = 0
        max-select-point = 0
        max-select-series = 0
        max-select-buckets = 0
      [logging]
        level = "debug"

Logs:

influxdb 01:54:14.77 
influxdb 01:54:14.83 Welcome to the Bitnami influxdb container
influxdb 01:54:14.86 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-influxdb
influxdb 01:54:14.87 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-influxdb/issues
influxdb 01:54:14.93 
influxdb 01:54:14.94 INFO  ==> ** Starting InfluxDB setup **
influxdb 01:54:15.04 DEBUG ==> Validating settings in INFLUXDB_* env vars...
influxdb 01:54:15.06 WARN  ==> Authentication is disabled over HTTP and HTTPS. For safety reasons, enable it in a production environment.
influxdb 01:54:15.06 WARN  ==> The INFLUXDB_ADMIN_USER environment variable will be ignored since authentication is disabled.
influxdb 01:54:15.06 WARN  ==> The INFLUXDB_USER environment variable will be ignored since authentication is disabled.
influxdb 01:54:15.07 INFO  ==> Initializing InfluxDB...
influxdb 01:54:15.14 INFO  ==> Custom configuration /opt/bitnami/influxdb/etc/influxdb.conf detected!
influxdb 01:54:15.15 WARN  ==> The 'INFLUXDB_' environment variables override the equivalent options in the configuration file.
influxdb 01:54:15.16 WARN  ==> If a configuration option is not specified in either the configuration file or in an environment variable, InfluxDB uses its internal default configuration
influxdb 01:54:15.17 INFO  ==> Deploying InfluxDB with persisted data
influxdb 01:54:15.23 INFO  ==> Loading user's custom files from /docker-entrypoint-initdb.d ...
influxdb 01:54:15.26 INFO  ==> Starting InfluxDB in background...
ts=2020-10-29T01:54:15.633014Z lvl=info msg="InfluxDB starting" log_id=0Q8Wqkp0000 version=1.8.3 branch=1.8 commit=563e6c3d1a7a2790763c6289501095dbec19244e
ts=2020-10-29T01:54:15.633307Z lvl=info msg="Go runtime" log_id=0Q8Wqkp0000 version=go1.13.8 maxprocs=8
ts=2020-10-29T01:54:15.738866Z lvl=info msg="Using data dir" log_id=0Q8Wqkp0000 service=store path=/bitnami/influxdb/data
ts=2020-10-29T01:54:15.740115Z lvl=info msg="Compaction settings" log_id=0Q8Wqkp0000 service=store max_concurrent_compactions=4 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
ts=2020-10-29T01:54:15.740202Z lvl=info msg="Open store (start)" log_id=0Q8Wqkp0000 service=store trace_id=0Q8WqlF0000 op_name=tsdb_open op_event=start
ts=2020-10-29T01:54:15.965438Z lvl=info msg="index opened with 8 partitions" log_id=0Q8Wqkp0000 index=tsi
[tcp] 2020/10/29 01:54:15 tcp.Mux: cannot read header byte: EOF
ts=2020-10-29T01:54:16.056494Z lvl=info msg="tsm1 WAL starting" log_id=0Q8Wqkp0000 engine=tsm1 service=wal segment_size=10485760
ts=2020-10-29T01:54:16.056657Z lvl=info msg="tsm1 WAL writing" log_id=0Q8Wqkp0000 engine=tsm1 service=wal path=/bitnami/influxdb/wal/_internal/monitor/4
ts=2020-10-29T01:54:16.076144Z lvl=info msg="Reading file" log_id=0Q8Wqkp0000 engine=tsm1 service=cacheloader path=/bitnami/influxdb/wal/_internal/monitor/4/_00001.wal size=4208915
ts=2020-10-29T01:54:16.457679Z lvl=info msg="index opened with 8 partitions" log_id=0Q8Wqkp0000 index=tsi
ts=2020-10-29T01:54:16.633338Z lvl=info msg="tsm1 WAL starting" log_id=0Q8Wqkp0000 engine=tsm1 service=wal segment_size=10485760
ts=2020-10-29T01:54:16.636793Z lvl=info msg="tsm1 WAL writing" log_id=0Q8Wqkp0000 engine=tsm1 service=wal path=/bitnami/influxdb/wal/_internal/monitor/1
ts=2020-10-29T01:54:16.643335Z lvl=info msg="Opened file" log_id=0Q8Wqkp0000 engine=tsm1 service=filestore path=/bitnami/influxdb/data/_internal/monitor/1/000000001-000000001.tsm id=0 duration=0.284ms
ts=2020-10-29T01:54:16.644664Z lvl=info msg="Reloaded WAL cache" log_id=0Q8Wqkp0000 engine=tsm1 path=/bitnami/influxdb/wal/_internal/monitor/1 duration=0.745ms
ts=2020-10-29T01:54:24.943568Z lvl=info msg="Reloaded WAL cache" log_id=0Q8Wqkp0000 engine=tsm1 path=/bitnami/influxdb/wal/_internal/monitor/4 duration=8874.949ms
ts=2020-10-29T01:54:43.347035Z lvl=info msg="Opened shard" log_id=0Q8Wqkp0000 service=store trace_id=0Q8WqlF0000 op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/_internal/monitor/4 duration=27550.824ms
ts=2020-10-29T01:54:43.356708Z lvl=info msg="Opened shard" log_id=0Q8Wqkp0000 service=store trace_id=0Q8WqlF0000 op_name=tsdb_open index_version=tsi1 path=/bitnami/influxdb/data/_internal/monitor/1 duration=27560.545ms
timeout reached before the port went into state "inuse"
influxdb 01:54:46.53 INFO  ==> Stopping InfluxDB...
Terminated
jakoberpf commented 2 years ago

Hi @minhhoangvn, where you able to resolve this problem? I am facing the same error message of timeout reached before the port went into state "inuse" but am clueless what the actual issue.

jakoberpf commented 2 years ago

Info for everyone who might encounter this issue in the future. Setting the value of diagnosticMode.enabled=false or diagnosticMode.enabled=true fixed it for us. Note that if you set it to true the db will be in actual diagnostic mode, so if this is not desired, set it to false.

dshmatov commented 2 years ago

@minhhoangvn @jakoberpf this might help.

brujoand commented 10 months ago

The link to that PR is now gone, could someone please explain what the issue was? I'm seeing this error now using docker.io/bitnami/influxdb:2.7.3-debian-11-r0.