taosdata / TDengine

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios
https://tdengine.com
GNU Affero General Public License v3.0
23.37k stars 4.86k forks source link

After dropping database ,taosd service coredump #16192

Closed songcocl closed 5 months ago

songcocl commented 2 years ago

docker使用tdengine/tdengine:3.0.0.0镜像启动服务。 用/rest/sql/tenant0接口执行DROP DATABASE tenant0后,服务崩溃,docker容器一直自动重启。 以下是重启的日志: sysctl: setting key "kernel.core_pattern": Read-only file system

config file not found

08/17 18:20:46.436566 00000051 TAOS_ADAPTER info "start server:00000051" model=main

08/17 18:20:46.436725 00000051 TAOS_ADAPTER info "init plugin opentsdb/v1" model=plugin

08/17 18:20:46.436871 00000051 TAOS_ADAPTER info "init plugin opentsdb_telnet/v1" model=plugin

08/17 18:20:46.437814 00000051 TAOS_ADAPTER info "opentsdb_telnet disabled" model=opentsdb_telnet

08/17 18:20:46.437825 00000051 TAOS_ADAPTER info "init plugin prometheus/v1" model=plugin

08/17 18:20:46.437949 00000051 TAOS_ADAPTER info "init plugin statsd/v1" model=plugin

08/17 18:20:46.439550 00000051 TAOS_ADAPTER info "init plugin collectd/v1" model=plugin

08/17 18:20:46.440594 00000051 TAOS_ADAPTER info "init plugin influxdb/v1" model=plugin

08/17 18:20:46.440719 00000051 TAOS_ADAPTER info "init plugin node_exporter/v1" model=plugin

08/17 18:20:46.442185 00000051 TAOS_ADAPTER info "node_exporter disabled" model=NodeExporter

08/17 18:20:46.442198 00000051 TAOS_ADAPTER info "all plugin init finish" model=plugin

08/17 18:20:46.442657 00000051 TAOS_ADAPTER info "UDP listening on "[::]:6044"" model=statsd

08/17 18:20:46.442819 00000051 TAOS_ADAPTER info "Started the statsd service on ":6044"" model=statsd

08/17 18:20:46.442871 00000051 TAOS_ADAPTER info "all plugin start finish" model=plugin

08/17 18:20:46.443612 00000051 TAOS_ADAPTER info "server on : 6041" model=main

run TDengine with single node.

08/17 18:20:46.907294 00000069 UTL fail to load apoll url

08/17 18:20:46.907318 00000069 UTL load from cfg file /etc/taos/taos.cfg success

08/17 18:20:46.907323 00000069 UTL failed to load env file: .env

08/17 18:20:46.907896 00000069 UTL load from env variables cfg success

08/17 18:20:46.908174 00000069 UTL fail to load apoll url

08/17 18:20:46.908208 00000069 UTL load from cfg file /etc/taos/taos.cfg success

08/17 18:20:46.908213 00000069 UTL failed to load env file: .env

08/17 18:20:46.908874 00000069 UTL load from env variables cfg success

08/17 18:20:46.908951 00000069 UTL global config

08/17 18:20:46.908957 00000069 UTL =================================================================

08/17 18:20:46.908959 00000069 UTL default firstEp tdengine1:6030

08/17 18:20:46.908961 00000069 UTL default secondEp tdengine1:6030

08/17 18:20:46.908963 00000069 UTL default fqdn tdengine1

08/17 18:20:46.908965 00000069 UTL default serverPort 6030

08/17 18:20:46.908967 00000069 UTL default tempDir /tmp

08/17 18:20:46.908969 00000069 UTL default minimalTmpDirGB 1.00

08/17 18:20:46.908973 00000069 UTL default shellActivityTimer 3

08/17 18:20:46.908975 00000069 UTL default compressMsgSize -1

08/17 18:20:46.908976 00000069 UTL default compressColData -1

08/17 18:20:46.908978 00000069 UTL default queryPolicy 1

08/17 18:20:46.908980 00000069 UTL default querySmaOptimize 0

08/17 18:20:46.908982 00000069 UTL cfg_file smlChildTableName tname

08/17 18:20:46.908984 00000069 UTL default smlTagName _tag_null

08/17 18:20:46.908986 00000069 UTL default smlDataFormat 1

08/17 18:20:46.908987 00000069 UTL default numOfTaskQueueThreads 4

08/17 18:20:46.908989 00000069 UTL default dataDir /var/lib/taos

08/17 18:20:46.908991 00000069 UTL default minimalDataDirGB 2.00

08/17 18:20:46.908993 00000069 UTL default supportVnodes 16

08/17 18:20:46.908994 00000069 UTL default maxShellConns 50000

08/17 18:20:46.908996 00000069 UTL default statusInterval 1

08/17 18:20:46.908998 00000069 UTL default minSlidingTime 10

08/17 18:20:46.909000 00000069 UTL default minIntervalTime 1

08/17 18:20:46.909001 00000069 UTL default maxNumOfDistinctRes 10000000

08/17 18:20:46.909003 00000069 UTL default countAlwaysReturnValue 1

08/17 18:20:46.909005 00000069 UTL default queryBufferSize -1

08/17 18:20:46.909006 00000069 UTL default printAuth 0

08/17 18:20:46.909008 00000069 UTL default multiProcess 0

08/17 18:20:46.909010 00000069 UTL default mnodeShmSize 20972544

08/17 18:20:46.909012 00000069 UTL default vnodeShmSize 104858624

08/17 18:20:46.909013 00000069 UTL default qnodeShmSize 41944064

08/17 18:20:46.909015 00000069 UTL default snodeShmSize 41944064

08/17 18:20:46.909017 00000069 UTL default bnodeShmSize 41944064

08/17 18:20:46.909019 00000069 UTL default numOfShmThreads 1

08/17 18:20:46.909020 00000069 UTL default numOfRpcThreads 4

08/17 18:20:46.909022 00000069 UTL default numOfCommitThreads 4

08/17 18:20:46.909023 00000069 UTL default numOfMnodeReadThreads 1

08/17 18:20:46.909025 00000069 UTL default numOfVnodeQueryThreads 16

08/17 18:20:46.909027 00000069 UTL default numOfVnodeStreamThreads 4

08/17 18:20:46.909029 00000069 UTL default numOfVnodeFetchThreads 4

08/17 18:20:46.909031 00000069 UTL default numOfVnodeWriteThreads 8

08/17 18:20:46.909032 00000069 UTL default numOfVnodeSyncThreads 16

08/17 18:20:46.909034 00000069 UTL default numOfQnodeQueryThreads 16

08/17 18:20:46.909035 00000069 UTL default numOfQnodeFetchThreads 4

08/17 18:20:46.909037 00000069 UTL default numOfSnodeSharedThreads 2

08/17 18:20:46.909039 00000069 UTL default numOfSnodeUniqueThreads 2

08/17 18:20:46.909040 00000069 UTL default rpcQueueMemoryAllowed 3352822988

08/17 18:20:46.909042 00000069 UTL default monitor 1

08/17 18:20:46.909049 00000069 UTL default monitorInterval 30

08/17 18:20:46.909051 00000069 UTL default monitorFqdn

08/17 18:20:46.909052 00000069 UTL default monitorPort 6043

08/17 18:20:46.909054 00000069 UTL default monitorMaxLogs 100

08/17 18:20:46.909056 00000069 UTL default monitorComp 0

08/17 18:20:46.909058 00000069 UTL default telemetryReporting 0

08/17 18:20:46.909059 00000069 UTL default telemetryInterval 86400

08/17 18:20:46.909061 00000069 UTL default telemetryServer telemetry.taosdata.com

08/17 18:20:46.909062 00000069 UTL default telemetryPort 80

08/17 18:20:46.909064 00000069 UTL default transPullupInterval 2

08/17 18:20:46.909066 00000069 UTL default mqRebalanceInterval 2

08/17 18:20:46.909067 00000069 UTL default ttlUnit 86400

08/17 18:20:46.909069 00000069 UTL default ttlPushInterval 86400

08/17 18:20:46.909071 00000069 UTL default udf 1

08/17 18:20:46.909073 00000069 UTL default configDir /etc/taos

08/17 18:20:46.909075 00000069 UTL default scriptDir /etc/taos

08/17 18:20:46.909076 00000069 UTL default logDir /var/log/taos

08/17 18:20:46.909078 00000069 UTL default minimalLogDirGB 1.00

08/17 18:20:46.909079 00000069 UTL default numOfLogLines 10000000

08/17 18:20:46.909081 00000069 UTL default asyncLog 1

08/17 18:20:46.909083 00000069 UTL default logKeepDays 0

08/17 18:20:46.909085 00000069 UTL default debugFlag 0

08/17 18:20:46.909086 00000069 UTL default simDebugFlag 143

08/17 18:20:46.909088 00000069 UTL default tmrDebugFlag 131

08/17 18:20:46.909089 00000069 UTL default uDebugFlag 131

08/17 18:20:46.909091 00000069 UTL default rpcDebugFlag 131

08/17 18:20:46.909092 00000069 UTL default jniDebugFlag 131

08/17 18:20:46.909094 00000069 UTL default qDebugFlag 131

08/17 18:20:46.909096 00000069 UTL default cDebugFlag 131

08/17 18:20:46.909097 00000069 UTL default dDebugFlag 135

08/17 18:20:46.909099 00000069 UTL default vDebugFlag 135

08/17 18:20:46.909101 00000069 UTL default mDebugFlag 135

08/17 18:20:46.909102 00000069 UTL default wDebugFlag 135

08/17 18:20:46.909104 00000069 UTL default sDebugFlag 135

08/17 18:20:46.909106 00000069 UTL default tsdbDebugFlag 131

08/17 18:20:46.909107 00000069 UTL default tqDebugFlag 135

08/17 18:20:46.909109 00000069 UTL default fsDebugFlag 135

08/17 18:20:46.909110 00000069 UTL default udfDebugFlag 135

08/17 18:20:46.909112 00000069 UTL default smaDebugFlag 135

08/17 18:20:46.909114 00000069 UTL default idxDebugFlag 135

08/17 18:20:46.909115 00000069 UTL default tdbDebugFlag 131

08/17 18:20:46.909117 00000069 UTL default metaDebugFlag 135

08/17 18:20:46.909119 00000069 UTL default timezone Asia/Shanghai (CST, +0800)

08/17 18:20:46.909121 00000069 UTL default locale en_US.UTF-8

08/17 18:20:46.909122 00000069 UTL default charset UTF-8

08/17 18:20:46.909124 00000069 UTL default enableCoreFile 1

08/17 18:20:46.909127 00000069 UTL default numOfCores 8.00

08/17 18:20:46.909129 00000069 UTL default openMax 1048576

08/17 18:20:46.909131 00000069 UTL default streamMax 16

08/17 18:20:46.909132 00000069 UTL default pageSizeKB 4

08/17 18:20:46.909134 00000069 UTL default totalMemoryKB 32742412

08/17 18:20:46.909135 00000069 UTL default os sysname Linux

08/17 18:20:46.909138 00000069 UTL default os nodename tdengine1

08/17 18:20:46.909140 00000069 UTL default os release 4.15.0-142-generic

08/17 18:20:46.909142 00000069 UTL default os version #146-Ubuntu SMP Tue Apr 13 01:11:19 UTC 2021

08/17 18:20:46.909144 00000069 UTL default os machine x86_64

08/17 18:20:46.909146 00000069 UTL default version 3.0.0.0

08/17 18:20:46.909147 00000069 UTL default compatible_version 3.0.0.0

08/17 18:20:46.909149 00000069 UTL default gitinfo 39c93de6c94cd3eacdd937fea4ca18562be07618

08/17 18:20:46.909151 00000069 UTL default buildinfo Built at 2022-08-13 00:43

08/17 18:20:46.909153 00000069 UTL =================================================================

08/17 18:20:46.910410 00000069 DND start to init dnode env

08/17 18:20:46.910454 00000069 DND dnode will run in single-process mode

08/17 18:20:46.911030 00000069 DND dnode env is initialized

08/17 18:20:46.911038 00000069 DND start to init service

08/17 18:20:46.911046 00000069 DND node:dnode, start to open

08/17 18:20:46.911049 00000069 UTL worker:dnode-mgmt is initialized, min:1 max:1

08/17 18:20:46.911131 00000080 UDF start to init udfd

08/17 18:20:46.911621 00000080 UDF ERROR can not spawn udfd. path: taosd/udfd, error: no such file or directory

08/17 18:20:46.911704 00000069 UDF udfd is cleaned up after spawn err

08/17 18:20:46.911716 00000069 DND ERROR failed to start udfd

08/17 18:20:46.911719 00000069 DND node:dnode, has been opened

08/17 18:20:46.911728 00000069 DND node:mnode, start to open

08/17 18:20:46.911755 00000069 WAL wal module is initialized, rsetId:3

08/17 18:20:46.911907 00000069 DND mnode start to open

08/17 18:20:46.913111 00000069 MND start to open mnode sync, standby:0

08/17 18:20:46.913234 00000069 SYN vgId:1, sync node use snapshot

08/17 18:20:46.913266 00000069 UTL worker:mnode-query is initialized, min:4 max:4

08/17 18:20:46.913362 00000069 UTL worker:mnode-fetch is initialized, min:1 max:1

08/17 18:20:46.913402 00000069 UTL worker:mnode-read is initialized, min:1 max:1

08/17 18:20:46.913428 00000069 UTL worker:mnode-write is initialized, min:1 max:1

08/17 18:20:46.913453 00000069 UTL worker:mnode-sync is initialized, min:1 max:1

08/17 18:20:46.913476 00000069 UTL worker:mnode-monitor is initialized, min:1 max:1

08/17 18:20:46.913508 00000069 DND node:mnode, has been opened

08/17 18:20:46.913521 00000069 DND node:vnode, start to open

08/17 18:20:46.913632 00000069 UTL worker:vnode-query is initialized, min:16 max:16

08/17 18:20:46.913640 00000069 UTL worker:vnode-stream is initialized, min:4 max:4

08/17 18:20:46.913643 00000069 UTL worker:vnode-mgmt is initialized, min:1 max:1

08/17 18:20:46.913666 00000069 UTL worker:vnode-monitor is initialized, min:1 max:1

08/17 18:20:46.913752 00000069 DND open 1 vnodes with 4 threads

08/17 18:20:46.915618 00000101 SYN vgId:3, sync node use snapshot

08/17 18:20:46.916119 00000069 DND successfully opened 1 vnodes

08/17 18:20:46.916136 00000069 UDF get dnodeId:1 from env, pipe path:/var/lib/taos/.udfd.sock.1

08/17 18:20:46.916198 00000069 UDF udfc initialized

08/17 18:20:46.916205 00000069 DND node:vnode, has been opened

08/17 18:20:46.918471 00000069 MND trans:13, execute in mnode which not leader

08/17 18:20:46.918520 00000069 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014

08/17 18:20:46.918552 00000069 MND trans:13, execute in mnode which not leader

08/17 18:20:46.918603 00000069 MND trans:14, execute in mnode which not leader

08/17 18:20:46.918614 00000069 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014

08/17 18:20:46.918637 00000069 MND trans:14, execute in mnode which not leader

08/17 18:20:46.918676 00000069 MND trans:15, execute in mnode which not leader

08/17 18:20:46.918687 00000069 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014

08/17 18:20:46.918710 00000069 MND trans:15, execute in mnode which not leader

08/17 18:20:46.918748 00000069 MND trans:16, execute in mnode which not leader

08/17 18:20:46.918759 00000069 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014

08/17 18:20:46.918781 00000069 MND trans:16, execute in mnode which not leader

08/17 18:20:46.918820 00000069 MND trans:17, execute in mnode which not leader

08/17 18:20:46.918830 00000069 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014

08/17 18:20:46.919471 00000069 MND mnode sync restore finished, and will handle outstanding transactions

08/17 18:20:46.919487 00000069 MND ERROR failed to execute redoActions since:Database not ready, code:0x80000014

08/17 18:20:46.950403 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:1, term:1, msg-index:1, weak:0, code:0, state:102 leader, type:vnode-create-stb

08/17 18:20:46.950456 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:2, term:1, msg-index:2, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950472 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:3, term:1, msg-index:3, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.950486 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:4, term:1, msg-index:4, weak:0, code:0, state:102 leader, type:vnode-create-stb

08/17 18:20:46.950504 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:5, term:1, msg-index:5, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950521 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:6, term:1, msg-index:6, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950540 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:7, term:1, msg-index:7, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950554 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:8, term:1, msg-index:8, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950568 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:9, term:1, msg-index:9, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950593 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:10, term:1, msg-index:10, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950611 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:11, term:1, msg-index:11, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950628 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:12, term:1, msg-index:12, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950646 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:13, term:1, msg-index:13, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950664 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:14, term:1, msg-index:14, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950682 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:15, term:1, msg-index:15, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950698 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:16, term:1, msg-index:16, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950762 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:17, term:1, msg-index:17, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950789 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:18, term:1, msg-index:18, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950809 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:19, term:1, msg-index:19, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950826 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:20, term:1, msg-index:20, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950845 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:21, term:1, msg-index:21, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.950872 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:22, term:1, msg-index:22, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950894 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:23, term:1, msg-index:23, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950909 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:24, term:1, msg-index:24, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950924 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:25, term:1, msg-index:25, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950946 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:26, term:1, msg-index:26, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.950970 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:28, term:2, msg-index:28, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.950997 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:30, term:3, msg-index:30, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.951026 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:32, term:4, msg-index:32, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.951057 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:34, term:5, msg-index:34, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.951077 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:35, term:5, msg-index:35, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.951117 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:37, term:6, msg-index:37, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.951151 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:39, term:7, msg-index:39, weak:0, code:0, state:102 leader, type:vnode-drop-ttl-stb

08/17 18:20:46.951175 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:40, term:7, msg-index:40, weak:0, code:0, state:102 leader, type:vnode-alter-stb

08/17 18:20:46.951218 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:41, term:7, msg-index:41, weak:0, code:0, state:102 leader, type:vnode-alter-stb

taosd: /home/ubuntu/zqwang/TDinternal/community/source/libs/tdb/src/db/tdbBtree.c:196: tdbBtreeInsert: Assertion `0' failed.

08/17 18:20:46.951236 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:42, term:7, msg-index:42, weak:0, code:0, state:102 leader, type:submit

08/17 18:20:46.951257 00000069 VND vgId:3, commit-cb is excuted, fsm:0x7f60e012baa0, index:43, term:7, msg-index:43, weak:0, code:0, state:102 leader, type:vnode-create-stb

Aborted (core dumped)

yu285 commented 2 years ago

可以添加微信 15652223354 ,我们具体看下

yu285 commented 5 months ago

3.0早期版本的bug 都已经都修复了,但是3.0.2.5 之前可能与现在的最新版(3.3.0.0)并不兼容,所以无法直接升级。因此建议重新部署一下最新的 3.3.0.0 ,后续升级都很方便,直接安装最新版软件即可。