[Bug]: The milvus datanode node is constantly trying to restart

Is there an existing issue for this?

[X] I have searched the existing issues

Environment

- Milvus version:2.2.5
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 16
- GPU: 4
- Others:

Current Behavior

and load data is not nok

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

[2023/04/25 05:11:16.370 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180157409] [numRows=1] [2023/04/25 05:11:16.370 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180157409] [memorySize=17014] [2023/04/25 05:11:16.370 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180157409] [channel=by-dev-rootcoord-dml_40_440468305005730068v1] [size=30] [limit=25267] [memorySize=17014] [2023/04/25 05:11:16.385 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.385 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=629887] [2023/04/25 05:11:16.385 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1107] [limit=25267] [memorySize=629887] [2023/04/25 05:11:16.409 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=2] [2023/04/25 05:11:16.409 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=183222] [2023/04/25 05:11:16.409 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=322] [limit=25267] [memorySize=183222] [2023/04/25 05:11:16.420 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 05:11:16.420 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=198016] [2023/04/25 05:11:16.420 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=348] [limit=25267] [memorySize=198016] [2023/04/25 05:11:16.426 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 05:11:16.426 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=631025] [2023/04/25 05:11:16.426 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1109] [limit=25267] [memorySize=631025] [2023/04/25 05:11:16.455 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.456 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=631594] [2023/04/25 05:11:16.456 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1110] [limit=25267] [memorySize=631594] [2023/04/25 05:11:16.500 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.501 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=632163] [2023/04/25 05:11:16.501 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1111] [limit=25267] [memorySize=632163] [2023/04/25 05:11:16.539 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.541 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=632732] [2023/04/25 05:11:16.541 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1112] [limit=25267] [memorySize=632732] [2023/04/25 05:11:16.676 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.677 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=633301] [2023/04/25 05:11:16.677 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1113] [limit=25267] [memorySize=633301] [2023/04/25 05:11:16.728 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 05:11:16.728 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=183791] [2023/04/25 05:11:16.728 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=323] [limit=25267] [memorySize=183791] [2023/04/25 05:11:16.749 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.750 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=633870] [2023/04/25 05:11:16.750 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1114] [limit=25267] [memorySize=633870] [2023/04/25 05:11:16.810 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 05:11:16.810 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=184360] [2023/04/25 05:11:16.810 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=324] [limit=25267] [memorySize=184360] [2023/04/25 05:11:16.870 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 05:11:16.870 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=184929] [2023/04/25 05:11:16.870 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=325] [limit=25267] [memorySize=184929] [2023/04/25 05:11:16.872 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.873 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=634439] [2023/04/25 05:11:16.873 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1115] [limit=25267] [memorySize=634439] [2023/04/25 05:11:16.886 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 05:11:16.886 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=198585] [2023/04/25 05:11:16.886 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=349] [limit=25267] [memorySize=198585] [2023/04/25 05:11:16.894 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.896 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=635008] [2023/04/25 05:11:16.896 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1116] [limit=25267] [memorySize=635008] [2023/04/25 05:11:16.937 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.937 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=635577] [2023/04/25 05:11:16.937 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1117] [limit=25267] [memorySize=635577] [2023/04/25 05:11:16.989 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:16.989 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=636146] [2023/04/25 05:11:16.989 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1118] [limit=25267] [memorySize=636146] [2023/04/25 05:11:17.038 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.039 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=636715] [2023/04/25 05:11:17.039 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1119] [limit=25267] [memorySize=636715] [2023/04/25 05:11:17.048 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 05:11:17.048 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=185498] [2023/04/25 05:11:17.048 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=326] [limit=25267] [memorySize=185498] [2023/04/25 05:11:17.096 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.096 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=637284] [2023/04/25 05:11:17.096 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1120] [limit=25267] [memorySize=637284] [2023/04/25 05:11:17.120 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.125 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=637853] [2023/04/25 05:11:17.125 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1121] [limit=25267] [memorySize=637853] [2023/04/25 05:11:17.135 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 05:11:17.135 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=23251] [2023/04/25 05:11:17.135 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=41] [limit=25267] [memorySize=23251] [2023/04/25 05:11:17.148 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.149 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=638422] [2023/04/25 05:11:17.149 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1122] [limit=25267] [memorySize=638422] [2023/04/25 05:11:17.161 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 05:11:17.161 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=186067] [2023/04/25 05:11:17.161 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=327] [limit=25267] [memorySize=186067] [2023/04/25 05:11:17.172 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 05:11:17.173 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=186636] [2023/04/25 05:11:17.173 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=328] [limit=25267] [memorySize=186636] [2023/04/25 05:11:17.224 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.225 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=638991] [2023/04/25 05:11:17.225 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1123] [limit=25267] [memorySize=638991] [2023/04/25 05:11:17.232 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.233 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=639560] [2023/04/25 05:11:17.233 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1124] [limit=25267] [memorySize=639560] [2023/04/25 05:11:17.241 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 05:11:17.241 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=199154] [2023/04/25 05:11:17.241 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=350] [limit=25267] [memorySize=199154] [2023/04/25 05:11:17.248 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.248 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=640129] [2023/04/25 05:11:17.248 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1125] [limit=25267] [memorySize=640129] [2023/04/25 05:11:17.257 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 05:11:17.258 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=199723] [2023/04/25 05:11:17.258 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=351] [limit=25267] [memorySize=199723] [2023/04/25 05:11:17.261 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_24_440468304999444475v1] [cpTs=441023389369106435] [cpTime=2023/04/24 21:11:57.503 +00:00] [2023/04/25 05:11:17.271 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 05:11:17.272 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=640698] [2023/04/25 05:11:17.272 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1126] [limit=25267] [memorySize=640698] [2023/04/25 05:11:17.277 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 05:11:17.277 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=23818] [2023/04/25 05:11:17.277 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=42] [limit=25267] [memorySize=23818]

Anything else?

No response

[2023/04/25 06:17:41.965 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=882] [limit=25267] [memorySize=501862] [2023/04/25 06:17:42.000 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.000 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=502431] [2023/04/25 06:17:42.000 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=883] [limit=25267] [memorySize=502431] [2023/04/25 06:17:42.006 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.006 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=172980] [2023/04/25 06:17:42.006 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=304] [limit=25267] [memorySize=172980] [2023/04/25 06:17:42.017 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.018 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=503569] [2023/04/25 06:17:42.018 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=885] [limit=25267] [memorySize=503569] [2023/04/25 06:17:42.026 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.026 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=504707] [2023/04/25 06:17:42.026 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=887] [limit=25267] [memorySize=504707] [2023/04/25 06:17:42.029 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174231137] [numRows=2] [2023/04/25 06:17:42.032 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174231137] [memorySize=3853809] [2023/04/25 06:17:42.032 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174231137] [channel=by-dev-rootcoord-dml_35_440468305000258989v0] [size=1197] [limit=4833] [memorySize=3853809] [2023/04/25 06:17:42.035 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.035 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=173549] [2023/04/25 06:17:42.035 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=305] [limit=25267] [memorySize=173549] [2023/04/25 06:17:42.039 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174231137] [numRows=1] [2023/04/25 06:17:42.041 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174231137] [memorySize=3857028] [2023/04/25 06:17:42.041 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174231137] [channel=by-dev-rootcoord-dml_35_440468305000258989v0] [size=1198] [limit=4833] [memorySize=3857028] [2023/04/25 06:17:42.054 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174231137] [numRows=1] [2023/04/25 06:17:42.056 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174231137] [memorySize=3860252] [2023/04/25 06:17:42.056 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174231137] [channel=by-dev-rootcoord-dml_35_440468305000258989v0] [size=1199] [limit=4833] [memorySize=3860252] [2023/04/25 06:17:42.071 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.072 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=174118] [2023/04/25 06:17:42.072 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=306] [limit=25267] [memorySize=174118] [2023/04/25 06:17:42.081 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.082 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=505276] [2023/04/25 06:17:42.082 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=888] [limit=25267] [memorySize=505276] [2023/04/25 06:17:42.084 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174231137] [numRows=1] [2023/04/25 06:17:42.086 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174231137] [memorySize=3863470] [2023/04/25 06:17:42.086 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174231137] [channel=by-dev-rootcoord-dml_35_440468305000258989v0] [size=1200] [limit=4833] [memorySize=3863470] [2023/04/25 06:17:42.146 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.146 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=506414] [2023/04/25 06:17:42.146 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=890] [limit=25267] [memorySize=506414] [2023/04/25 06:17:42.146 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:42.146 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=14179] [2023/04/25 06:17:42.146 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=25] [limit=25267] [memorySize=14179] [2023/04/25 06:17:42.185 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.185 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=174687] [2023/04/25 06:17:42.185 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=307] [limit=25267] [memorySize=174687] [2023/04/25 06:17:42.200 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.200 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=506983] [2023/04/25 06:17:42.200 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=891] [limit=25267] [memorySize=506983] [2023/04/25 06:17:42.237 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.237 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=507552] [2023/04/25 06:17:42.237 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=892] [limit=25267] [memorySize=507552] [2023/04/25 06:17:42.251 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.251 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=508690] [2023/04/25 06:17:42.251 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=894] [limit=25267] [memorySize=508690] [2023/04/25 06:17:42.272 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.272 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=509259] [2023/04/25 06:17:42.272 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=895] [limit=25267] [memorySize=509259] [2023/04/25 06:17:42.289 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.290 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=509828] [2023/04/25 06:17:42.290 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=896] [limit=25267] [memorySize=509828] [2023/04/25 06:17:42.302 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.302 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=510397] [2023/04/25 06:17:42.302 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=897] [limit=25267] [memorySize=510397] [2023/04/25 06:17:42.317 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.317 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=510966] [2023/04/25 06:17:42.317 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=898] [limit=25267] [memorySize=510966] [2023/04/25 06:17:42.346 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.347 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=511535] [2023/04/25 06:17:42.347 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=899] [limit=25267] [memorySize=511535] [2023/04/25 06:17:42.381 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.381 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=512104] [2023/04/25 06:17:42.381 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=900] [limit=25267] [memorySize=512104] [2023/04/25 06:17:42.401 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.401 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=512673] [2023/04/25 06:17:42.401 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=901] [limit=25267] [memorySize=512673] [2023/04/25 06:17:42.403 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:42.403 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=139409] [2023/04/25 06:17:42.403 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=245] [limit=25267] [memorySize=139409] [2023/04/25 06:17:42.429 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.429 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=513242] [2023/04/25 06:17:42.429 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=902] [limit=25267] [memorySize=513242] [2023/04/25 06:17:42.433 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:42.433 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=14746] [2023/04/25 06:17:42.433 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=26] [limit=25267] [memorySize=14746] [2023/04/25 06:17:42.447 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:42.447 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=139978] [2023/04/25 06:17:42.447 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=246] [limit=25267] [memorySize=139978] [2023/04/25 06:17:42.447 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.447 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=513811] [2023/04/25 06:17:42.447 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=903] [limit=25267] [memorySize=513811] [2023/04/25 06:17:42.495 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.495 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=514949] [2023/04/25 06:17:42.495 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=905] [limit=25267] [memorySize=514949] [2023/04/25 06:17:42.496 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.496 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=175256] [2023/04/25 06:17:42.496 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=308] [limit=25267] [memorySize=175256] [2023/04/25 06:17:42.507 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:42.508 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=516656] [2023/04/25 06:17:42.508 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=908] [limit=25267] [memorySize=516656] [2023/04/25 06:17:42.523 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.523 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=517225] [2023/04/25 06:17:42.523 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=909] [limit=25267] [memorySize=517225] [2023/04/25 06:17:42.536 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.536 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=175825] [2023/04/25 06:17:42.536 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=309] [limit=25267] [memorySize=175825] [2023/04/25 06:17:42.555 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.555 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=517794] [2023/04/25 06:17:42.555 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=910] [limit=25267] [memorySize=517794] [2023/04/25 06:17:42.593 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:42.594 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=519501] [2023/04/25 06:17:42.594 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=913] [limit=25267] [memorySize=519501] [2023/04/25 06:17:42.642 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.642 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=520070] [2023/04/25 06:17:42.642 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=914] [limit=25267] [memorySize=520070] [2023/04/25 06:17:42.665 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.665 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=176394] [2023/04/25 06:17:42.665 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=310] [limit=25267] [memorySize=176394] [2023/04/25 06:17:42.675 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.675 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=520639] [2023/04/25 06:17:42.675 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=915] [limit=25267] [memorySize=520639] [2023/04/25 06:17:42.689 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:42.689 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=176963] [2023/04/25 06:17:42.689 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=311] [limit=25267] [memorySize=176963] [2023/04/25 06:17:42.702 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.702 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=521208] [2023/04/25 06:17:42.702 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=916] [limit=25267] [memorySize=521208] [2023/04/25 06:17:42.714 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.714 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=521777] [2023/04/25 06:17:42.714 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=917] [limit=25267] [memorySize=521777] [2023/04/25 06:17:42.746 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:42.746 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=140547] [2023/04/25 06:17:42.746 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=247] [limit=25267] [memorySize=140547] [2023/04/25 06:17:42.788 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.788 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=522346] [2023/04/25 06:17:42.788 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=918] [limit=25267] [memorySize=522346] [2023/04/25 06:17:42.806 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.806 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=522915] [2023/04/25 06:17:42.806 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=919] [limit=25267] [memorySize=522915] [2023/04/25 06:17:42.807 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:42.807 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=141116] [2023/04/25 06:17:42.807 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=248] [limit=25267] [memorySize=141116] [2023/04/25 06:17:42.854 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:42.854 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=523484] [2023/04/25 06:17:42.854 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=920] [limit=25267] [memorySize=523484] [2023/04/25 06:17:42.888 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=2] [2023/04/25 06:17:42.888 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=15880] [2023/04/25 06:17:42.888 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=28] [limit=25267] [memorySize=15880] [2023/04/25 06:17:42.912 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:42.913 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=525191] [2023/04/25 06:17:42.913 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=923] [limit=25267] [memorySize=525191] [2023/04/25 06:17:42.936 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.936 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=526329] [2023/04/25 06:17:42.936 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=925] [limit=25267] [memorySize=526329] [2023/04/25 06:17:42.961 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:42.962 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=528036] [2023/04/25 06:17:42.962 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=928] [limit=25267] [memorySize=528036] [2023/04/25 06:17:42.980 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:42.980 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=529174] [2023/04/25 06:17:42.980 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=930] [limit=25267] [memorySize=529174] [2023/04/25 06:17:42.998 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:42.998 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=530881] [2023/04/25 06:17:42.998 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=933] [limit=25267] [memorySize=530881] [2023/04/25 06:17:43.015 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.015 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=531450] [2023/04/25 06:17:43.015 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=934] [limit=25267] [memorySize=531450] [2023/04/25 06:17:43.037 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.037 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=532588] [2023/04/25 06:17:43.037 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=936] [limit=25267] [memorySize=532588] [2023/04/25 06:17:43.048 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.049 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=533157] [2023/04/25 06:17:43.049 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=937] [limit=25267] [memorySize=533157] [2023/04/25 06:17:43.065 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.065 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=533726] [2023/04/25 06:17:43.065 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=938] [limit=25267] [memorySize=533726] [2023/04/25 06:17:43.123 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:43.124 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=535433] [2023/04/25 06:17:43.124 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=941] [limit=25267] [memorySize=535433] [2023/04/25 06:17:43.133 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.133 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=536002] [2023/04/25 06:17:43.133 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=942] [limit=25267] [memorySize=536002] [2023/04/25 06:17:43.143 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.143 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=537140] [2023/04/25 06:17:43.143 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=944] [limit=25267] [memorySize=537140] [2023/04/25 06:17:43.151 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:43.151 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=141685] [2023/04/25 06:17:43.151 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=249] [limit=25267] [memorySize=141685] [2023/04/25 06:17:43.175 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.176 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=538278] [2023/04/25 06:17:43.176 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=946] [limit=25267] [memorySize=538278] [2023/04/25 06:17:43.198 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:43.198 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=142254] [2023/04/25 06:17:43.198 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=250] [limit=25267] [memorySize=142254] [2023/04/25 06:17:43.199 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.199 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=538847] [2023/04/25 06:17:43.199 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=947] [limit=25267] [memorySize=538847] [2023/04/25 06:17:43.232 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.232 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=539416] [2023/04/25 06:17:43.232 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=948] [limit=25267] [memorySize=539416] [2023/04/25 06:17:43.243 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.243 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=539985] [2023/04/25 06:17:43.243 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=949] [limit=25267] [memorySize=539985] [2023/04/25 06:17:43.245 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_44_440468305137570141v1] [cpTs=441024664856952835] [cpTime=2023/04/24 22:33:03.103 +00:00] [2023/04/25 06:17:43.254 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.254 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=541123] [2023/04/25 06:17:43.254 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=951] [limit=25267] [memorySize=541123] [2023/04/25 06:17:43.266 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.267 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=541692] [2023/04/25 06:17:43.267 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=952] [limit=25267] [memorySize=541692] [2023/04/25 06:17:43.270 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_37_440468305000270529v0] [cpTs=441025140228620292] [cpTime=2023/04/24 23:03:16.502 +00:00] [2023/04/25 06:17:43.285 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.286 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=542261] [2023/04/25 06:17:43.286 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=953] [limit=25267] [memorySize=542261] [2023/04/25 06:17:43.307 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.307 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=542830] [2023/04/25 06:17:43.307 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=954] [limit=25267] [memorySize=542830] [2023/04/25 06:17:43.324 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.324 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=543399] [2023/04/25 06:17:43.324 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=955] [limit=25267] [memorySize=543399] [2023/04/25 06:17:43.356 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.356 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=543968] [2023/04/25 06:17:43.356 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=956] [limit=25267] [memorySize=543968] [2023/04/25 06:17:43.401 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:43.401 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=16447] [2023/04/25 06:17:43.401 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=29] [limit=25267] [memorySize=16447] [2023/04/25 06:17:43.406 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.406 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=544537] [2023/04/25 06:17:43.406 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=957] [limit=25267] [memorySize=544537] [2023/04/25 06:17:43.431 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.432 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=545106] [2023/04/25 06:17:43.432 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=958] [limit=25267] [memorySize=545106] [2023/04/25 06:17:43.487 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.487 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=545675] [2023/04/25 06:17:43.487 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=959] [limit=25267] [memorySize=545675] [2023/04/25 06:17:43.507 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.509 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=546244] [2023/04/25 06:17:43.509 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=960] [limit=25267] [memorySize=546244] [2023/04/25 06:17:43.535 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:43.535 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=17014] [2023/04/25 06:17:43.535 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=30] [limit=25267] [memorySize=17014] [2023/04/25 06:17:43.538 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.542 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=546813] [2023/04/25 06:17:43.542 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=961] [limit=25267] [memorySize=546813] [2023/04/25 06:17:43.571 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_38_440468305000270529v1] [cpTs=441024905452716035] [cpTime=2023/04/24 22:48:20.903 +00:00] [2023/04/25 06:17:43.641 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.641 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=547382] [2023/04/25 06:17:43.641 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=962] [limit=25267] [memorySize=547382] [2023/04/25 06:17:43.767 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_23_440468304999444475v0] [cpTs=441024963334111235] [cpTime=2023/04/24 22:52:01.703 +00:00] [2023/04/25 06:17:43.865 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_27_440468304999451676v0] [cpTs=441024871478853638] [cpTime=2023/04/24 22:46:11.303 +00:00] [2023/04/25 06:17:43.868 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:43.868 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=142823] [2023/04/25 06:17:43.868 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=251] [limit=25267] [memorySize=142823] [2023/04/25 06:17:43.872 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.873 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=548520] [2023/04/25 06:17:43.873 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=964] [limit=25267] [memorySize=548520] [2023/04/25 06:17:43.911 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.912 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=549089] [2023/04/25 06:17:43.912 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=965] [limit=25267] [memorySize=549089] [2023/04/25 06:17:43.912 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_21_440468304999444369v0] [cpTs=441025133046136835] [cpTime=2023/04/24 23:02:49.103 +00:00] [2023/04/25 06:17:43.913 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.914 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=549658] [2023/04/25 06:17:43.914 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=966] [limit=25267] [memorySize=549658] [2023/04/25 06:17:43.928 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=2] [2023/04/25 06:17:43.929 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=143961] [2023/04/25 06:17:43.929 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=253] [limit=25267] [memorySize=143961] [2023/04/25 06:17:43.948 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.951 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=550796] [2023/04/25 06:17:43.952 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=968] [limit=25267] [memorySize=550796] [2023/04/25 06:17:43.988 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:43.988 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=551365] [2023/04/25 06:17:43.988 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=969] [limit=25267] [memorySize=551365] [2023/04/25 06:17:43.989 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:43.989 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=144530] [2023/04/25 06:17:43.989 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=254] [limit=25267] [memorySize=144530] [2023/04/25 06:17:43.990 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_41_440468305137569063v0] [cpTs=441024896906821640] [cpTime=2023/04/24 22:47:48.303 +00:00] [2023/04/25 06:17:43.996 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:43.997 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=552503] [2023/04/25 06:17:43.997 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=971] [limit=25267] [memorySize=552503] [2023/04/25 06:17:44.004 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.004 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=145099] [2023/04/25 06:17:44.004 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=255] [limit=25267] [memorySize=145099] [2023/04/25 06:17:44.035 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:44.037 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=554210] [2023/04/25 06:17:44.038 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=974] [limit=25267] [memorySize=554210] [2023/04/25 06:17:44.051 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.051 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=145668] [2023/04/25 06:17:44.051 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=256] [limit=25267] [memorySize=145668] [2023/04/25 06:17:44.086 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.086 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=554779] [2023/04/25 06:17:44.086 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=975] [limit=25267] [memorySize=554779] [2023/04/25 06:17:44.104 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.104 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=555348] [2023/04/25 06:17:44.104 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=976] [limit=25267] [memorySize=555348] [2023/04/25 06:17:44.104 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=2] [2023/04/25 06:17:44.105 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=146806] [2023/04/25 06:17:44.105 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=258] [limit=25267] [memorySize=146806] [2023/04/25 06:17:44.114 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180157409] [numRows=1] [2023/04/25 06:17:44.114 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180157409] [memorySize=12478] [2023/04/25 06:17:44.114 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180157409] [channel=by-dev-rootcoord-dml_40_440468305005730068v1] [size=22] [limit=25267] [memorySize=12478] [2023/04/25 06:17:44.116 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:44.120 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=556486] [2023/04/25 06:17:44.120 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=978] [limit=25267] [memorySize=556486] [2023/04/25 06:17:44.128 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.129 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=557055] [2023/04/25 06:17:44.129 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=979] [limit=25267] [memorySize=557055] [2023/04/25 06:17:44.150 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.150 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=147375] [2023/04/25 06:17:44.150 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=259] [limit=25267] [memorySize=147375] [2023/04/25 06:17:44.167 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.167 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.167 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=147944] [2023/04/25 06:17:44.167 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=260] [limit=25267] [memorySize=147944] [2023/04/25 06:17:44.167 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=557624] [2023/04/25 06:17:44.167 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=980] [limit=25267] [memorySize=557624] [2023/04/25 06:17:44.210 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.210 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=558193] [2023/04/25 06:17:44.211 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=981] [limit=25267] [memorySize=558193] [2023/04/25 06:17:44.241 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:44.241 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=559331] [2023/04/25 06:17:44.241 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=983] [limit=25267] [memorySize=559331] [2023/04/25 06:17:44.257 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.258 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=559900] [2023/04/25 06:17:44.258 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=984] [limit=25267] [memorySize=559900] [2023/04/25 06:17:44.331 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=2] [2023/04/25 06:17:44.332 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=149082] [2023/04/25 06:17:44.332 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=262] [limit=25267] [memorySize=149082] [2023/04/25 06:17:44.341 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.343 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=560469] [2023/04/25 06:17:44.343 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=985] [limit=25267] [memorySize=560469] [2023/04/25 06:17:44.345 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.346 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=149651] [2023/04/25 06:17:44.346 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=263] [limit=25267] [memorySize=149651] [2023/04/25 06:17:44.382 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.382 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=561038] [2023/04/25 06:17:44.382 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=986] [limit=25267] [memorySize=561038] [2023/04/25 06:17:44.413 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.414 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=150220] [2023/04/25 06:17:44.414 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=264] [limit=25267] [memorySize=150220] [2023/04/25 06:17:44.414 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:44.419 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=562176] [2023/04/25 06:17:44.419 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=988] [limit=25267] [memorySize=562176] [2023/04/25 06:17:44.447 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_29_440468304999451768v0] [cpTs=440995957519941637] [cpTime=2023/04/23 16:07:53.303 +00:00] [2023/04/25 06:17:44.474 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.474 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=562745] [2023/04/25 06:17:44.474 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=989] [limit=25267] [memorySize=562745] [2023/04/25 06:17:44.475 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.475 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=150789] [2023/04/25 06:17:44.475 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=265] [limit=25267] [memorySize=150789] [2023/04/25 06:17:44.488 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.489 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=563314] [2023/04/25 06:17:44.489 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=990] [limit=25267] [memorySize=563314] [2023/04/25 06:17:44.512 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:44.512 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=17581] [2023/04/25 06:17:44.512 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=31] [limit=25267] [memorySize=17581] [2023/04/25 06:17:44.516 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.516 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=151358] [2023/04/25 06:17:44.516 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=266] [limit=25267] [memorySize=151358] [2023/04/25 06:17:44.534 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.535 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=563883] [2023/04/25 06:17:44.535 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=991] [limit=25267] [memorySize=563883] [2023/04/25 06:17:44.572 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:44.573 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=565021] [2023/04/25 06:17:44.573 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=993] [limit=25267] [memorySize=565021] [2023/04/25 06:17:44.611 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=3] [2023/04/25 06:17:44.613 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=153065] [2023/04/25 06:17:44.613 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=269] [limit=25267] [memorySize=153065] [2023/04/25 06:17:44.616 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.618 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=565590] [2023/04/25 06:17:44.618 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=994] [limit=25267] [memorySize=565590] [2023/04/25 06:17:44.632 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.633 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=153634] [2023/04/25 06:17:44.633 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=270] [limit=25267] [memorySize=153634] [2023/04/25 06:17:44.650 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_24_440468304999444475v1] [cpTs=441024633347244045] [cpTime=2023/04/24 22:31:02.903 +00:00] [2023/04/25 06:17:44.665 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.666 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=566159] [2023/04/25 06:17:44.666 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=995] [limit=25267] [memorySize=566159] [2023/04/25 06:17:44.675 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.675 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=566728] [2023/04/25 06:17:44.675 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=996] [limit=25267] [memorySize=566728] [2023/04/25 06:17:44.680 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:44.680 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=177532] [2023/04/25 06:17:44.680 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=312] [limit=25267] [memorySize=177532] [2023/04/25 06:17:44.680 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.680 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=154203] [2023/04/25 06:17:44.680 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=271] [limit=25267] [memorySize=154203] [2023/04/25 06:17:44.691 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:44.695 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=567866] [2023/04/25 06:17:44.695 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=998] [limit=25267] [memorySize=567866] [2023/04/25 06:17:44.702 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:44.704 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=569573] [2023/04/25 06:17:44.704 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1001] [limit=25267] [memorySize=569573] [2023/04/25 06:17:44.710 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.710 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=570142] [2023/04/25 06:17:44.710 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1002] [limit=25267] [memorySize=570142] [2023/04/25 06:17:44.723 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.724 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=154772] [2023/04/25 06:17:44.724 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=272] [limit=25267] [memorySize=154772] [2023/04/25 06:17:44.748 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.749 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=155341] [2023/04/25 06:17:44.749 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=273] [limit=25267] [memorySize=155341] [2023/04/25 06:17:44.749 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.750 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=570711] [2023/04/25 06:17:44.750 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1003] [limit=25267] [memorySize=570711] [2023/04/25 06:17:44.760 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.762 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=571280] [2023/04/25 06:17:44.762 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1004] [limit=25267] [memorySize=571280] [2023/04/25 06:17:44.765 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:44.765 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_22_440468304999444369v1] [cpTs=441012883843448849] [cpTime=2023/04/24 10:04:02.103 +00:00] [2023/04/25 06:17:44.766 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=178101] [2023/04/25 06:17:44.766 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=313] [limit=25267] [memorySize=178101] [2023/04/25 06:17:44.773 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_42_440468305137569063v1] [cpTs=441025204821164035] [cpTime=2023/04/24 23:07:22.903 +00:00] [2023/04/25 06:17:44.811 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:44.811 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=178670] [2023/04/25 06:17:44.811 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=314] [limit=25267] [memorySize=178670] [2023/04/25 06:17:44.824 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_28_440468304999451676v1] [cpTs=441024234311647235] [cpTime=2023/04/24 22:05:40.703 +00:00] [2023/04/25 06:17:44.833 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.834 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=571849] [2023/04/25 06:17:44.834 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1005] [limit=25267] [memorySize=571849] [2023/04/25 06:17:44.857 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.857 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=155910] [2023/04/25 06:17:44.857 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=274] [limit=25267] [memorySize=155910] [2023/04/25 06:17:44.865 +00:00] [INFO] [datanode/flow_graph_time_tick_node.go:115] ["UpdateChannelCheckpoint success"] [channel=by-dev-rootcoord-dml_43_440468305137570141v0] [cpTs=441025179602911235] [cpTime=2023/04/24 23:05:46.703 +00:00] [2023/04/25 06:17:44.884 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.884 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=156479] [2023/04/25 06:17:44.884 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=275] [limit=25267] [memorySize=156479] [2023/04/25 06:17:44.888 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:44.888 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=572418] [2023/04/25 06:17:44.888 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1006] [limit=25267] [memorySize=572418] [2023/04/25 06:17:44.890 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:44.890 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=18148] [2023/04/25 06:17:44.890 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=32] [limit=25267] [memorySize=18148] [2023/04/25 06:17:44.894 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:44.894 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=157048] [2023/04/25 06:17:44.894 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=276] [limit=25267] [memorySize=157048] [2023/04/25 06:17:44.951 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=3] [2023/04/25 06:17:44.953 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=574125] [2023/04/25 06:17:44.953 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1009] [limit=25267] [memorySize=574125] [2023/04/25 06:17:44.964 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:44.964 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=179239] [2023/04/25 06:17:44.964 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=315] [limit=25267] [memorySize=179239] [2023/04/25 06:17:44.977 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:44.977 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=18715] [2023/04/25 06:17:44.977 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=33] [limit=25267] [memorySize=18715] [2023/04/25 06:17:45.020 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.021 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=574694] [2023/04/25 06:17:45.021 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1010] [limit=25267] [memorySize=574694] [2023/04/25 06:17:45.073 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.073 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=575263] [2023/04/25 06:17:45.073 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1011] [limit=25267] [memorySize=575263] [2023/04/25 06:17:45.081 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:45.081 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=19282] [2023/04/25 06:17:45.081 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=34] [limit=25267] [memorySize=19282] [2023/04/25 06:17:45.104 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:45.104 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=157617] [2023/04/25 06:17:45.104 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=277] [limit=25267] [memorySize=157617] [2023/04/25 06:17:45.234 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:45.234 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=158186] [2023/04/25 06:17:45.234 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=278] [limit=25267] [memorySize=158186] [2023/04/25 06:17:45.255 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.257 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=575832] [2023/04/25 06:17:45.257 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1012] [limit=25267] [memorySize=575832] [2023/04/25 06:17:45.308 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.309 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=576401] [2023/04/25 06:17:45.309 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1013] [limit=25267] [memorySize=576401] [2023/04/25 06:17:45.314 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:45.315 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=179808] [2023/04/25 06:17:45.315 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=316] [limit=25267] [memorySize=179808] [2023/04/25 06:17:45.352 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.352 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=576970] [2023/04/25 06:17:45.353 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1014] [limit=25267] [memorySize=576970] [2023/04/25 06:17:45.369 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.369 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=577539] [2023/04/25 06:17:45.369 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1015] [limit=25267] [memorySize=577539] [2023/04/25 06:17:45.372 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:45.372 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=19849] [2023/04/25 06:17:45.372 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=35] [limit=25267] [memorySize=19849] [2023/04/25 06:17:45.372 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=2] [2023/04/25 06:17:45.373 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=578677] [2023/04/25 06:17:45.373 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1017] [limit=25267] [memorySize=578677] [2023/04/25 06:17:45.389 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.390 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=579246] [2023/04/25 06:17:45.390 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1018] [limit=25267] [memorySize=579246] [2023/04/25 06:17:45.399 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187146587] [numRows=1] [2023/04/25 06:17:45.399 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187146587] [memorySize=158755] [2023/04/25 06:17:45.399 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187146587] [channel=by-dev-rootcoord-dml_25_440468304999444555v0] [size=279] [limit=25267] [memorySize=158755] [2023/04/25 06:17:45.402 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180157409] [numRows=1] [2023/04/25 06:17:45.402 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180157409] [memorySize=13045] [2023/04/25 06:17:45.402 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180157409] [channel=by-dev-rootcoord-dml_40_440468305005730068v1] [size=23] [limit=25267] [memorySize=13045] [2023/04/25 06:17:45.434 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=2] [2023/04/25 06:17:45.434 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=20983] [2023/04/25 06:17:45.434 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=37] [limit=25267] [memorySize=20983] [2023/04/25 06:17:45.439 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.440 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=579815] [2023/04/25 06:17:45.440 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1019] [limit=25267] [memorySize=579815] [2023/04/25 06:17:45.443 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305180166857] [numRows=1] [2023/04/25 06:17:45.443 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305180166857] [memorySize=21550] [2023/04/25 06:17:45.443 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305180166857] [channel=by-dev-rootcoord-dml_39_440468305005730068v0] [size=38] [limit=25267] [memorySize=21550] [2023/04/25 06:17:45.443 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305187147298] [numRows=1] [2023/04/25 06:17:45.443 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305187147298] [memorySize=180377] [2023/04/25 06:17:45.443 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305187147298] [channel=by-dev-rootcoord-dml_26_440468304999444555v1] [size=317] [limit=25267] [memorySize=180377] [2023/04/25 06:17:45.472 +00:00] [INFO] [datanode/channel_meta.go:476] ["updating segment num row"] ["Segment ID"=440468305174447198] [numRows=1] [2023/04/25 06:17:45.473 +00:00] [INFO] [datanode/channel_meta.go:491] ["updating segment memorySize"] ["Segment ID"=440468305174447198] [memorySize=580384] [2023/04/25 06:17:45.473 +00:00] [INFO] [datanode/flow_graph_insert_buffer_node.go:288] ["segment buffer status"] [no.=0] [segmentID=440468305174447198] [channel=by-dev-rootcoord-dml_20_440468304999444264v1] [size=1020] [limit=25267] [memorySize=580384]

logs.tar.gz

please help me check it

@qilaijun101 it seems that pulsar or etcd is not work well, as there are many 'tt send failures' in the logs: [2023/04/25 06:33:44.870 +00:00] [ERROR] [datanode/flow_graph_time_ticker.go:130] ["send hard time tick failed"] [error="message send timeout: TimeoutError"] [stack="github.com/milvus-io/milvus/internal/datanode.(*mergedTimeTickerSender).work\n\t/go/src/github.com/milvus-io/milvus/internal/datanode/flow_graph_time_ticker.go:130"] could you please double check the etcd and pulsar service? moreover, what requests are you running when the datanode restarted

/assign @qilaijun101

etcd node

etcd log

pulsar service

pulsar log

07:33:52.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:33:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:00.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:02.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:10.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:12.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:20.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:22.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:30.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:32.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:34:40.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:42.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:50.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:34:52.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:34:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:00.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:35:02.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:10.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:12.548 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:35:20.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:35:22.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:35:30.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:32.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:40.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:35:42.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:50.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:35:52.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:35:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:00.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:02.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:10.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:12.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:36:20.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:22.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:30.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:32.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:40.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:42.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:50.104 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:36:52.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:36:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:00.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:02.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:37:10.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:12.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:20.104 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:37:22.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:37:30.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:32.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:40.104 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:37:42.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:50.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:37:52.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:37:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:00.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:02.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:38:10.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:12.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:38:20.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:22.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:38:30.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:32.548 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:38:40.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:42.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:50.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:38:52.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:38:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:38:53.949 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40634] New connection opened 07:38:53.951 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40634] complete connection, init proxy handler. authenticated with none role null, hasProxyToBrokerUrl: false 07:38:53.953 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xb025b832, L:/10.23.241.30:47478 - R:milvus-pulsar-broker.ns-milvus-prod.svc.cluster.local/10.23.243.50:6650]] Connected to server 07:38:53.958 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x5b8a3171, L:/10.23.241.30:47480 - R:milvus-pulsar-broker.ns-milvus-prod.svc.cluster.local/10.23.243.50:6650]] Connected to server 07:38:53.960 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40654] New connection opened 07:38:53.960 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40654] complete connection, init proxy handler. authenticated with none role null, hasProxyToBrokerUrl: true 07:39:00.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:02.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:10.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:12.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:20.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:22.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:30.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:32.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:40.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:39:42.548 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:39:50.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:39:52.548 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:39:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:00.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:02.548 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:10.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:40:12.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:40:20.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:22.548 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:30.104 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:32.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:40.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:40:42.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:40:50.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:40:50.543 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_23/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_23_440468304999444475v0?force=true HTTP/1.1" 204 0 "-" "None" 5 07:40:50.548 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_22/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_22_440468304999444369v1?force=true HTTP/1.1" 204 0 "-" "None" 4 07:40:50.554 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_0/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_0_440468304997496608v0?force=true HTTP/1.1" 204 0 "-" "None" 5 07:40:50.561 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_24/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_24_440468304999444475v1?force=true HTTP/1.1" 204 0 "-" "None" 6 07:40:50.566 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_19/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_19_440468304999444264v0?force=true HTTP/1.1" 204 0 "-" "None" 4 07:40:50.571 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_20/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_20_440468304999444264v1?force=true HTTP/1.1" 204 0 "-" "None" 4 07:40:50.579 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_21/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_21_440468304999444369v0?force=true HTTP/1.1" 204 0 "-" "None" 7 07:40:50.582 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_25/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_25_440468304999444555v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:40:50.588 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_26/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_26_440468304999444555v1?force=true HTTP/1.1" 204 0 "-" "None" 5 07:40:50.593 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_27/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_27_440468304999451676v0?force=true HTTP/1.1" 204 0 "-" "None" 5 07:40:50.598 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_28/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_28_440468304999451676v1?force=true HTTP/1.1" 204 0 "-" "None" 4 07:40:50.601 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_29/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_29_440468304999451768v0?force=true HTTP/1.1" 404 35 "-" "None" 2 07:40:50.607 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_30/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_30_440468304999451768v1?force=true HTTP/1.1" 204 0 "-" "None" 6 07:40:50.610 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_35/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_35_440468305000258989v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:40:50.619 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_36/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_36_440468305000258989v1?force=true HTTP/1.1" 204 0 "-" "None" 8 07:40:50.628 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_37/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_37_440468305000270529v0?force=true HTTP/1.1" 204 0 "-" "None" 8 07:40:50.632 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_38/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_38_440468305000270529v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:40:50.638 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_39/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_39_440468305005730068v0?force=true HTTP/1.1" 204 0 "-" "None" 5 07:40:50.641 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_40/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_40_440468305005730068v1?force=true HTTP/1.1" 204 0 "-" "None" 2 07:40:50.646 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_41/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_41_440468305137569063v0?force=true HTTP/1.1" 204 0 "-" "None" 4 07:40:50.651 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_42/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_42_440468305137569063v1?force=true HTTP/1.1" 204 0 "-" "None" 4 07:40:50.655 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_43/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_43_440468305137570141v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:40:50.659 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:40:50 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_44/subscription/by-dev-dataNode-486-by-dev-rootcoord-dml_44_440468305137570141v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:40:52.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:40:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:00.103 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:02.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:41:10.103 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:12.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:41:20.104 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:41:22.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:41:23.451 [pulsar-proxy-io-2-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer 07:41:23.451 [pulsar-proxy-io-2-1] WARN org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40634] Got exception NativeIoException : readAddress(..) failed: Connection reset by peer null 07:41:23.452 [pulsar-proxy-io-2-1] WARN io.netty.channel.DefaultChannelPipeline - An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer 07:41:23.452 [pulsar-proxy-io-2-1] WARN org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40654] Got exception NativeIoException : readAddress(..) failed: Connection reset by peer null 07:41:23.452 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://milvus-pulsar-proxy-0.milvus-pulsar-proxy.ns-milvus-prod.svc.cluster.local:6650/ 07:41:23.452 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40634] Connection closed 07:41:23.452 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnectionPool - Closing ProxyConnectionPool. 07:41:23.452 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://milvus-pulsar-proxy-0.milvus-pulsar-proxy.ns-milvus-prod.svc.cluster.local:6650/ 07:41:23.453 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40654] Connection closed 07:41:23.453 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnectionPool - Closing ProxyConnectionPool. 07:41:23.453 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0x5b8a3171, L:/10.23.241.30:47480 ! R:milvus-pulsar-broker.ns-milvus-prod.svc.cluster.local/10.23.243.50:6650] Disconnected 07:41:23.453 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xb025b832, L:/10.23.241.30:47478 ! R:milvus-pulsar-broker.ns-milvus-prod.svc.cluster.local/10.23.243.50:6650] Disconnected 07:41:25.323 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40736] New connection opened 07:41:25.324 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40736] complete connection, init proxy handler. authenticated with none role null, hasProxyToBrokerUrl: false 07:41:25.328 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xbf169ff2, L:/10.23.241.30:47566 - R:milvus-pulsar-broker.ns-milvus-prod.svc.cluster.local/10.23.243.50:6650]] Connected to server 07:41:25.330 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xa59fec27, L:/10.23.241.30:47568 - R:milvus-pulsar-broker.ns-milvus-prod.svc.cluster.local/10.23.243.50:6650]] Connected to server 07:41:25.332 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40742] New connection opened 07:41:25.332 [pulsar-proxy-io-2-1] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/10.23.243.41:40742] complete connection, init proxy handler. authenticated with none role null, hasProxyToBrokerUrl: true 07:41:30.104 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:41:32.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:40.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:42.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:50.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:41:52.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:41:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:42:00.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:02.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:10.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:12.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:20.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:22.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:30.104 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:42:32.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:40.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:42.547 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:50.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:42:52.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:42:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:00.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:02.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:10.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:12.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:20.103 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:22.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:43:25.218 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_23/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_23_440468304999444475v0?force=true HTTP/1.1" 204 0 "-" "None" 6 07:43:25.224 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_22/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_22_440468304999444369v1?force=true HTTP/1.1" 204 0 "-" "None" 5 07:43:25.228 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_0/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_0_440468304997496608v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.232 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_24/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_24_440468304999444475v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.236 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_19/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_19_440468304999444264v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.247 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_20/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_20_440468304999444264v1?force=true HTTP/1.1" 204 0 "-" "None" 10 07:43:25.251 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_21/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_21_440468304999444369v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.254 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_25/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_25_440468304999444555v0?force=true HTTP/1.1" 204 0 "-" "None" 2 07:43:25.258 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_26/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_26_440468304999444555v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.262 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_27/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_27_440468304999451676v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.267 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_28/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_28_440468304999451676v1?force=true HTTP/1.1" 204 0 "-" "None" 4 07:43:25.271 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_29/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_29_440468304999451768v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.275 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_30/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_30_440468304999451768v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.279 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_35/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_35_440468305000258989v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.283 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_36/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_36_440468305000258989v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.292 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_37/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_37_440468305000270529v0?force=true HTTP/1.1" 204 0 "-" "None" 6 07:43:25.296 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_38/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_38_440468305000270529v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.299 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_39/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_39_440468305005730068v0?force=true HTTP/1.1" 204 0 "-" "None" 2 07:43:25.303 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_40/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_40_440468305005730068v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.307 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_41/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_41_440468305137569063v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.312 [pulsar-external-web-6-43] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_42/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_42_440468305137569063v1?force=true HTTP/1.1" 204 0 "-" "None" 4 07:43:25.316 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_43/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_43_440468305137570141v0?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:25.319 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.126 - - [25/Apr/2023:07:43:25 +0000] "DELETE /admin/v2/persistent/public/default/by-dev-rootcoord-dml_44/subscription/by-dev-dataNode-487-by-dev-rootcoord-dml_44_440468305137570141v1?force=true HTTP/1.1" 204 0 "-" "None" 3 07:43:30.103 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:32.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:32 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:40.104 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:40 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:43:42.547 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:42 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:50.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:50 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:43:52.547 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:43:52 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:44:00.103 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:00 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:44:02.547 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:02 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:44:10.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:10 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:44:12.547 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:12 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 1 07:44:20.103 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:20 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:44:22.547 [pulsar-external-web-6-83] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:22 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0 07:44:30.104 [pulsar-external-web-6-50] INFO org.eclipse.jetty.server.RequestLog - 10.23.241.6 - - [25/Apr/2023:07:44:30 +0000] "GET /status.html HTTP/1.1" 200 2 "-" "kube-probe/." 0

/assign @LoveEachDay could you please take a look?

/unassign

/assign @LoveEachDay could you please take a look?

/unassign

@LoveEachDay please help me check it ,thank you

@qilaijun101

The error message shows the timetick lag is too large. Could you provide a screenshot for rootcoord metrics dashboard?

@qilaijun101 The error message shows the timetick lag is too large. Could you provide a screenshot for rootcoord metrics dashboard? please check it

@qilaijun101 The Time Tick Delay is so large.

@qilaijun101 Could you give us a birdwatcher backup?

backup

how can i do it ?????, My local computer has birdwatcher

backup

how can i do it ?????, My local computer has birdwatcher

can i to connect the metd?

bw_etcd_ALL.230425-162308.bak.gz please check it @LoveEachDay

bw_etcd_ALL.230425-162308.bak.gz please check it @LoveEachDay

@LoveEachDay please check it ,thanks

@qilaijun101 Ok.

@LoveEachDay What is the current progres,need more logs from me ?

@qilaijun101 Could you also upload milvus logs at here?

And also could you double check the this config:

dataCoord:
   channel:
      watchTimeoutInterval: 120

Try to increase the watchTimeoutInterval to 600 .

channel @LoveEachDay The cluster milvus had been installed by helm, I can not find the config with dataCoord, could you please tell me which config has the dataCoord info ?

and the milvus-datacoord file

apiVersion: v1 kind: Service metadata: annotations: meta.helm.sh/release-name: milvus meta.helm.sh/release-namespace: ns-milvus-prod creationTimestamp: "2023-03-31T08:45:20Z" labels: app.kubernetes.io/instance: milvus app.kubernetes.io/managed-by: Helm app.kubernetes.io/name: milvus app.kubernetes.io/version: 2.2.5 component: datacoord helm.sh/chart: milvus-4.0.14 managedFields:

apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:meta.helm.sh/release-name: {} f:meta.helm.sh/release-namespace: {} f:labels: .: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/managed-by: {} f:app.kubernetes.io/name: {} f:app.kubernetes.io/version: {} f:component: {} f:helm.sh/chart: {} f:spec: f:ports: .: {} k:{"port":9091,"protocol":"TCP"}: .: {} f:name: {} f:port: {} f:protocol: {} f:targetPort: {} k:{"port":13333,"protocol":"TCP"}: .: {} f:name: {} f:port: {} f:protocol: {} f:targetPort: {} f:selector: .: {} f:app.kubernetes.io/instance: {} f:app.kubernetes.io/name: {} f:component: {} f:sessionAffinity: {} f:type: {} manager: helm operation: Update time: "2023-03-31T08:45:20Z" name: milvus-datacoord namespace: ns-milvus-prod resourceVersion: "18738505274" selfLink: /api/v1/namespaces/ns-milvus-prod/services/milvus-datacoord uid: 15e9ccd5-5032-4d28-8950-388fd17dc899 spec: clusterIP: 172.17.0.206 clusterIPs:
172.17.0.206 ports:
name: datacoord port: 13333 protocol: TCP targetPort: datacoord
name: metrics port: 9091 protocol: TCP targetPort: metrics selector: app.kubernetes.io/instance: milvus app.kubernetes.io/name: milvus component: datacoord sessionAffinity: None type: ClusterIP status: loadBalancer: {}

@qilaijun101 You can login into a milvus pod, checkout the /milvus/configs/milvus.yaml.

watchTimeoutInterval

please check it , and how can i edit ?

Which milvus-chart version do you use? You can specify the configs in values.yaml at the following section:

extraConfigFiles:
   user.yaml: |+
     dataCoord:
       channel:
          watchTimeoutInterval: 600

Then helm upgrade your instance.

Which milvus-chart version do you use? You can specify the configs in values.yaml at the following section:
extraConfigFiles:
   user.yaml: |+
     dataCoord:
       channel:
          watchTimeoutInterval: 600
Then helm upgrade your instance.

helm.sh/chart: milvus-4.0.14

user.yaml @LoveEachDay

could you give me the helm command with the user.yaml to upgrad the milvus?

user.yaml @LoveEachDay

could you give me the helm command with the user.yaml to upgrad the milvus?

You can refer this doc: https://mp.weixin.qq.com/s/FQ8vvbyfQ9wjbE-mrGP42w

bw_etcd_ALL.230426-112607.bak.gz @LoveEachDay @JackLCL 我这边使用此命令进行升级:helm upgrade -f milvus.yml milvus milvus/milvus --version 4.0.14 -n ns-milvus-prod

发现并没有生效,且大部分服务无法正常启动,请协助看一下

@LoveEachDay The rootcoord will not run ,please check the log

2023/04/26 03:23:38 maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined

__  _________ _   ____  ______

/ |/ / _/ /| | / / / / / /
/ /|/ // // /| |/ / // /\ \
// ///__/_/__/_/

Welcome to use Milvus! Version: v2.2.5 Built: Wed Mar 29 11:32:15 UTC 2023 GitCommit: 47e28fbe GoVersion: go version go1.18.3 linux/amd64

open pid file: /run/milvus/rootcoord.pid lock pid file: /run/milvus/rootcoord.pid [2023/04/26 03:23:38.411 +00:00] [INFO] [roles/roles.go:192] ["starting running Milvus components"] [2023/04/26 03:23:38.411 +00:00] [INFO] [roles/roles.go:134] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so] [2023/04/26 03:23:38.411 +00:00] [INFO] [management/server.go:68] ["management listen"] [addr=:9091] [2023/04/26 03:23:38.420 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.420 +00:00] [INFO] [paramtable/quota_param.go:532] ["init disk quota"] [diskQuota(MB)=+inf] [2023/04/26 03:23:38.421 +00:00] [INFO] [paramtable/component_param.go:1402] ["init segment max idle time"] [value=10m0s] [2023/04/26 03:23:38.421 +00:00] [INFO] [paramtable/component_param.go:1407] ["init segment min size from idle to sealed"] [value=16] [2023/04/26 03:23:38.421 +00:00] [INFO] [paramtable/component_param.go:1412] ["init segment max binlog file to sealed"] [value=32] [2023/04/26 03:23:38.422 +00:00] [INFO] [paramtable/base_table.go:142] ["cannot find etcd.endpoints"] [2023/04/26 03:23:38.422 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] [2023/04/26 03:23:38.422 +00:00] [INFO] [rootcoord/root_coord.go:175] ["update rootcoord state"] [state=Abnormal] [2023/04/26 03:23:38.435 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.435 +00:00] [DEBUG] [paramtable/grpc_param.go:153] [initServerMaxSendSize] [role=rootcoord] [grpc.serverMaxSendSize=536870912] [2023/04/26 03:23:38.435 +00:00] [DEBUG] [paramtable/grpc_param.go:175] [initServerMaxRecvSize] [role=rootcoord] [grpc.serverMaxRecvSize=536870912] [2023/04/26 03:23:38.435 +00:00] [INFO] [rootcoord/service.go:160] ["init params done.."] [2023/04/26 03:23:38.436 +00:00] [INFO] [rootcoord/service.go:179] ["etcd connect done ..."] [2023/04/26 03:23:38.436 +00:00] [INFO] [rootcoord/service.go:234] ["start grpc "] [port=53100] [2023/04/26 03:23:38.536 +00:00] [INFO] [rootcoord/service.go:185] ["grpc init done ..."] [2023/04/26 03:23:38.536 +00:00] [INFO] [rootcoord/service.go:188] ["RootCoord start to create DataCoord client"] [2023/04/26 03:23:38.536 +00:00] [INFO] [sessionutil/session_util.go:197] ["Session try to connect to etcd"] [2023/04/26 03:23:38.537 +00:00] [INFO] [sessionutil/session_util.go:212] ["Session connect to etcd success"] [2023/04/26 03:23:38.544 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max send size"] [role=datacoord] [grpc.clientMaxSendSize=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max recv size"] [role=datacoord] [grpc.clientMaxRecvSize=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init dial timeout"] [role=datacoord] [grpc.client.dialTimeout=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive timeout"] [role=datacoord] [grpc.client.keepAliveTimeout=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive time"] [role=datacoord] [grpc.client.keepAliveTime=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max attempts"] [role=datacoord] [grpc.client.maxMaxAttempts=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init initial back off"] [role=datacoord] [grpc.client.initialBackOff=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max back off"] [role=datacoord] [grpc.client.maxBackoff=104857600] [2023/04/26 03:23:38.545 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init back off multiplier"] [role=datacoord] [grpc.client.backoffMultiplier=104857600] [2023/04/26 03:23:38.545 +00:00] [INFO] [rootcoord/service.go:196] ["RootCoord start to create IndexCoord client"] [2023/04/26 03:23:38.545 +00:00] [INFO] [sessionutil/session_util.go:197] ["Session try to connect to etcd"] [2023/04/26 03:23:38.546 +00:00] [INFO] [sessionutil/session_util.go:212] ["Session connect to etcd success"] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max send size"] [role=indexcoord] [grpc.clientMaxSendSize=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max recv size"] [role=indexcoord] [grpc.clientMaxRecvSize=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init dial timeout"] [role=indexcoord] [grpc.client.dialTimeout=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive timeout"] [role=indexcoord] [grpc.client.keepAliveTimeout=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive time"] [role=indexcoord] [grpc.client.keepAliveTime=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max attempts"] [role=indexcoord] [grpc.client.maxMaxAttempts=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init initial back off"] [role=indexcoord] [grpc.client.initialBackOff=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max back off"] [role=indexcoord] [grpc.client.maxBackoff=104857600] [2023/04/26 03:23:38.555 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init back off multiplier"] [role=indexcoord] [grpc.client.backoffMultiplier=104857600] [2023/04/26 03:23:38.555 +00:00] [INFO] [rootcoord/service.go:204] ["RootCoord start to create QueryCoord client"] [2023/04/26 03:23:38.555 +00:00] [INFO] [sessionutil/session_util.go:197] ["Session try to connect to etcd"] [2023/04/26 03:23:38.555 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.556 +00:00] [INFO] [sessionutil/session_util.go:212] ["Session connect to etcd success"] [2023/04/26 03:23:38.568 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max send size"] [role=querycoord] [grpc.clientMaxSendSize=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max recv size"] [role=querycoord] [grpc.clientMaxRecvSize=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init dial timeout"] [role=querycoord] [grpc.client.dialTimeout=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive timeout"] [role=querycoord] [grpc.client.keepAliveTimeout=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive time"] [role=querycoord] [grpc.client.keepAliveTime=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max attempts"] [role=querycoord] [grpc.client.maxMaxAttempts=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init initial back off"] [role=querycoord] [grpc.client.initialBackOff=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max back off"] [role=querycoord] [grpc.client.maxBackoff=104857600] [2023/04/26 03:23:38.568 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init back off multiplier"] [role=querycoord] [grpc.client.backoffMultiplier=104857600] [2023/04/26 03:23:38.568 +00:00] [INFO] [sessionutil/session_util.go:197] ["Session try to connect to etcd"] [2023/04/26 03:23:38.569 +00:00] [INFO] [sessionutil/session_util.go:212] ["Session connect to etcd success"] [2023/04/26 03:23:38.576 +00:00] [INFO] [sessionutil/session_util.go:295] ["Session get serverID success"] [key=id] [ServerId=705] [2023/04/26 03:23:38.583 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.583 +00:00] [INFO] [paramtable/quota_param.go:532] ["init disk quota"] [diskQuota(MB)=+inf] [2023/04/26 03:23:38.583 +00:00] [INFO] [paramtable/component_param.go:1402] ["init segment max idle time"] [value=10m0s] [2023/04/26 03:23:38.583 +00:00] [INFO] [paramtable/component_param.go:1407] ["init segment min size from idle to sealed"] [value=16] [2023/04/26 03:23:38.583 +00:00] [INFO] [paramtable/component_param.go:1412] ["init segment max binlog file to sealed"] [value=32] [2023/04/26 03:23:38.584 +00:00] [INFO] [paramtable/base_table.go:142] ["cannot find etcd.endpoints"] [2023/04/26 03:23:38.584 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] [2023/04/26 03:23:38.584 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2023/04/26 03:23:38.584 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=] [2023/04/26 03:23:38.584 +00:00] [INFO] [rootcoord/root_coord.go:175] ["update rootcoord state"] [state=Initializing] [2023/04/26 03:23:38.592 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/04/26 03:23:38.592 +00:00] [INFO] [paramtable/quota_param.go:532] ["init disk quota"] [diskQuota(MB)=+inf] [2023/04/26 03:23:38.592 +00:00] [INFO] [paramtable/component_param.go:1402] ["init segment max idle time"] [value=10m0s] [2023/04/26 03:23:38.592 +00:00] [INFO] [paramtable/component_param.go:1407] ["init segment min size from idle to sealed"] [value=16] [2023/04/26 03:23:38.592 +00:00] [INFO] [paramtable/component_param.go:1412] ["init segment max binlog file to sealed"] [value=32] [2023/04/26 03:23:38.593 +00:00] [INFO] [paramtable/base_table.go:142] ["cannot find etcd.endpoints"] [2023/04/26 03:23:38.593 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] [2023/04/26 03:23:38.614 +00:00] [INFO] [tso/tso.go:121] ["sync and save timestamp"] [last=2023/04/26 03:20:11.605 +00:00] [save=2023/04/26 03:23:41.613 +00:00] [next=2023/04/26 03:23:38.613 +00:00] [2023/04/26 03:23:38.617 +00:00] [INFO] [tso/tso.go:121] ["sync and save timestamp"] [last=2023/04/26 03:20:11.611 +00:00] [save=2023/04/26 03:23:41.615 +00:00] [next=2023/04/26 03:23:38.615 +00:00] time="2023-04-26T03:23:38Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:38Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:38Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:38Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:39Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:39Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:40Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:40Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:41Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:41Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:45Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:45Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:52Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:23:52Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:07Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:07Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" [2023/04/26 03:24:34.404 +00:00] [WARN] [retry/retry.go:44] ["retry func failed"] ["retry time"=0] [error="connection error"] time="2023-04-26T03:24:34Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:34Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:34Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:34Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:35Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:35Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:36Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:36Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:38Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:38Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:41Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:41Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:47Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:24:47Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:01Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:01Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:30Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:30Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:30Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:30Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:31Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:31Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:32Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:32Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:33Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:33Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:37Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:37Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:43Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:43Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:56Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:25:56Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:28Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:28Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:28Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:28Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:28Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:28Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:29Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:29Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:31Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:31Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:34Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:34Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:42Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:42Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:56Z" level=info msg="[Connecting to broker]" remote_addr="pulsar://milvus-pulsar-proxy:6650" time="2023-04-26T03:26:56Z" level=warning msg="[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650" [2023/04/26 03:27:08.191 +00:00] [ERROR] [roles/roles.go:318] ["Get signal to exit\n"] [signal=terminated] [stack="github.com/milvus-io/milvus/cmd/roles.(MilvusRoles).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:318\ngithub.com/milvus-io/milvus/cmd/milvus.(run).execute\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:112\ngithub.com/milvus-io/milvus/cmd/milvus.RunMilvus\n\t/go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60\nmain.main\n\t/go/src/github.com/milvus-io/milvus/cmd/main.go:26\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/service.go:281] ["Rootcoord stop"] [Address=10.23.241.238:53100] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/root_coord.go:175] ["update rootcoord state"] [state=Abnormal] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/root_coord.go:696] ["stop rootcoord scheduler"] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/root_coord.go:703] ["cancel rootcoord goroutines"] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/root_coord.go:711] ["revoke rootcoord session"] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/service.go:310] ["Rootcoord begin to stop grpc server"] [2023/04/26 03:27:08.191 +00:00] [INFO] [rootcoord/service.go:313] ["Graceful stop grpc server..."]

logs0426.tar.gz

get the log by export-milvus-log.sh, please help me check it

@LoveEachDay

"[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650"

@qilaijun101 Could you double check that pulsar service is running well?

"[Failed to connect to broker.]" error="dial tcp: lookup milvus-pulsar-proxy on 172.17.16.193:53: no such host" remote_addr="pulsar://milvus-pulsar-proxy:6650"
@qilaijun101 Could you double check that pulsar service is running well?

I can not find pulsar service? why ?

@LoveEachDay
helm upgrade -f milvus.yml milvus milvus/milvus --version 4.0.14 -n ns-milvus-prod

and the milvus.yml content

common: DiskIndex: BeamWidthRatio: 4 BuildNumThreadsRatio: 1 LoadNumThreadRatio: 8 MaxDegree: 56 PQCodeBudgetGBRatio: 0.125 SearchCacheBudgetGBRatio: 0.1 SearchListSize: 100 chanNamePrefix: cluster: by-dev cmd: cmd dataCoordSegmentInfo: segment-info-channel dataCoordStatistic: datacoord-statistics-channel dataCoordTimeTick: datacoord-timetick-channel queryNodeStats: query-node-stats queryTimeTick: queryTimeTick rootCoordDelta: rootcoord-delta rootCoordDml: rootcoord-dml rootCoordStatistics: rootcoord-statistics rootCoordTimeTick: rootcoord-timetick search: search searchResult: searchResult defaultIndexName: _default_idx defaultPartitionName: _default entityExpiration: -1 gracefulStopTimeout: 30 gracefulTime: 5000 indexSliceSize: 16 retentionDuration: 0 security: authorizationEnabled: false tlsMode: 0 session: retryTimes: 30 ttl: 20 simdType: auto storageType: minio subNamePrefix: dataCoordSubNamePrefix: dataCoord dataNodeSubNamePrefix: dataNode proxySubNamePrefix: proxy queryNodeSubNamePrefix: queryNode rootCoordSubNamePrefix: rootCoord threadCoreCoefficient: 10 dataCoord: address: milvus-datacoord channel: balanceInterval: 360 balanceSilentDuration: 300 watchTimeoutInterval: 600 compaction: enableAutoCompaction: true enableActiveStandby: false enableCompaction: true enableGarbageCollection: true gc: dropTolerance: 3600 interval: 3600 missingTolerance: 86400 port: 13333 segment: assignmentExpiration: 2000 compactableProportion: 0.85 diskSegmentMaxSize: 2048 maxBinlogFileNumber: 32 maxIdleTime: 600 maxLife: 86400 maxSize: 512 minSizeFromIdleToSealed: 16 sealProportion: 0.23 smallProportion: 0.5 dataNode: dataSync: flowGraph: maxParallelism: 1024 maxQueueLength: 1024 memory: forceSyncEnable: false forceSyncSegmentNum: 1 watermarkCluster: 0.5 watermarkStandalone: 0.2 port: 21124 segment: deleteBufBytes: 67108864 insertBufSize: 16777216 syncPeriod: 600 etcd: data: dir: default.etcd endpoints:

milvus-etcd:2379 kvSubPath: kv log: level: info path: stdout metaSubPath: meta rootPath: by-dev ssl: enabled: false tlsCACert: /path/to/ca.pem tlsCert: /path/to/etcd-client.pem tlsKey: /path/to/etcd-client-key.pem tlsMinVersion: 1.3 use: embed: false grpc: client: backoffMultiplier: 2 dialTimeout: 5000 initialBackOff: 1 keepAliveTime: 10000 keepAliveTimeout: 20000 maxBackoff: 60 maxMaxAttempts: 5 clientMaxRecvSize: 104857600 clientMaxSendSize: 104857600 log: level: WARNING server: retryTimes: 5 serverMaxRecvSize: 536870912 serverMaxSendSize: 536870912 indexCoord: address: milvus-indexcoord bindIndexNodeMode: address: localhost:22930 enable: false nodeID: 0 withCred: false enableActiveStandby: false gc: interval: 600 minSegmentNumRowsToEnableIndex: 1024 port: 31000 scheduler: interval: 1000 indexNode: enableDisk: true gracefulStopTimeout: 30 maxDiskUsagePercentage: 95 port: 21121 scheduler: buildParallel: 1 localStorage: path: /var/lib/milvus/data/ log: file: maxAge: 10 maxBackups: 20 maxSize: 300 rootPath: "" format: text level: info stdout: "true" messageQueue: pulsar metastore: type: etcd minio: accessKeyID: minioadmin address: milvus-minio bucketName: milvus-bucket cloudProvider: aws iamEndpoint: null port: 9000 rootPath: file secretAccessKey: minioadmin useIAM: false useSSL: false mysql: address: localhost dbName: milvus_meta driverName: mysql maxIdleConns: 5 maxOpenConns: 20 password: 123456 port: 3306 username: root proxy: ginLogging: true grpc: clientMaxRecvSize: 104857600 clientMaxSendSize: 104857600 serverMaxRecvSize: 67108864 serverMaxSendSize: 67108864 http: debug_mode: false enabled: true internalPort: 19529 maxDimension: 32768 maxFieldNum: 64 maxNameLength: 255 maxShardNum: 64 maxTaskNum: 1024 msgStream: timeTick: bufSize: 512 port: 19530 timeTickInterval: 200 pulsar: address: milvus-pulsar-proxy maxMessageSize: 5242880 namespace: default port: 6650 tenant: public webport: 80 queryCoord: address: milvus-querycoord autoBalance: true autoHandoff: true balanceIntervalSeconds: 60 channelTaskTimeout: 60000 checkHandoffInterval: 5000 checkInterval: 1000 distPullInterval: 500 enableActiveStandby: false loadTimeoutSeconds: 1800 memoryUsageMaxDifferencePercentage: 30 overloadedMemoryThresholdPercentage: 90 port: 19531 refreshTargetsIntervalSeconds: 300 segmentTaskTimeout: 120000 taskExecutionCap: 256 taskMergeCap: 8 queryNode: cache: enabled: true memoryLimit: 2147483648 cacheSize: 32 dataSync: flowGraph: maxParallelism: 1024 maxQueueLength: 1024 enableDisk: true gracefulStopTimeout: 30 grouping: enabled: true maxNQ: 1000 topKMergeRatio: 10 loadMemoryUsageFactor: 3 maxDiskUsagePercentage: 95 port: 21123 scheduler: cpuRatio: 10 maxReadConcurrentRatio: 2 maxTimestampLag: 86400 receiveChanSize: 10240 unsolvedQueueSize: 10240 segcore: chunkRows: 1024 smallIndex: nlist: 128 nprobe: 16 stats: publishInterval: 1000 quotaAndLimits: compactionRate: enabled: false max: -1 ddl: collectionRate: -1 enabled: false partitionRate: -1 dml: bulkLoadRate: max: -1 deleteRate: max: -1 enabled: false insertRate: max: -1 dql: enabled: false queryRate: max: -1 searchRate: max: -1 enabled: true flushRate: enabled: false max: -1 indexRate: enabled: false max: -1 limitReading: coolOffSpeed: 0.9 forceDeny: false queueProtection: enabled: false nqInQueueThreshold: -1 queueLatencyThreshold: -1 resultProtection: enabled: false maxReadResultRate: -1 limitWriting: diskProtection: diskQuota: -1 enabled: true forceDeny: false memProtection: dataNodeMemoryHighWaterLevel: 0.95 dataNodeMemoryLowWaterLevel: 0.85 enabled: true queryNodeMemoryHighWaterLevel: 0.95 queryNodeMemoryLowWaterLevel: 0.85 ttProtection: enabled: false maxTimeTickDelay: 300 quotaCenterCollectInterval: 3 rootCoord: address: milvus-rootcoord dmlChannelNum: 256 enableActiveStandby: false importTaskExpiration: 900 importTaskRetention: 86400 maxPartitionNum: 4096 minSegmentSizeToEnableIndex: 1024 port: 53100 tls: caPemPath: configs/cert/ca.pem serverKeyPath: configs/cert/server.key serverPemPath: configs/cert/server.pem

@LoveEachDay @JackLCL I need your help

I use this command to roll back milvus, but the datanode also can not run. @LoveEachDay @JackLCL

milvus-datanode-2.txt

@LoveEachDay update the log file.please check it

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

watchTimeoutInterval to 600

does this help? for the error with a large timetick delay?

milvus-io / milvus