search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
29.42k stars 2.82k forks source link

[Bug]: Timeout occurs when create a GPU index after create an index and delete it #35841

Open qwevdb opened 2 weeks ago

qwevdb commented 2 weeks ago

Is there an existing issue for this?

Environment

- Milvus version: milvus v2.4.5-gpu
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.4.5
- OS(Ubuntu or CentOS): Ubuntu 24.04 LTS
- CPU/Memory: Intel Core i7-8700K / 48G
- GPU: NVIDIA GeForce RTX 1080Ti
- Others:

Current Behavior

Timeout occurs when creating a GPU index in the collection after creating and then deleting an index in the same collection.

Expected Behavior

Run normally and produce accurate results.

Steps To Reproduce

  1. create an index in the collection
  2. search
  3. delete this index
  4. create a GPU index in the same collection
  5. search
  6. timeout occurs
from pymilvus import Collection, connections, FieldSchema, CollectionSchema, DataType
import numpy as np

np.random.seed(0)
connections.connect(
    alias="bench",
    host="localhost",
    port="19530"
)
collection_name = "Benchmark"
dim = 2
idx = FieldSchema(
    name="id",
    dtype=DataType.INT64,
    is_primary=True,
)
vector = FieldSchema(
    name="vector",
    dtype=DataType.FLOAT_VECTOR,
    dim=dim,
)
fields = [idx, vector]
schema = CollectionSchema(fields=fields, description="Benchmark")
collection = Collection(
    name=collection_name,
    schema=schema,
    using="bench",
)

datas = []
number = 1024 
limit = 5
for i in range(number):
    data = {'id': i,
            'vector': np.random.rand(dim).tolist()}
    datas.append(data)
collection.insert(datas)
index_params1 = {
    "index_type": "IVF_FLAT",
    "params": {"nlist": 128},
    "metric_type": "L2"
}
index_params2 = {
    "index_type": "GPU_IVF_FLAT",
    "params": {"nlist": 128},
    "metric_type": "L2"
}
collection.create_index("vector", index_params1, timeout=100)
collection.flush()
collection.load()

query_vector = np.random.rand(dim).tolist()
res1 = collection.search(
    data=[query_vector],
    anns_field="vector",
    param={"metric_type": "L2",
           "params": {"nprobe": 10}},
    limit = limit
)
collection.release()
collection.drop_index()
collection.flush()
print(res1)

collection.create_index("vector", index_params2, timeout=100)  
# timeout
collection.flush()
collection.load()

res2 = collection.search(
    data=[query_vector],
    anns_field="vector",
    param={"metric_type": "L2",
           "params": {"nprobe": 10}},
    limit = limit
)
print(res2)
collection.drop()

Result：

data: ["['id: 275, distance: 0.000675233721267432, entity: {}', 'id: 823, distance: 0.0009240064537152648, entity: {}', 'id: 172, distance: 0.0009869951754808426, entity: {}', 'id: 497, distance: 0.0012439348502084613, entity: {}', 'id: 824, distance: 0.0015659565106034279, entity: {}']"]
Retry timeout: 100s
RPC error: [wait_for_creating_index], <MilvusException: (code=1, message=Retry timeout: 100s, message=collection Benchmark create index  timeout in 100s)>, <Time:{'RPC start': '2024-08-27 21:16:58.125073', 'RPC error': '2024-08-27 21:18:38.600050'}>
Retry timeout: 100s
RPC error: [create_index], <MilvusException: (code=1, message=Retry timeout: 100s, message=Retry timeout: 100s, message=collection Benchmark create index  timeout in 100s)>, <Time:{'RPC start': '2024-08-27 21:16:58.111692', 'RPC error': '2024-08-27 21:18:38.600509'}>
Traceback (most recent call last):
  File "/home/xxx/mlivus/dev3/milvus_test3.py", line 66, in <module>
    collection.create_index("vector", index_params2, timeout=100)
  File "/home/xxx/anaconda3/envs/milvus/lib/python3.10/site-packages/pymilvus/orm/collection.py", line 1373, in create_index
    return conn.create_index(self._name, field_name, index_params, timeout=timeout, **kwargs)
  File "/home/xxx/anaconda3/envs/milvus/lib/python3.10/site-packages/pymilvus/decorators.py", line 148, in handler
    raise e from e
  File "/home/xxx/anaconda3/envs/milvus/lib/python3.10/site-packages/pymilvus/decorators.py", line 144, in handler
    return func(*args, **kwargs)
  File "/home/xxx/anaconda3/envs/milvus/lib/python3.10/site-packages/pymilvus/decorators.py", line 183, in handler
    return func(self, *args, **kwargs)
  File "/home/xxx/anaconda3/envs/milvus/lib/python3.10/site-packages/pymilvus/decorators.py", line 108, in handler
    raise MilvusException(
pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 100s, message=Retry timeout: 100s, message=collection Benchmark create index  timeout in 100s)>

Milvus Log

No response

Anything else?

I encountered the aforementioned issues exclusively on the NVIDIA GeForce RTX 1080Ti GPU, whereas the RTX 4090 GPU yielded results without any errors.

yanliang567 commented 2 weeks ago

@qwevdb please offer milvus logs, For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs. /assign @qwevdb /unassign

qwevdb commented 2 weeks ago

/assign @yanliang567

Milvus Log: 

[2024/08/30 16:07:52.537 +00:00] [INFO] [proxy/impl.go:5843] ["connect received"] [traceID=952e0213ebd51b8e8b248037eaa32a47] [sdk_type=Python] [sdk_version=2.4.5] [local_time="2024-08-31 00:07:52.537274"] [user=] [host=OMEN] [db=default]
[2024/08/30 16:07:52.538 +00:00] [INFO] [rootcoord/root_coord.go:933] ["received request to list databases"] [traceID=952e0213ebd51b8e8b248037eaa32a47] [msgID=0]
[2024/08/30 16:07:52.538 +00:00] [INFO] [rootcoord/root_coord.go:959] ["done to list databases"] [traceID=952e0213ebd51b8e8b248037eaa32a47] [msgID=0] ["num of databases"=1]
[2024/08/30 16:07:52.538 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=952e0213ebd51b8e8b248037eaa32a47] [sdk_type=Python] [sdk_version=2.4.5] [local_time="2024-08-31 00:07:52.537274"] [user=] [host=OMEN] [identifier=452207331906682883] [last_active_time=2024/08/30 16:07:52.538 +00:00]
[2024/08/30 16:07:52.539 +00:00] [WARN] [rootcoord/root_coord.go:1181] ["failed to describe collection"] [traceID=3faa2e4db3d6c4def34e8433e7662900] [collectionName=Benchmark] [dbName=default] [id=0] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[database=default][collection=Benchmark]"]
[2024/08/30 16:07:52.540 +00:00] [INFO] [rootcoord/root_coord.go:2741] ["received request to describe database "] [traceID=a6225129aebe6d2419b2018b6404347d] [dbName=default]
[2024/08/30 16:07:52.541 +00:00] [INFO] [rootcoord/root_coord.go:2765] ["done to describe database"] [traceID=a6225129aebe6d2419b2018b6404347d] [dbName=default] [ts=452207331906682886]
[2024/08/30 16:07:52.541 +00:00] [WARN] [rootcoord/root_coord.go:1181] ["failed to describe collection"] [traceID=78f6f5fc6647a629d93645ff8ec6444d] [collectionName=Benchmark] [dbName=default] [id=0] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[database=default][collection=Benchmark]"]
[2024/08/30 16:07:52.541 +00:00] [INFO] [rootcoord/root_coord.go:972] ["received request to create collection"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [dbName=default] [name=Benchmark] [role=rootcoord]
[2024/08/30 16:07:52.542 +00:00] [INFO] [rootcoord/create_collection_task.go:316] ["assign partitions when create collection"] [collectionName=Benchmark] [partitionNames="[_default]"]
[2024/08/30 16:07:52.542 +00:00] [INFO] [rootcoord/timeticksync.go:379] ["add dml channels"] [channels="[by-dev-rootcoord-dml_0]"]
[2024/08/30 16:07:52.542 +00:00] [INFO] [proxy/impl.go:116] ["received request to invalidate collection meta cache"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [module=Proxy] [role=proxy] [db=default] [collectionName=Benchmark] [collectionID=0] [msgType=DropCollection] [partitionName=]
[2024/08/30 16:07:52.542 +00:00] [INFO] [proxy/meta_cache.go:997] ["clearing shard cache for collection"] [collectionName=Benchmark]
[2024/08/30 16:07:52.542 +00:00] [INFO] [proxy/impl.go:133] ["complete to invalidate collection meta cache with collection name"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [module=Proxy] [role=proxy] [db=default] [collectionName=Benchmark] [collectionID=0] [msgType=DropCollection] [partitionName=] [collectionName=Benchmark]
[2024/08/30 16:07:52.542 +00:00] [INFO] [proxy/channels_mgr.go:274] ["dml stream removed"] [collection_id=0]
[2024/08/30 16:07:52.542 +00:00] [INFO] [proxy/impl.go:170] ["complete to invalidate collection meta cache"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [module=Proxy] [role=proxy] [db=default] [collectionName=Benchmark] [collectionID=0] [msgType=DropCollection] [partitionName=]
[2024/08/30 16:07:52.545 +00:00] [INFO] [rootcoord/meta_table.go:397] ["add collection to meta table"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [dbID=1] [collection=Benchmark] [id=452207326985453669] [ts=452207331906682889]
[2024/08/30 16:07:52.545 +00:00] [INFO] [rootcoord/broker.go:162] ["watching channels"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [ts=452207331906682889] [collection=452207326985453669] [vChannels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:52.545 +00:00] [INFO] [datacoord/services.go:1239] ["receive watch channels request"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [collectionID=452207326985453669] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:52.545 +00:00] [INFO] [datacoord/channel_manager_v2.go:236] ["Add channel"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:52.545 +00:00] [INFO] [datacoord/handler.go:63] [GetDataVChanPositions] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [numOfSegments=0]
[2024/08/30 16:07:52.545 +00:00] [INFO] [datacoord/handler.go:284] ["segment position not found, setting channel seek position to channel start position"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207331906682889] [posTime=2024/08/30 16:07:52.520 +00:00]
[2024/08/30 16:07:52.545 +00:00] [INFO] [datacoord/handler.go:318] ["channel seek position set from collection start position"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207331906682889] [posTime=2024/08/30 16:07:52.520 +00:00]
[2024/08/30 16:07:52.547 +00:00] [INFO] [datacoord/policy.go:425] ["Assign channels to nodes by channel count"] ["toAssign channel count"=1] ["original nodeID"=-9223372036854775808] ["exclusive nodes"="[]"] [operations="[\"{type=Watch,nodeID=1,channels=\\\"[by-dev-rootcoord-dml_0_452207326985453669v0]\\\"}\",\"{type=Delete,nodeID=-9223372036854775808,channels=\\\"[by-dev-rootcoord-dml_0_452207326985453669v0]\\\"}\"]"] [nodesLackOfChannels="[1]"]
[2024/08/30 16:07:52.547 +00:00] [INFO] [datacoord/handler.go:63] [GetDataVChanPositions] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [numOfSegments=0]
[2024/08/30 16:07:52.547 +00:00] [INFO] [datacoord/handler.go:284] ["segment position not found, setting channel seek position to channel start position"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207331906682889] [posTime=2024/08/30 16:07:52.520 +00:00]
[2024/08/30 16:07:52.547 +00:00] [INFO] [datacoord/handler.go:318] ["channel seek position set from collection start position"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207331906682889] [posTime=2024/08/30 16:07:52.520 +00:00]
[2024/08/30 16:07:52.547 +00:00] [INFO] [datacoord/channel_manager_v2.go:256] ["Assign channel"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [updates="[\"{type=Watch,nodeID=1,channels=\\\"[by-dev-rootcoord-dml_0_452207326985453669v0]\\\"}\",\"{type=Delete,nodeID=-9223372036854775808,channels=\\\"[by-dev-rootcoord-dml_0_452207326985453669v0]\\\"}\"]"]
[2024/08/30 16:07:52.548 +00:00] [INFO] [datacoord/kv_catalog.go:465] ["NON remove flag tombstone added"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:52.548 +00:00] [INFO] [rootcoord/broker.go:179] ["done to watch channels"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [ts=452207331906682889] [collection=452207326985453669] [vChannels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:52.549 +00:00] [INFO] [rootcoord/meta_table.go:436] ["change collection state"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [collection=452207326985453669] [state=CollectionCreated] [ts=452207331906682889]
[2024/08/30 16:07:52.549 +00:00] [INFO] [rootcoord/root_coord.go:1007] ["done to create collection"] [traceID=54d68af3d77db6a2ec4cc14e62f9c8ec] [role=rootcoord] [name=Benchmark] [ts=452207331906682889]
[2024/08/30 16:07:52.560 +00:00] [INFO] [proxy/meta_cache.go:397] ["meta update success"] [database=default] [collectionName=Benchmark] [collectionID=452207326985453669]
[2024/08/30 16:07:52.560 +00:00] [INFO] [msgstream/mq_msgstream.go:116] ["Msg Stream state"] [can_produce=true]
[2024/08/30 16:07:52.561 +00:00] [WARN] [server/rocksmq_impl.go:398] ["rocksmq topic already exists "] [topic=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:52.561 +00:00] [INFO] [proxy/channels_mgr.go:232] ["create message stream"] [collection=452207326985453669] [virtual_channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"] [physical_channels="[by-dev-rootcoord-dml_0]"]
[2024/08/30 16:07:52.561 +00:00] [INFO] [datacoord/services.go:197] ["handle assign segment request"] [traceID=0f5dafc1fd9c937424985c33a8e5bb5b] [collectionID=452207326985453669] [partitionID=452207326985453670] [channelName=by-dev-rootcoord-dml_0_452207326985453669v0] [count=1024] ["segment level"=Legacy]
[2024/08/30 16:07:52.561 +00:00] [INFO] [datacoord/meta.go:206] ["meta update: add collection"] [collectionID=452207326985453669]
[2024/08/30 16:07:52.561 +00:00] [INFO] [datacoord/meta.go:211] ["meta update: add collection - complete"] [collectionID=452207326985453669]
[2024/08/30 16:07:52.562 +00:00] [INFO] [datacoord/meta.go:395] ["meta update: adding segment - Start"] [traceID=0f5dafc1fd9c937424985c33a8e5bb5b] [segmentID=452207326985653680]
[2024/08/30 16:07:52.563 +00:00] [INFO] [datacoord/meta.go:407] ["meta update: adding segment - complete"] [traceID=0f5dafc1fd9c937424985c33a8e5bb5b] [segmentID=452207326985653680]
[2024/08/30 16:07:52.563 +00:00] [INFO] [datacoord/segment_manager.go:438] ["datacoord: estimateTotalRows: "] [traceID=0f5dafc1fd9c937424985c33a8e5bb5b] [CollectionID=452207326985453669] [SegmentID=452207326985653680] [Rows=33554432] [Channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:52.563 +00:00] [INFO] [datacoord/meta.go:1131] ["meta update: add allocation - complete"] [segmentID=452207326985653680]
[2024/08/30 16:07:52.563 +00:00] [INFO] [datacoord/services.go:226] ["success to assign segments"] [traceID=0f5dafc1fd9c937424985c33a8e5bb5b] [collectionID=452207326985453669] [assignments="[{\"SegmentID\":452207326985653680,\"NumOfRows\":1024,\"ExpireTime\":452207332430970899}]"]
[2024/08/30 16:07:52.565 +00:00] [INFO] [proxy/impl.go:1996] ["CreateIndex received"] [traceID=b56c08c19817c4d992d41efbfe96dace] [role=proxy] [db=default] [collection=Benchmark] [field=vector] [extra_params="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"]
[2024/08/30 16:07:52.565 +00:00] [INFO] [proxy/impl.go:2009] ["CreateIndex enqueued"] [traceID=b56c08c19817c4d992d41efbfe96dace] [role=proxy] [db=default] [collection=Benchmark] [field=vector] [extra_params="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [BeginTs=452207331906682902] [EndTs=452207331906682902]
[2024/08/30 16:07:52.565 +00:00] [INFO] [proxy/task_index.go:438] ["proxy create index"] [traceID=b56c08c19817c4d992d41efbfe96dace] [collectionID=452207326985453669] [fieldID=101] [indexName=] [typeParams="[{\"key\":\"dim\",\"value\":\"2\"}]"] [indexParams="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [newExtraParams="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"]
[2024/08/30 16:07:52.565 +00:00] [INFO] [datacoord/index_service.go:176] ["receive CreateIndex request"] [traceID=b56c08c19817c4d992d41efbfe96dace] [collectionID=452207326985453669] [IndexName=] [fieldID=101] [TypeParams="[{\"key\":\"dim\",\"value\":\"2\"}]"] [IndexParams="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"nlist\",\"value\":\"128\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"]
[2024/08/30 16:07:52.566 +00:00] [INFO] [datacoord/index_meta.go:274] ["meta update: CreateIndex"] [collectionID=452207326985453669] [fieldID=101] [indexID=452207326985653683] [indexName=vector]
[2024/08/30 16:07:52.567 +00:00] [INFO] [datacoord/index_meta.go:287] ["meta update: CreateIndex success"] [collectionID=452207326985453669] [fieldID=101] [indexID=452207326985653683] [indexName=vector]
[2024/08/30 16:07:52.567 +00:00] [INFO] [datacoord/index_service.go:262] ["CreateIndex successfully"] [traceID=b56c08c19817c4d992d41efbfe96dace] [collectionID=452207326985453669] [IndexName=vector] [fieldID=101] [IndexID=452207326985653683]
[2024/08/30 16:07:52.567 +00:00] [INFO] [datacoord/index_service.go:129] ["receive create index notify"] [collectionID=452207326985453669]
[2024/08/30 16:07:52.567 +00:00] [INFO] [proxy/impl.go:2027] ["CreateIndex done"] [traceID=b56c08c19817c4d992d41efbfe96dace] [role=proxy] [db=default] [collection=Benchmark] [field=vector] [extra_params="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [BeginTs=452207331906682902] [EndTs=452207331906682902]
[2024/08/30 16:07:52.568 +00:00] [INFO] [proxy/impl.go:6009] ["AllocTimestamp request receive"]
[2024/08/30 16:07:52.568 +00:00] [INFO] [proxy/impl.go:6018] ["AllocTimestamp request success"] [timestamp=452207331906682904]
[2024/08/30 16:07:52.755 +00:00] [INFO] [datacoord/channel_manager_v2.go:540] ["Notify channel operations to datanode"] [assignment=1] ["total operation count"=1] ["channel names"="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:52.756 +00:00] [INFO] [datacoord/handler.go:63] [GetDataVChanPositions] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [numOfSegments=1]
[2024/08/30 16:07:52.756 +00:00] [INFO] [datacoord/handler.go:284] ["segment position not found, setting channel seek position to channel start position"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207331906682889] [posTime=2024/08/30 16:07:52.520 +00:00]
[2024/08/30 16:07:52.756 +00:00] [INFO] [datacoord/handler.go:318] ["channel seek position set from collection start position"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207331906682889] [posTime=2024/08/30 16:07:52.520 +00:00]
[2024/08/30 16:07:52.756 +00:00] [INFO] [datacoord/channel_manager_v2.go:644] ["Notify channel operation"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [assignment=1] [operation=ToWatch]
[2024/08/30 16:07:52.757 +00:00] [INFO] [datanode/services.go:334] ["DataNode receives NotifyChannelOperation"] [traceID=b9778ecfa5e8b4e04b834526926fcc4b] ["operation count"=1]
[2024/08/30 16:07:52.757 +00:00] [INFO] [datanode/channel_manager.go:309] ["Start to execute channel operation"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [opID=452207326985453672] [state=ToWatch]
[2024/08/30 16:07:52.757 +00:00] [INFO] [datanode/channel_manager.go:356] ["Start timer for ToWatch operation"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [opID=452207326985453672] [timeout=5m0s]
[2024/08/30 16:07:52.757 +00:00] [INFO] [datacoord/channel_manager_v2.go:567] ["Finish to notify channel operations to datanode"] [assignment=1] ["operation count"=1] ["success count"=1] ["failure count"=0]
[2024/08/30 16:07:52.760 +00:00] [INFO] [msgdispatcher/manager.go:66] ["create new dispatcherManager"] [role=datanode] [nodeID=1] [pchannel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:52.760 +00:00] [INFO] [msgdispatcher/dispatcher.go:93] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_0] [subName=datanode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true] [isMain=true]
[2024/08/30 16:07:52.760 +00:00] [INFO] [msgstream/mq_msgstream.go:116] ["Msg Stream state"] [can_produce=true]
[2024/08/30 16:07:52.760 +00:00] [INFO] [msgdispatcher/manager.go:154] ["dispatcherManager is running..."] [role=datanode] [nodeID=1] [pchannel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:52.760 +00:00] [INFO] [msgstream/mq_msgstream.go:872] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_0] [MessageID="AQDEASuQRgY="]
[2024/08/30 16:07:52.760 +00:00] [INFO] [msgstream/mq_msgstream.go:882] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:52.761 +00:00] [INFO] [msgstream/mq_msgstream.go:938] ["skip msg"] [source=0] [type=CreateCollection] [size=284] [position=<nil>]
[2024/08/30 16:07:52.761 +00:00] [INFO] [msgstream/mq_msgstream.go:938] ["skip msg"] [source=1] [type=TimeTick] [size=17] [position=<nil>]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/dispatcher.go:113] ["seek successfully"] [pchannel=by-dev-rootcoord-dml_0] [subName=datanode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true] [isMain=true] [posTime=2024/08/30 16:07:52.520 +00:00] [tsLag=304.219309ms]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/dispatcher.go:149] ["add new target"] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0] [isMain=true]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/manager.go:101] ["add main dispatcher"] [role=datanode] [nodeID=1] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/dispatcher.go:177] ["get signal"] [pchannel=by-dev-rootcoord-dml_0] [signal=start] [isMain=true]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/dispatcher.go:200] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_0] [signal=start] [isMain=true]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/client.go:85] ["register done"] [role=datanode] [nodeID=1] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:52.824 +00:00] [INFO] [datanode/flow_graph_dmstream_input_node.go:53] ["datanode seek successfully when register to msgDispatcher"] [nodeID=1] [collectionID=452207326985453669] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0] [msgID="\u0001\u0000\ufffd\u0001+\ufffdF\u0006"] [tsTime=2024/08/30 16:07:52.520 +00:00] [tsLag=304.614516ms]
[2024/08/30 16:07:52.824 +00:00] [INFO] [msgdispatcher/dispatcher.go:205] ["begin to work"] [pchannel=by-dev-rootcoord-dml_0] [isMain=true]
[2024/08/30 16:07:52.824 +00:00] [INFO] [datanode/flow_graph_dd_node.go:306] ["ddNode add sealed and growing segments"] [collectionID=452207326985453669] ["No. sealed segments"=0] ["No. growing segments"=0]
[2024/08/30 16:07:52.824 +00:00] [INFO] [datanode/data_sync_service.go:93] ["dataSyncService starting flow graph"] [collectionID=452207326985453669] [vChanName=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:52.824 +00:00] [INFO] [datanode/channel_manager.go:379] ["Stop timer for ToWatch operation succeeded"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [opID=452207326985453672] [timeout=5m0s]
[2024/08/30 16:07:52.824 +00:00] [INFO] [datanode/channel_manager.go:194] ["Success to watch"] [opID=452207326985453672] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [State=WatchSuccess]
[2024/08/30 16:07:53.024 +00:00] [INFO] [io/io_pool.go:54] ["init BFApplyPool"] [poolSize=48]
[2024/08/30 16:07:53.028 +00:00] [INFO] [writebuffer/write_buffer.go:283] ["checkpoint evaluated"] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [cpSource=segmentBuffer] [segmentID=452207326985653680] [cpTimestamp=452207331906682889]
[2024/08/30 16:07:53.070 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=6a1f692d69d1958254a3aebfd5eb7853] [collectionID=452207326985453669] [indexName=] [timestamp=452207331906682904]
[2024/08/30 16:07:53.070 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=452207326985453669] [indexID=452207326985653683] [totalRows=0] [indexRows=0] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/08/30 16:07:53.070 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=6a1f692d69d1958254a3aebfd5eb7853] [collectionID=452207326985453669] [indexName=]
[2024/08/30 16:07:53.073 +00:00] [INFO] [datacoord/services.go:74] ["receive flush request"] [traceID=8b83c668179ac310dbed0a27cdd0fbf5] [dbID=0] [collectionID=452207326985453669]
[2024/08/30 16:07:53.074 +00:00] [INFO] [datacoord/meta.go:549] ["meta update: setting segment state - complete"] [segmentID=452207326985653680] ["target state"=Sealed]
[2024/08/30 16:07:53.074 +00:00] [INFO] [datacoord/session_manager.go:308] ["SessionManagerImpl.FlushChannels start"] [traceID=8b83c668179ac310dbed0a27cdd0fbf5] [nodeID=1] [flushTs=2024/08/30 16:07:53.069 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:53.075 +00:00] [INFO] [datanode/services.go:389] ["DataNode receives FlushChannels request"] [traceID=8b83c668179ac310dbed0a27cdd0fbf5] [nodeId=1] [flushTs=452207332050599939] ["flushTs in Time"=2024/08/30 16:07:53.069 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:53.075 +00:00] [INFO] [datanode/services.go:404] ["success to FlushChannels"] [traceID=8b83c668179ac310dbed0a27cdd0fbf5] [nodeId=1] [flushTs=452207332050599939] ["flushTs in Time"=2024/08/30 16:07:53.069 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:53.075 +00:00] [INFO] [datacoord/session_manager.go:315] ["SessionManagerImpl.FlushChannels successfully"] [traceID=8b83c668179ac310dbed0a27cdd0fbf5] [nodeID=1] [flushTs=2024/08/30 16:07:53.069 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:53.075 +00:00] [INFO] [datacoord/services.go:166] ["flush response with segments"] [traceID=8b83c668179ac310dbed0a27cdd0fbf5] [dbID=0] [collectionID=452207326985453669] [collectionID=452207326985453669] [sealSegments="[452207326985653680]"] [flushedSegmentsCount=0] [timeOfSeal=2024/08/30 16:07:53.069 +00:00] [flushTs=2024/08/30 16:07:53.069 +00:00]
[2024/08/30 16:07:53.077 +00:00] [INFO] [datacoord/services.go:1291] ["DataCoord receive GetFlushState request, Flushed is false"] [traceID=611bf69a108dbd1addd78af203627cfe] [collection=452207326985453669] [flushTs=2024/08/30 16:07:53.069 +00:00] [unflushed="[452207326985653680]"] [len=1]
[2024/08/30 16:07:53.223 +00:00] [INFO] [writebuffer/write_buffer.go:283] ["checkpoint evaluated"] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [cpSource=segmentBuffer] [segmentID=452207326985653680] [cpTimestamp=452207331906682889]
[2024/08/30 16:07:53.424 +00:00] [INFO] [writebuffer/write_buffer.go:360] ["SyncPolicy selects segments"] [segmentIDs="[452207326985653680]"] [reason="flush ts"]
[2024/08/30 16:07:53.424 +00:00] [INFO] [writebuffer/write_buffer.go:293] ["write buffer get segments to sync"] [segmentIDs="[452207326985653680]"]
[2024/08/30 16:07:53.463 +00:00] [INFO] [syncmgr/sync_manager.go:163] ["sync mgr sumbit task with key"] [key=452207326985653680]
[2024/08/30 16:07:53.463 +00:00] [INFO] [writebuffer/write_buffer.go:283] ["checkpoint evaluated"] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [cpSource=syncManager] [segmentID=452207326985653680] [cpTimestamp=452207331906682889]
[2024/08/30 16:07:53.473 +00:00] [INFO] [syncmgr/meta_writer.go:70] [SaveBinlogPath] [SegmentID=452207326985653680] [CollectionID=452207326985453669] [ParitionID=452207326985453670] [startPos="[{\"start_position\":{\"channel_name\":\"by-dev-rootcoord-dml_0_452207326985453669v0\",\"msgID\":\"AQDEASuQRgY=\",\"timestamp\":452207331906682889},\"segmentID\":452207326985653680}]"] [checkPoints="[{\"segmentID\":452207326985653680,\"position\":{\"channel_name\":\"by-dev-rootcoord-dml_0_452207326985453669v0\",\"msgID\":\"CQDEASuQRgY=\",\"msgGroup\":\"datanode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true\",\"timestamp\":452207332077076481},\"num_of_rows\":1024}]"] [binlogNum=4] [statslogNum=1] [deltalogNum=0] [vChannelName=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:53.474 +00:00] [INFO] [datacoord/services.go:458] ["receive SaveBinlogPaths request"] [traceID=f3e62e8111e919fd0c12eaecc16fd011] [nodeID=1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [collectionID=452207326985453669] [segmentID=452207326985653680] [level=L1] [isFlush=false] [isDropped=false] [checkpoints="[{\"segmentID\":452207326985653680,\"position\":{\"channel_name\":\"by-dev-rootcoord-dml_0_452207326985453669v0\",\"msgID\":\"CQDEASuQRgY=\",\"msgGroup\":\"datanode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true\",\"timestamp\":452207332077076481},\"num_of_rows\":1024}]"]
[2024/08/30 16:07:53.475 +00:00] [INFO] [datacoord/meta.go:874] ["meta update: update flush segments info - update flush segments info successfully"]
[2024/08/30 16:07:53.475 +00:00] [INFO] [datacoord/services.go:532] ["SaveBinlogPaths sync segment with meta"] [traceID=f3e62e8111e919fd0c12eaecc16fd011] [nodeID=1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [collectionID=452207326985453669] [segmentID=452207326985653680] [level=L1] [binlogs="[{\"fieldID\":100,\"binlogs\":[{\"entries_num\":1024,\"timestamp_from\":452207331906682895,\"timestamp_to\":452207331906682895,\"log_size\":3672,\"logID\":452207326985653689,\"memory_size\":8192}]},{\"fieldID\":101,\"binlogs\":[{\"entries_num\":1024,\"timestamp_from\":452207331906682895,\"timestamp_to\":452207331906682895,\"log_size\":9891,\"logID\":452207326985653690,\"memory_size\":8196}]},{\"binlogs\":[{\"entries_num\":1024,\"timestamp_from\":452207331906682895,\"timestamp_to\":452207331906682895,\"log_size\":2818,\"logID\":452207326985653691,\"memory_size\":8192}]},{\"fieldID\":1,\"binlogs\":[{\"entries_num\":1024,\"timestamp_from\":452207331906682895,\"timestamp_to\":452207331906682895,\"log_size\":458,\"logID\":452207326985653692,\"memory_size\":8192}]}]"] [deltalogs=null] [statslogs="[{\"fieldID\":100,\"binlogs\":[{\"entries_num\":1024,\"timestamp_from\":452207331906682895,\"timestamp_to\":452207331906682895,\"log_size\":2572,\"logID\":452207326985653693,\"memory_size\":2572}]}]"]
[2024/08/30 16:07:53.475 +00:00] [INFO] [syncmgr/task.go:214] ["task done"] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [level=L1] [flushedSize=32772]
[2024/08/30 16:07:53.580 +00:00] [INFO] [datacoord/services.go:1291] ["DataCoord receive GetFlushState request, Flushed is false"] [traceID=d22ccf33a5210cb703a416576e52ad97] [collection=452207326985453669] [flushTs=2024/08/30 16:07:53.069 +00:00] [unflushed="[452207326985653680]"] [len=1]
[2024/08/30 16:07:53.627 +00:00] [INFO] [datacoord/meta.go:1446] ["UpdateChannelCheckpoint done"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [ts=452207332116135939] [time=2024/08/30 16:07:53.319 +00:00]
[2024/08/30 16:07:53.627 +00:00] [INFO] [writebuffer/manager.go:227] ["reset channel flushTs"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:53.755 +00:00] [INFO] [datacoord/channel_manager_v2.go:597] ["Check ToWatch/ToRelease channel operations progress"] ["channel count"=1] ["channel names"="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:53.756 +00:00] [INFO] [datanode/services.go:364] ["DataNode receives CheckChannelOperationProgress"] [traceID=c7caf047856b551508737c6ccf067475] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [operation=ToWatch]
[2024/08/30 16:07:53.757 +00:00] [INFO] [datacoord/channel_manager_v2.go:669] ["Got channel operation progress"] [opID=452207326985453672] [nodeID=1] ["check operation"=ToWatch] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] ["got state"=WatchSuccess] [progress=100]
[2024/08/30 16:07:53.757 +00:00] [INFO] [datacoord/channel_manager_v2.go:630] ["Finish to Check ToWatch/ToRelease channel operations progress"] ["channel count"=1] ["channel names"="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:54.084 +00:00] [INFO] [datacoord/services.go:1291] ["DataCoord receive GetFlushState request, Flushed is false"] [traceID=4cf05ee558f9b8af4e24dcb15f33e87e] [collection=452207326985453669] [flushTs=2024/08/30 16:07:53.069 +00:00] [unflushed="[452207326985653680]"] [len=1]
[2024/08/30 16:07:54.588 +00:00] [INFO] [datacoord/services.go:1291] ["DataCoord receive GetFlushState request, Flushed is false"] [traceID=893a6e790aa134c2b88458b9cb3e8fcf] [collection=452207326985453669] [flushTs=2024/08/30 16:07:53.069 +00:00] [unflushed="[452207326985653680]"] [len=1]
[2024/08/30 16:07:54.773 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=d947273bc4ec96d88645fdac69b4b0df] [collectionID=452207326985453669] [indexName=] [timestamp=0]
[2024/08/30 16:07:54.773 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=452207326985453669] [indexID=452207326985653683] [totalRows=0] [indexRows=0] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/08/30 16:07:54.773 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=d947273bc4ec96d88645fdac69b4b0df] [collectionID=452207326985453669] [indexName=]
[2024/08/30 16:07:55.091 +00:00] [INFO] [datacoord/services.go:1291] ["DataCoord receive GetFlushState request, Flushed is false"] [traceID=c8d05f9f9db1a035e2d3d343aba9b1ee] [collection=452207326985453669] [flushTs=2024/08/30 16:07:53.069 +00:00] [unflushed="[452207326985653680]"] [len=1]
[2024/08/30 16:07:55.320 +00:00] [INFO] [datacoord/services.go:1503] ["start flushing segments"] [traceID=0dc513f5192a7f5232bd392571e77cfb] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [sourceID=1] [ts=452207332588257281] [segmentIDs="[452207326985653680]"]
[2024/08/30 16:07:55.320 +00:00] [INFO] [datanode/services.go:84] ["receive FlushSegments request"] [traceID=9c019b5a05903a4884b4f6e87e0d46b7] [nodeID=1] [collectionID=452207326985453669] [channelName=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentIDs="[452207326985653680]"]
[2024/08/30 16:07:55.321 +00:00] [INFO] [datanode/services.go:101] ["success to FlushSegments"] [traceID=9c019b5a05903a4884b4f6e87e0d46b7] [nodeID=1] [collectionID=452207326985453669] [channelName=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentIDs="[452207326985653680]"]
[2024/08/30 16:07:55.321 +00:00] [INFO] [datacoord/session_manager.go:187] ["flush call succeeded"] [nodeID=1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:55.424 +00:00] [INFO] [writebuffer/write_buffer.go:360] ["SyncPolicy selects segments"] [segmentIDs="[452207326985653680]"] [reason="segment flushing"]
[2024/08/30 16:07:55.424 +00:00] [INFO] [writebuffer/write_buffer.go:293] ["write buffer get segments to sync"] [segmentIDs="[452207326985653680]"]
[2024/08/30 16:07:55.425 +00:00] [INFO] [syncmgr/sync_manager.go:163] ["sync mgr sumbit task with key"] [key=452207326985653680]
[2024/08/30 16:07:55.431 +00:00] [INFO] [syncmgr/meta_writer.go:70] [SaveBinlogPath] [SegmentID=452207326985653680] [CollectionID=452207326985453669] [ParitionID=452207326985453670] [startPos="[]"] [checkPoints="[{\"segmentID\":452207326985653680,\"position\":{\"channel_name\":\"by-dev-rootcoord-dml_0_452207326985453669v0\",\"msgID\":\"EwDEASuQRgY=\",\"msgGroup\":\"datanode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true\",\"timestamp\":452207332588257282},\"num_of_rows\":1024}]"] [binlogNum=0] [statslogNum=1] [deltalogNum=0] [vChannelName=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:55.432 +00:00] [INFO] [datacoord/services.go:458] ["receive SaveBinlogPaths request"] [traceID=288d1db0bd51fc07f560ce6587d58797] [nodeID=1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [collectionID=452207326985453669] [segmentID=452207326985653680] [level=L1] [isFlush=true] [isDropped=false] [checkpoints="[{\"segmentID\":452207326985653680,\"position\":{\"channel_name\":\"by-dev-rootcoord-dml_0_452207326985453669v0\",\"msgID\":\"EwDEASuQRgY=\",\"msgGroup\":\"datanode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true\",\"timestamp\":452207332588257282},\"num_of_rows\":1024}]"]
[2024/08/30 16:07:55.434 +00:00] [INFO] [datacoord/meta.go:874] ["meta update: update flush segments info - update flush segments info successfully"]
[2024/08/30 16:07:55.434 +00:00] [INFO] [datacoord/services.go:532] ["SaveBinlogPaths sync segment with meta"] [traceID=288d1db0bd51fc07f560ce6587d58797] [nodeID=1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [collectionID=452207326985453669] [segmentID=452207326985653680] [level=L1] [binlogs=null] [deltalogs=null] [statslogs="[{\"fieldID\":100,\"binlogs\":[{\"entries_num\":1024,\"log_size\":2574,\"logID\":1,\"memory_size\":2574}]}]"]
[2024/08/30 16:07:55.434 +00:00] [INFO] [datacoord/server.go:1015] ["flush successfully"] [segmentID=452207326985653680]
[2024/08/30 16:07:55.435 +00:00] [INFO] [syncmgr/task.go:214] ["task done"] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [level=L1] [flushedSize=0]
[2024/08/30 16:07:55.436 +00:00] [INFO] [datacoord/meta.go:549] ["meta update: setting segment state - complete"] [segmentID=452207326985653680] ["target state"=Flushed]
[2024/08/30 16:07:55.436 +00:00] [INFO] [datacoord/server.go:1062] ["flush segment complete"] [id=452207326985653680]
[2024/08/30 16:07:55.436 +00:00] [INFO] [datacoord/index_service.go:140] ["receive new flushed segment"] [segmentID=452207326985653680]
[2024/08/30 16:07:55.436 +00:00] [INFO] [datacoord/compaction_trigger.go:489] ["the number of candidate segments is 0, skip to handle compaction"]
[2024/08/30 16:07:55.436 +00:00] [INFO] [datacoord/index_service.go:58] ["create index for segment"] [segmentID=452207326985653680] [indexID=452207326985653683]
[2024/08/30 16:07:55.437 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=452207326985453669] [segmentID=452207326985653680] [indexID=452207326985653683] [buildID=452207326985853700]
[2024/08/30 16:07:55.439 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=452207326985453669] [segmentID=452207326985653680] [indexID=452207326985653683] [buildID=452207326985853700]
[2024/08/30 16:07:55.440 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=452207326985853700]
[2024/08/30 16:07:55.440 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=1]
[2024/08/30 16:07:55.440 +00:00] [INFO] [indexnode/indexnode_service.go:242] ["Get Index Job Stats"] [traceID=c58f121015ae23402c4a18d27e1c9581] [unissued=0] [active=0] [slot=1]
[2024/08/30 16:07:55.441 +00:00] [INFO] [datacoord/indexnode_manager.go:135] ["peek client success"] [nodeID=1]
[2024/08/30 16:07:55.441 +00:00] [INFO] [datacoord/indexnode_manager.go:148] ["peek client success"] [nodeID=1]
[2024/08/30 16:07:55.444 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=4]
[2024/08/30 16:07:55.445 +00:00] [INFO] [indexnode/indexnode_service.go:56] ["IndexNode building index ..."] [traceID=9341c0986801fd54967839301eda0ce1] [clusterID=by-dev] [indexBuildID=452207326985853700] [collectionID=452207326985453669] [indexID=0] [indexName=] [indexFilePrefix=files/index_files] [indexVersion=1] [dataPaths="[]"] [typeParams="[{\"key\":\"dim\",\"value\":\"2\"}]"] [indexParams="[{\"key\":\"index_type\",\"value\":\"IVF_FLAT\"},{\"key\":\"nlist\",\"value\":\"128\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [numRows=1024] [current_index_version=4] [storepath=] [storeversion=0] [indexstorepath=] [dim=0]
[2024/08/30 16:07:55.450 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=a-bucket] [root=files]
[2024/08/30 16:07:55.450 +00:00] [INFO] [indexnode/task.go:318] ["IndexNode IndexBuilderTask Enqueue"] [buildID=452207326985853700] [segmentID=0]
[2024/08/30 16:07:55.450 +00:00] [INFO] [indexnode/indexnode_service.go:140] ["IndexNode successfully scheduled"] [traceID=9341c0986801fd54967839301eda0ce1] [clusterID=by-dev] [indexBuildID=452207326985853700] [indexName=]
[2024/08/30 16:07:55.451 +00:00] [INFO] [indexnode/task.go:324] ["Begin to prepare indexBuildTask"] [buildID=452207326985853700] [Collection=0] [SegmentID=0]
[2024/08/30 16:07:55.451 +00:00] [INFO] [indexnode/task.go:375] ["Successfully prepare indexBuildTask"] [buildID=452207326985853700] [Collection=0] [SegmentID=0]
[2024/08/30 16:07:55.451 +00:00] [INFO] [datacoord/index_builder.go:433] ["index task assigned successfully"] [buildID=452207326985853700] [segmentID=452207326985653680] [nodeID=1]
[2024/08/30 16:07:55.453 +00:00] [INFO] [datacoord/index_meta.go:753] ["meta update: segment index in progress success"] [buildID=452207326985853700] [segmentID=452207326985653680]
[2024/08/30 16:07:55.459 +00:00] [INFO] [indexnode/task.go:516] ["index params are ready"] [buildID=452207326985853700] ["index params"="{\"dim\":\"2\",\"index_type\":\"IVF_FLAT\",\"metric_type\":\"L2\",\"nlist\":\"128\"}"]
I20240830 16:07:55.460377    25 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=warn, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
I20240830 16:07:55.466375    25 ChunkManager.cpp:112] [SERVER][AwsChunkManager][milvus] init AwsChunkManager with parameter[endpoint=minio:9000][bucket_name=a-bucket][root_path=files][use_secure=false]
I20240830 16:07:55.466647    25 index_factory.cc:54] [KNOWHERE][Create][milvus] use key IVF_FLAT_fp32 to create knowhere index IVF_FLAT with version 4
I20240830 16:07:55.466815    25 ThreadPools.h:51] [SERVER][SetUpCoefficients][milvus] Init ThreadPools, high_priority_co=10, middle=5, low=1
I20240830 16:07:55.466892    25 ThreadPool.h:52] [SERVER][ThreadPool][milvus] Init thread pool:high_priority_thread_pool with min worker num:12 and max worker num:16
W20240830 16:07:55.492487    75 ivf.cc:319] [KNOWHERE][MatchNlist][knowhere_build0] nlist(128) is too large, adjust to a proper value
W20240830 16:07:55.492606    75 ivf.cc:321] [KNOWHERE][MatchNlist][knowhere_build0] Row num 1024 match nlist 26
I20240830 16:07:55.504073    25 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][milvus] Build index: done (11.932752 ms)
[2024/08/30 16:07:55.504 +00:00] [INFO] [indexnode/task.go:561] ["Successfully build index"] [buildID=452207326985853700] [Collection=452207326985453669] [SegmentID=452207326985653680] [currentIndexVersion=4]
I20240830 16:07:55.504436    25 ivf.cc:1071] [KNOWHERE][SerializeImpl][milvus] request version 4
I20240830 16:07:55.504729    25 ivf.cc:1077] [KNOWHERE][SerializeImpl][milvus] write IVF_FLAT, file size 16939
I20240830 16:07:55.504865    25 ThreadPool.h:52] [SERVER][ThreadPool][milvus] Init thread pool:middle_priority_thread_pool with min worker num:12 and max worker num:16
[2024/08/30 16:07:55.524 +00:00] [INFO] [indexnode/task.go:599] ["Successfully save index files"] [buildID=452207326985853700] [Collection=452207326985453669] [partition=452207326985453670] [SegmentId=452207326985653680]
[2024/08/30 16:07:55.594 +00:00] [INFO] [datacoord/services.go:1319] ["GetFlushState all flushed"] [traceID=94ffe8bcdeed24757ba41efb30e4e3d2] [collection=452207326985453669] [flushTs=2024/08/30 16:07:53.069 +00:00]
[2024/08/30 16:07:55.596 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669] [indexName=] [timestamp=0]
[2024/08/30 16:07:55.597 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=452207326985453669] [indexID=452207326985653683] [totalRows=1024] [indexRows=0] [pendingIndexRows=1024] [state=Finished] [failReason=]
[2024/08/30 16:07:55.597 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669] [indexName=]
[2024/08/30 16:07:55.597 +00:00] [INFO] [querycoordv2/services.go:196] ["load collection request received"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669] [replicaNumber=1] [resourceGroups="[]"] [refreshMode=false] [schema="name:\"Benchmark\" description:\"Benchmark\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"2\" > > "] [fieldIndexes="[452207326985653683]"]
[2024/08/30 16:07:55.597 +00:00] [INFO] [job/scheduler.go:150] ["start to pre-execute job"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669]
[2024/08/30 16:07:55.597 +00:00] [INFO] [job/scheduler.go:158] ["start to execute job"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669]
[2024/08/30 16:07:55.597 +00:00] [INFO] [meta/failed_load_cache.go:107] ["FailedLoadCache removes cache"] [collectionID=452207326985453669]
[2024/08/30 16:07:55.598 +00:00] [INFO] [job/job_load.go:139] ["find partitions to load"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669] [partitions="[452207326985453670]"]
[2024/08/30 16:07:55.602 +00:00] [INFO] [meta/replica_manager.go:357] ["new replica recovery found"] [replicaID=452207327133564929] [newRONodes="[]"] [roToRWNodes="[]"] [newIncomingNodes="[1]"]
[2024/08/30 16:07:55.604 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669]
[2024/08/30 16:07:55.606 +00:00] [INFO] [querynodev2/services.go:381] ["received load partitions request"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669] [partitionIDs="[452207326985453670]"]
[2024/08/30 16:07:55.606 +00:00] [INFO] [querynodev2/services.go:393] ["load partitions done"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669] [partitionIDs="[452207326985453670]"]
[2024/08/30 16:07:55.609 +00:00] [INFO] [observers/target_observer.go:153] ["manually trigger update next target"] [collectionID=452207326985453669]
[2024/08/30 16:07:55.609 +00:00] [INFO] [observers/target_observer.go:282] ["observer trigger update next target"] [collectionID=452207326985453669]
[2024/08/30 16:07:55.610 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=5dcbe30952093340cf194e5833b63d3e] [collectionID=452207326985453669] [partitionIDs="[]"]
[2024/08/30 16:07:55.610 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [numOfSegments=1] ["indexed segment"=0]
[2024/08/30 16:07:55.610 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207332116135939] [posTime=2024/08/30 16:07:53.319 +00:00]
[2024/08/30 16:07:55.610 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=5dcbe30952093340cf194e5833b63d3e] [collectionID=452207326985453669] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2024/08/30 16:07:55.610 +00:00] [INFO] [observers/target_observer.go:166] ["manually trigger update target done"] [collectionID=452207326985453669]
[2024/08/30 16:07:55.610 +00:00] [INFO] [observers/target_observer.go:168] ["notify manually trigger update target done"] [collectionID=452207326985453669]
[2024/08/30 16:07:55.611 +00:00] [INFO] [job/scheduler.go:144] ["start to post-execute job"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669]
[2024/08/30 16:07:55.611 +00:00] [INFO] [job/scheduler.go:146] ["job finished"] [traceID=0d636b56211c5e0df430df7eccd2ed8a] [collectionID=452207326985453669]
[2024/08/30 16:07:55.611 +00:00] [INFO] [querycoordv2/server.go:305] ["switch to new balancer"] [name=ScoreBasedBalancer]
[2024/08/30 16:07:55.611 +00:00] [INFO] [balance/utils.go:115] ["create channel task"] [collection=452207326985453669] [replica=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [from=-1] [to=1]
[2024/08/30 16:07:55.611 +00:00] [INFO] [task/scheduler.go:282] ["task added"] [task="[id=1725034054321] [type=Grow] [source=channel_checker] [reason=lacks of channel] [collectionID=452207326985453669] [replicaID=452207327133564929] [resourceGroup=__default_resource_group] [priority=High] [actionsCount=1] [actions={[type=Grow][node=1]}] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:55.614 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=e944d620682a616ce0bbb2bfc7963610] [collections="[452207326985453669]"]
[2024/08/30 16:07:55.733 +00:00] [INFO] [balance/utils.go:115] ["create channel task"] [collection=452207326985453669] [replica=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [from=-1] [to=1]
[2024/08/30 16:07:55.760 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=1]
[2024/08/30 16:07:55.762 +00:00] [INFO] [datacoord/index_builder.go:487] ["this task has been finished"] [buildID=452207326985853700] ["index state"=Finished]
[2024/08/30 16:07:55.764 +00:00] [INFO] [datacoord/index_meta.go:697] ["finish index task success"] [buildID=452207326985853700] [state=Finished] ["fail reason"=] [current_index_version=4]
[2024/08/30 16:07:55.819 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=0e0cc4a6e7127b77efce679a42e2a46c] [collections="[452207326985453669]"]
[2024/08/30 16:07:55.868 +00:00] [INFO] [task/scheduler.go:604] ["processed tasks"] [nodeID=1] [toProcessNum=1] [committedNum=1] [toRemoveNum=0]
[2024/08/30 16:07:55.868 +00:00] [INFO] [task/executor.go:120] ["execute the action of task"] [taskID=1725034054321] [collectionID=452207326985453669] [replicaID=452207327133564929] [step=0] [source=channel_checker]
[2024/08/30 16:07:55.868 +00:00] [INFO] [task/scheduler.go:610] ["process tasks related to node done"] [nodeID=1] [processingTaskNum=1] [waitingTaskNum=0] [segmentTaskNum=0] [channelTaskNum=1]
[2024/08/30 16:07:55.870 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669]
[2024/08/30 16:07:55.871 +00:00] [INFO] [task/executor.go:386] ["subscribe channel..."] [taskID=1725034054321] [collectionID=452207326985453669] [replicaID=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [node=1] [source=channel_checker] [checkpoint=452207332116135939] [sinceCheckpoint=2.552038976s]
[2024/08/30 16:07:55.873 +00:00] [INFO] [querynodev2/services.go:200] ["received watch channel request"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [currentNodeID=1] [version=1725034075871035377]
[2024/08/30 16:07:55.873 +00:00] [INFO] [segments/collection.go:95] ["put new collection"] [collectionID=452207326985453669] [schema="name:\"Benchmark\" description:\"Benchmark\" fields:<fieldID:100 name:\"id\" is_primary_key:true data_type:Int64 > fields:<fieldID:101 name:\"vector\" data_type:FloatVector type_params:<key:\"dim\" value:\"2\" > > fields:<name:\"RowID\" description:\"row id\" data_type:Int64 > fields:<fieldID:1 name:\"Timestamp\" description:\"time stamp\" data_type:Int64 > "]
I20240830 16:07:55.874210    36 Collection.cpp:64] [SERVER][parseIndexMeta][milvus] index meta info: maxRowCount : {4026531} FieldId : {101 IndexParams : { index_type : IVF_FLAT, metric_type : L2, nlist : 128,  }TypeParams : {dim : 2, }}
[2024/08/30 16:07:55.874 +00:00] [INFO] [delegator/delegator.go:864] ["Init delete cache with list delete buffer"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [replicaID=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [version=1725034075871035377] [startTs=452207332116135939] [sizePerBlock=1048576] [startTime=2024/08/30 16:07:53.319 +00:00]
[2024/08/30 16:07:55.874 +00:00] [INFO] [delegator/delegator.go:894] ["finish build new shardDelegator"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [replicaID=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [version=1725034075871035377] [startTs=452207332116135939]
[2024/08/30 16:07:55.874 +00:00] [WARN] [delegator/delegator.go:754] ["tsafeManager failed to get lastest"] [error="cannot found tSafer, vChannel = by-dev-rootcoord-dml_0_452207326985453669v0"]
[2024/08/30 16:07:55.874 +00:00] [INFO] [tsafe/manager.go:69] ["add tSafe done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [timestamp=2024/08/30 16:07:53.319 +00:00]
[2024/08/30 16:07:55.874 +00:00] [INFO] [pipeline/manager.go:66] ["start create pipeine"] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:55.875 +00:00] [INFO] [delegator/delegator_data.go:407] ["loading growing segments..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [replicaID=452207327133564929] [segmentIDs="[]"]
[2024/08/30 16:07:55.875 +00:00] [INFO] [segments/segment_loader.go:611] ["no segment to load"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentType=Growing]
[2024/08/30 16:07:55.875 +00:00] [INFO] [delegator/delegator_data.go:439] ["load growing segments done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [replicaID=452207327133564929] [segmentIDs="[]"]
[2024/08/30 16:07:55.875 +00:00] [INFO] [delegator/delegator_data.go:396] ["add growing segments to delegator"] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [replicaID=452207327133564929] [segmentIDs="[]"]
[2024/08/30 16:07:55.875 +00:00] [INFO] [msgdispatcher/manager.go:66] ["create new dispatcherManager"] [role=querynode] [nodeID=1] [pchannel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:55.875 +00:00] [INFO] [msgdispatcher/dispatcher.go:93] ["creating dispatcher..."] [pchannel=by-dev-rootcoord-dml_0] [subName=querynode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true] [isMain=true]
[2024/08/30 16:07:55.875 +00:00] [INFO] [msgdispatcher/manager.go:154] ["dispatcherManager is running..."] [role=querynode] [nodeID=1] [pchannel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:55.875 +00:00] [INFO] [msgstream/mq_msgstream.go:116] ["Msg Stream state"] [can_produce=true]
[2024/08/30 16:07:55.875 +00:00] [INFO] [msgstream/mq_msgstream.go:872] ["MsgStream begin to seek start msg: "] [channel=by-dev-rootcoord-dml_0] [MessageID="CgDEASuQRgY="]
[2024/08/30 16:07:55.875 +00:00] [INFO] [msgstream/mq_msgstream.go:882] ["MsgStream seek finished"] [channel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/dispatcher.go:113] ["seek successfully"] [pchannel=by-dev-rootcoord-dml_0] [subName=querynode-1-by-dev-rootcoord-dml_0_452207326985453669v0-true] [isMain=true] [posTime=2024/08/30 16:07:53.319 +00:00] [tsLag=2.557396535s]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/dispatcher.go:149] ["add new target"] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0] [isMain=true]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/manager.go:101] ["add main dispatcher"] [role=querynode] [nodeID=1] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/dispatcher.go:177] ["get signal"] [pchannel=by-dev-rootcoord-dml_0] [signal=start] [isMain=true]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/dispatcher.go:200] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_0] [signal=start] [isMain=true]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/client.go:85] ["register done"] [role=querynode] [nodeID=1] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:55.876 +00:00] [INFO] [pipeline/stream_pipeline.go:80] ["stream pipeline seeks from position with msgDispatcher"] [pchannel=by-dev-rootcoord-dml_0] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0] [checkpointTs=2024/08/30 16:07:53.319 +00:00] [tsLag=2.557580377s] [elapse=1.221335ms]
[2024/08/30 16:07:55.876 +00:00] [INFO] [msgdispatcher/dispatcher.go:205] ["begin to work"] [pchannel=by-dev-rootcoord-dml_0] [isMain=true]
[2024/08/30 16:07:55.876 +00:00] [INFO] [querynodev2/services.go:337] ["watch dml channel success"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [currentNodeID=1]
[2024/08/30 16:07:55.877 +00:00] [INFO] [task/executor.go:401] ["subscribe channel done"] [taskID=1725034054321] [collectionID=452207326985453669] [replicaID=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [node=1] [source=channel_checker] [taskID=1725034054321] ["time taken"=8.146385ms]
[2024/08/30 16:07:55.933 +00:00] [INFO] [balance/utils.go:115] ["create channel task"] [collection=452207326985453669] [replica=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [from=-1] [to=1]
[2024/08/30 16:07:55.968 +00:00] [INFO] [observers/leader_cache_observer.go:76] ["receive event, trigger leader cache update"] [event=452207326985453669]
[2024/08/30 16:07:55.971 +00:00] [INFO] [proxy/impl.go:188] ["received request to invalidate shard leader cache"] [traceID=ab92f4cde9a8f0cae00c3d8fbeaa90af] [module=Proxy] [role=proxy] [collectionIDs="[452207326985453669]"]
[2024/08/30 16:07:55.971 +00:00] [INFO] [proxy/meta_cache.go:1004] ["Invalidate shard cache for collections"] [collectionIDs="[452207326985453669]"]
[2024/08/30 16:07:55.971 +00:00] [INFO] [proxy/impl.go:193] ["complete to invalidate shard leader cache"] [traceID=ab92f4cde9a8f0cae00c3d8fbeaa90af] [module=Proxy] [role=proxy] [collectionIDs="[452207326985453669]"]
[2024/08/30 16:07:56.022 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=df9235b8c230d6124e8b5989b9bd3b07] [collections="[452207326985453669]"]
[2024/08/30 16:07:56.133 +00:00] [INFO] [observers/collection_observer.go:322] ["partition load progress"] [collectionID=452207326985453669] [partitionID=452207326985453670] [subChannelCount=1] [loadSegmentCount=0]
[2024/08/30 16:07:56.133 +00:00] [INFO] [observers/collection_observer.go:345] ["load status updated"] [collectionID=452207326985453669] [partitionID=452207326985453670] [partitionLoadPercentage=50] [collectionLoadPercentage=50]
[2024/08/30 16:07:56.134 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=452207326985453669] [segmentID=452207326985653680] [replica=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [level=L1] [from=-1] [to=1]
[2024/08/30 16:07:56.134 +00:00] [INFO] [task/scheduler.go:282] ["task added"] [task="[id=1725034054322] [type=Grow] [source=segment_checker] [reason=lacks of segment] [collectionID=452207326985453669] [replicaID=452207327133564929] [resourceGroup=__default_resource_group] [priority=Normal] [actionsCount=1] [actions={[type=Grow][node=1][streaming=false]}] [segmentID=452207326985653680]"]
[2024/08/30 16:07:56.226 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=0362182ce943ccd909b48d4e0f769235] [collections="[452207326985453669]"]
[2024/08/30 16:07:56.333 +00:00] [INFO] [observers/collection_observer.go:322] ["partition load progress"] [collectionID=452207326985453669] [partitionID=452207326985453670] [subChannelCount=1] [loadSegmentCount=0]
[2024/08/30 16:07:56.333 +00:00] [INFO] [balance/utils.go:70] ["create segment task"] [collection=452207326985453669] [segmentID=452207326985653680] [replica=452207327133564929] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [level=L1] [from=-1] [to=1]
[2024/08/30 16:07:56.368 +00:00] [INFO] [task/executor.go:120] ["execute the action of task"] [taskID=1725034054322] [collectionID=452207326985453669] [replicaID=452207327133564929] [step=0] [source=segment_checker]
[2024/08/30 16:07:56.368 +00:00] [INFO] [task/scheduler.go:813] ["task removed"] [taskID=1725034054321] [collectionID=452207326985453669] [replicaID=452207327133564929] [status=succeeded] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:56.368 +00:00] [INFO] [task/scheduler.go:604] ["processed tasks"] [nodeID=1] [toProcessNum=1] [committedNum=1] [toRemoveNum=1]
[2024/08/30 16:07:56.368 +00:00] [INFO] [task/scheduler.go:610] ["process tasks related to node done"] [nodeID=1] [processingTaskNum=1] [waitingTaskNum=0] [segmentTaskNum=1] [channelTaskNum=0]
[2024/08/30 16:07:56.373 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669]
[2024/08/30 16:07:56.373 +00:00] [INFO] [task/executor.go:228] ["load segments..."] [taskID=1725034054322] [collectionID=452207326985453669] [replicaID=452207327133564929] [segmentID=452207326985653680] [node=1] [source=segment_checker] [shardLeader=1]
[2024/08/30 16:07:56.375 +00:00] [INFO] [querynodev2/services.go:410] ["received load segments request"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentID=452207326985653680] [level=L1] [currentNodeID=1] [version=1725034076373724731] [needTransfer=true] [loadScope=Full]
[2024/08/30 16:07:56.375 +00:00] [INFO] [querynodev2/services.go:410] ["received load segments request"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentID=452207326985653680] [level=L1] [currentNodeID=1] [version=1725034076373724731] [needTransfer=false] [loadScope=Full]
[2024/08/30 16:07:56.375 +00:00] [INFO] [querynodev2/services.go:474] ["start to load segments..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentID=452207326985653680] [level=L1] [currentNodeID=1]
[2024/08/30 16:07:56.375 +00:00] [INFO] [segments/segment_loader.go:624] ["start loading..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentType=Sealed] [requestSegments="[452207326985653680]"] [preparedSegments="[452207326985653680]"] [segmentNum=1] [afterFilter=1]
[2024/08/30 16:07:56.380 +00:00] [INFO] [segments/segment_loader.go:1551] ["predict memory and disk usage while loading (in MiB)"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [maxSegmentSize(MB)=0.08255672454833984] [committedMemSize(MB)=0] [memLimit(MB)=48103.6328125] [memUsage(MB)=114.60546875] [committedDiskSize(MB)=0] [diskUsage(MB)=0] [predictMemUsage(MB)=114.68802547454834] [predictDiskUsage(MB)=0] [mmapFieldCount=0]
[2024/08/30 16:07:56.380 +00:00] [INFO] [segments/segment_loader.go:854] ["request resource for loading segments (unit in MiB)"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [segmentIDs="[452207326985653680]"] [memory=0.08255672454833984] [committedMemory=0.08255672454833984] [disk=0] [committedDisk=0]
[2024/08/30 16:07:56.381 +00:00] [INFO] [segments/segment.go:306] ["create segment"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [segmentType=Sealed] [level=L1]
[2024/08/30 16:07:56.381 +00:00] [INFO] [segments/segment_loader.go:727] ["start to load segments in parallel"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentType=Sealed] [requestSegments="[452207326985653680]"] [preparedSegments="[452207326985653680]"] [segmentNum=1] [concurrencyLevel=1]
[2024/08/30 16:07:56.381 +00:00] [INFO] [segments/segment_loader.go:700] ["load segment..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentType=Sealed] [requestSegments="[452207326985653680]"] [preparedSegments="[452207326985653680]"] [partitionID=452207326985453670] [segmentID=452207326985653680] [segmentType=L1]
[2024/08/30 16:07:56.381 +00:00] [INFO] [segments/segment_loader.go:1099] ["start loading segment files"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentID=452207326985653680] [rowNum=1024] [segmentType=Sealed]
[2024/08/30 16:07:56.382 +00:00] [INFO] [segments/segment.go:1156] ["add field data info done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] ["row count"=1024]
[2024/08/30 16:07:56.382 +00:00] [INFO] [segments/segment_loader.go:1040] ["Start loading fields..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [segmentID=452207326985653680] [indexedFields="[101]"]
[2024/08/30 16:07:56.382 +00:00] [INFO] [load_index_info.go:119] ["load with index params"] [indexParams="{\"dim\":\"2\",\"index_type\":\"IVF_FLAT\",\"metric_type\":\"L2\",\"nlist\":\"128\"}"]
I20240830 16:07:56.383231    26 load_index_c.cpp:235] [SERVER][AppendIndexV2][milvus] [collection=452207326985453669][segment=452207326985653680][field=101][enable_mmap=false] load index 452207326985653683
I20240830 16:07:56.383507    26 index_factory.cc:54] [KNOWHERE][Create][milvus] use key IVF_FLAT_fp32 to create knowhere index IVF_FLAT with version 4
I20240830 16:07:56.383644    26 VectorMemIndex.cpp:320] [SERVER][Load][milvus] load index files: 1
I20240830 16:07:56.383771    26 VectorMemIndex.cpp:343] [SERVER][Load][milvus] load with slice meta: false
I20240830 16:07:56.393317    26 VectorMemIndex.cpp:405] [SERVER][Load][milvus] construct binary set...
I20240830 16:07:56.393417    26 VectorMemIndex.cpp:408] [SERVER][Load][milvus] add index data to binary set: IVF_FLAT
I20240830 16:07:56.393473    26 VectorMemIndex.cpp:421] [SERVER][Load][milvus] load index into Knowhere...
I20240830 16:07:56.394021    26 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][milvus] Load index: done (0.276882 ms)
I20240830 16:07:56.394084    26 VectorMemIndex.cpp:424] [SERVER][Load][milvus] load vector index done
I20240830 16:07:56.394131    26 load_index_c.cpp:298] [SERVER][AppendIndexV2][milvus] [collection=452207326985453669][segment=452207326985653680][field=101][enable_mmap=false] load index 452207326985653683 done
I20240830 16:07:56.394399    31 SegmentSealedImpl.cpp:108] [SERVER][LoadVecIndex][milvus] Before setting field_bit for field index, fieldID:101. segmentID:452207326985653680, 
I20240830 16:07:56.394526    31 SegmentSealedImpl.cpp:125] [SERVER][LoadVecIndex][milvus] Has load vec index done, fieldID:101. segmentID:452207326985653680, 
[2024/08/30 16:07:56.394 +00:00] [INFO] [segments/segment.go:1345] ["updateSegmentIndex done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=101]
[2024/08/30 16:07:56.394 +00:00] [INFO] [segments/segment_loader.go:1254] ["load field binlogs done for sealed segment with index"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [rowCount=1024] [fieldID=101] [binlog="[{\"entries_num\":1024,\"timestamp_from\":452207331906682895,\"timestamp_to\":452207331906682895,\"log_path\":\"files/insert_log/452207326985453669/452207326985453670/452207326985653680/101/452207326985653690\",\"log_size\":9891,\"logID\":452207326985653690,\"memory_size\":8196}]"] [current_index_version=4] [load_duration=12.654046ms]
[2024/08/30 16:07:56.395 +00:00] [INFO] [segments/segment.go:963] ["start loading field data for field"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=1] [rowCount=1024]
[2024/08/30 16:07:56.395 +00:00] [INFO] [segments/segment.go:963] ["start loading field data for field"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=100] [rowCount=1024]
[2024/08/30 16:07:56.395 +00:00] [INFO] [segments/segment.go:963] ["start loading field data for field"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=0] [rowCount=1024]
[2024/08/30 16:07:56.395 +00:00] [INFO] [segments/segment.go:993] ["submitted loadFieldData task to load pool"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=100] [rowCount=1024]
[2024/08/30 16:07:56.395 +00:00] [INFO] [segments/segment.go:993] ["submitted loadFieldData task to load pool"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=0] [rowCount=1024]
[2024/08/30 16:07:56.395 +00:00] [INFO] [segments/segment.go:993] ["submitted loadFieldData task to load pool"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=1] [rowCount=1024]
I20240830 16:07:56.396189    26 SegmentSealedImpl.cpp:253] [SERVER][LoadFieldData][milvus] segment 452207326985653680 loads field 0 with num_rows 1024
I20240830 16:07:56.396296    35 SegmentSealedImpl.cpp:253] [SERVER][LoadFieldData][milvus] segment 452207326985653680 loads field 100 with num_rows 1024
I20240830 16:07:56.396319    26 SegmentSealedImpl.cpp:266] [SERVER][LoadFieldData][milvus] segment 452207326985653680 submits load field 0 task to thread pool
I20240830 16:07:56.396301    76 SegmentSealedImpl.cpp:253] [SERVER][LoadFieldData][milvus] segment 452207326985653680 loads field 1 with num_rows 1024
I20240830 16:07:56.396591    76 SegmentSealedImpl.cpp:266] [SERVER][LoadFieldData][milvus] segment 452207326985653680 submits load field 1 task to thread pool
I20240830 16:07:56.396567    35 SegmentSealedImpl.cpp:266] [SERVER][LoadFieldData][milvus] segment 452207326985653680 submits load field 100 task to thread pool
I20240830 16:07:56.406666    76 SegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][milvus] segment 452207326985653680 loads field 1 done
[2024/08/30 16:07:56.406 +00:00] [INFO] [segments/segment.go:1016] ["load field done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=1] [rowCount=1024]
I20240830 16:07:56.409180    26 SegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][milvus] segment 452207326985653680 loads field 0 done
[2024/08/30 16:07:56.409 +00:00] [INFO] [segments/segment.go:1016] ["load field done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=0] [rowCount=1024]
I20240830 16:07:56.409727    35 SegmentSealedImpl.cpp:275] [SERVER][LoadFieldData][milvus] segment 452207326985653680 loads field 100 done
[2024/08/30 16:07:56.409 +00:00] [INFO] [segments/segment.go:1016] ["load field done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [fieldID=100] [rowCount=1024]
[2024/08/30 16:07:56.410 +00:00] [INFO] [segments/segment_loader.go:1223] ["load field binlogs done for sealed segment"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collection=452207326985453669] [segment=452207326985653680] [len(field)=3] [segmentType=Sealed]
[2024/08/30 16:07:56.410 +00:00] [INFO] [segments/segment_loader.go:1080] ["Finish loading segment"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [segmentID=452207326985653680] [loadFieldsIndexSpan=12.844373ms] [complementScalarDataSpan=26.321µs] [loadRawDataSpan=15.099534ms] [patchEntryNumberSpan=5.122µs]
[2024/08/30 16:07:56.425 +00:00] [INFO] [segments/segment_loader.go:1357] ["loading delta..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [segmentID=452207326985653680] [deltaNum=0]
[2024/08/30 16:07:56.425 +00:00] [INFO] [segments/segment_loader.go:1392] ["there are no delta logs saved with segment, skip loading delete record"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [segmentID=452207326985653680] [deltaNum=0]
[2024/08/30 16:07:56.425 +00:00] [INFO] [segments/segment_loader.go:697] ["load segment done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentType=Sealed] [requestSegments="[452207326985653680]"] [preparedSegments="[452207326985653680]"] [partitionID=452207326985453670] [segmentID=452207326985653680] [segmentType=L1]
[2024/08/30 16:07:56.425 +00:00] [INFO] [segments/segment_loader.go:745] ["all segment load done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentType=Sealed] [requestSegments="[452207326985653680]"] [preparedSegments="[452207326985653680]"]
[2024/08/30 16:07:56.430 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=f3db40f41ff678acfd21f5f1f1676570] [collections="[452207326985453669]"]
[2024/08/30 16:07:56.436 +00:00] [INFO] [querynodev2/services.go:487] ["load segments done..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionID=452207326985453670] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentID=452207326985653680] [level=L1] [currentNodeID=1] [segments="[452207326985653680]"]
[2024/08/30 16:07:56.436 +00:00] [INFO] [segments/segment_loader.go:946] ["start loading remote..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentIDs="[452207326985653680]"] [segmentNum=1]
[2024/08/30 16:07:56.436 +00:00] [INFO] [segments/segment_loader.go:956] ["loading bloom filter for remote..."] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [segmentIDs="[452207326985653680]"]
[2024/08/30 16:07:56.444 +00:00] [INFO] [segments/segment_loader.go:1346] ["Successfully load pk stats"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [segmentID=452207326985653680] [time=7.710637ms] [size=14723]
[2024/08/30 16:07:56.444 +00:00] [INFO] [delegator/delegator_data.go:737] ["register sealed segment bfs into pko candidates"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [replicaID=452207327133564929] [segmentID=452207326985653680]
[2024/08/30 16:07:56.444 +00:00] [INFO] [delegator/delegator_data.go:742] ["load delete done"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [replicaID=452207327133564929]
[2024/08/30 16:07:56.445 +00:00] [INFO] [task/executor.go:237] ["load segments done"] [taskID=1725034054322] [collectionID=452207326985453669] [replicaID=452207327133564929] [segmentID=452207326985653680] [node=1] [source=segment_checker] [shardLeader=1] [elapsed=71.559041ms]
[2024/08/30 16:07:56.533 +00:00] [INFO] [observers/collection_observer.go:322] ["partition load progress"] [collectionID=452207326985453669] [partitionID=452207326985453670] [subChannelCount=1] [loadSegmentCount=1]
[2024/08/30 16:07:56.533 +00:00] [WARN] [observers/collection_observer.go:336] ["failed to manual check current target, skip update load status"] [collectionID=452207326985453669] [partitionID=452207326985453670]
[2024/08/30 16:07:56.536 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=6ffbc56b7cb8435cdc8a97c0ffbafb6b] [collectionID=452207326985453669]
[2024/08/30 16:07:56.537 +00:00] [INFO] [querynodev2/services.go:1306] ["sync action"] [traceID=df091da98090bf954d4e7579694dad5e] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [currentNodeID=1] [Action=UpdateVersion] [TargetVersion=1725034075609426231]
[2024/08/30 16:07:56.538 +00:00] [INFO] [delegator/distribution.go:299] ["Update readable segment version"] [oldVersion=0] [newVersion=1725034075609426231] [growingSegmentNum=0] [sealedSegmentNum=1]
[2024/08/30 16:07:56.538 +00:00] [INFO] [observers/target_observer.go:493] ["observer trigger update current target"] [collectionID=452207326985453669]
[2024/08/30 16:07:56.539 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=3aafb59b4cf3f8481dcf16b4ffba3140] [collectionID=452207326985453669] [partitionIDs="[]"]
[2024/08/30 16:07:56.539 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [numOfSegments=1] ["indexed segment"=1]
[2024/08/30 16:07:56.539 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [posTs=452207332116135939] [posTime=2024/08/30 16:07:53.319 +00:00]
[2024/08/30 16:07:56.539 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=3aafb59b4cf3f8481dcf16b4ffba3140] [collectionID=452207326985453669] [partitionIDs="[]"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] ["# of unflushed segments"=0] ["# of flushed segments"=1] ["# of dropped segments"=0] ["# of indexed segments"=0] ["# of l0 segments"=0]
[2024/08/30 16:07:56.634 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=7743e437c51443f7f0759a9b18e0a3f3] [collections="[452207326985453669]"]
[2024/08/30 16:07:56.733 +00:00] [INFO] [observers/collection_observer.go:322] ["partition load progress"] [collectionID=452207326985453669] [partitionID=452207326985453670] [subChannelCount=1] [loadSegmentCount=1]
[2024/08/30 16:07:56.736 +00:00] [INFO] [observers/collection_observer.go:345] ["load status updated"] [collectionID=452207326985453669] [partitionID=452207326985453670] [partitionLoadPercentage=100] [collectionLoadPercentage=100]
[2024/08/30 16:07:56.736 +00:00] [INFO] [observers/collection_observer.go:267] ["Load task finish"] [traceID=20a90b3073f1cb163bf0868e09ba3b91] [collectionID=452207326985453669] [partitionIDs="[]"] [loadType=LoadCollection]
[2024/08/30 16:07:56.760 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=1]
[2024/08/30 16:07:56.760 +00:00] [INFO] [indexnode/indexnode_service.go:199] ["drop index build jobs"] [traceID=ec591b4d7c4ba0031bf0ea22dfb1fc1b] [clusterID=by-dev] [indexBuildIDs="[452207326985853700]"]
[2024/08/30 16:07:56.760 +00:00] [INFO] [indexnode/taskinfo_ops.go:110] ["delete task infos"] [traceID=ec591b4d7c4ba0031bf0ea22dfb1fc1b] [cluster_id=by-dev] [build_id=452207326985853700]
[2024/08/30 16:07:56.760 +00:00] [INFO] [indexnode/indexnode_service.go:218] ["drop index build jobs success"] [traceID=ec591b4d7c4ba0031bf0ea22dfb1fc1b] [clusterID=by-dev] [indexBuildIDs="[452207326985853700]"]
[2024/08/30 16:07:56.761 +00:00] [INFO] [datacoord/index_builder.go:531] ["IndexCoord notify IndexNode drop the index task success"] [buildID=452207326985853700] [nodeID=1]
[2024/08/30 16:07:56.838 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=16507d5d45247ad30d2942b5e139c9dc] [collections="[452207326985453669]"]
[2024/08/30 16:07:56.843 +00:00] [INFO] [proxy/meta_cache.go:928] ["no shard cache for collection, try to get shard leaders from QueryCoord"] [traceID=2651b168eeaca6b9be1fec2a8d9b35f6] [collectionName=Benchmark] [collectionID=452207326985453669]
I20240830 16:07:56.855255   145 thread_pool.h:53] [KNOWHERE][operator()][knowhere_search] Successfully set priority of knowhere thread.
[2024/08/30 16:07:56.861 +00:00] [INFO] [querycoordv2/services.go:254] ["release collection request received"] [traceID=50560f56676b7b376b9d3918e594ae86] [collectionID=452207326985453669]
[2024/08/30 16:07:56.861 +00:00] [INFO] [job/scheduler.go:150] ["start to pre-execute job"] [traceID=50560f56676b7b376b9d3918e594ae86] [collectionID=452207326985453669]
[2024/08/30 16:07:56.861 +00:00] [INFO] [job/scheduler.go:158] ["start to execute job"] [traceID=50560f56676b7b376b9d3918e594ae86] [collectionID=452207326985453669]
[2024/08/30 16:07:56.862 +00:00] [INFO] [querynodev2/services.go:510] ["received release partitions request"] [traceID=50560f56676b7b376b9d3918e594ae86] [collection=452207326985453669] [partitions="[452207326985453670]"]
[2024/08/30 16:07:56.862 +00:00] [INFO] [segments/collection.go:198] ["remove partition"] [collection=452207326985453669] [partition=452207326985453670]
[2024/08/30 16:07:56.862 +00:00] [INFO] [querynodev2/services.go:525] ["release partitions done"] [traceID=50560f56676b7b376b9d3918e594ae86] [collection=452207326985453669] [partitions="[452207326985453670]"]
[2024/08/30 16:07:56.867 +00:00] [INFO] [meta/target_manager.go:281] ["remove collection from targets"] [collectionID=452207326985453669]
[2024/08/30 16:07:56.868 +00:00] [INFO] [job/utils.go:58] ["wait for release done"] [collection=452207326985453669] [partitions="[]"] [channel=1] [segments=1]
[2024/08/30 16:07:56.868 +00:00] [INFO] [task/scheduler.go:813] ["task removed"] [taskID=1725034054322] [collectionID=452207326985453669] [replicaID=452207327133564929] [status=succeeded] [segmentID=452207326985653680]
[2024/08/30 16:07:56.868 +00:00] [INFO] [task/scheduler.go:604] ["processed tasks"] [nodeID=1] [toProcessNum=0] [committedNum=0] [toRemoveNum=1]
[2024/08/30 16:07:56.868 +00:00] [INFO] [task/scheduler.go:610] ["process tasks related to node done"] [nodeID=1] [processingTaskNum=0] [waitingTaskNum=0] [segmentTaskNum=0] [channelTaskNum=0]
[2024/08/30 16:07:56.868 +00:00] [INFO] [task/scheduler.go:282] ["task added"] [task="[id=1725034054323] [type=Reduce] [source=channel_checker] [reason=collection released] [collectionID=452207326985453669] [replicaID=-1] [resourceGroup=] [priority=Normal] [actionsCount=1] [actions={[type=Reduce][node=1]}] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:56.868 +00:00] [INFO] [task/scheduler.go:282] ["task added"] [task="[id=1725034054324] [type=Reduce] [source=segment_checker] [reason=collection released] [collectionID=452207326985453669] [replicaID=-1] [resourceGroup=] [priority=Normal] [actionsCount=1] [actions={[type=Reduce][node=1][streaming=false]}] [segmentID=452207326985653680]"]
[2024/08/30 16:07:57.069 +00:00] [INFO] [job/utils.go:58] ["wait for release done"] [collection=452207326985453669] [partitions="[]"] [channel=1] [segments=1]
[2024/08/30 16:07:57.269 +00:00] [INFO] [job/utils.go:58] ["wait for release done"] [collection=452207326985453669] [partitions="[]"] [channel=1] [segments=1]
[2024/08/30 16:07:57.368 +00:00] [INFO] [task/executor.go:120] ["execute the action of task"] [taskID=1725034054324] [collectionID=452207326985453669] [replicaID=-1] [step=0] [source=segment_checker]
[2024/08/30 16:07:57.368 +00:00] [INFO] [task/executor.go:298] ["release segment..."] [taskID=1725034054324] [collectionID=452207326985453669] [replicaID=-1] [segmentID=452207326985653680] [node=1] [source=segment_checker]
[2024/08/30 16:07:57.368 +00:00] [INFO] [task/scheduler.go:604] ["processed tasks"] [nodeID=1] [toProcessNum=2] [committedNum=2] [toRemoveNum=0]
[2024/08/30 16:07:57.368 +00:00] [INFO] [task/executor.go:120] ["execute the action of task"] [taskID=1725034054323] [collectionID=452207326985453669] [replicaID=-1] [step=0] [source=channel_checker]
[2024/08/30 16:07:57.368 +00:00] [INFO] [task/scheduler.go:610] ["process tasks related to node done"] [nodeID=1] [processingTaskNum=2] [waitingTaskNum=0] [segmentTaskNum=1] [channelTaskNum=1]
[2024/08/30 16:07:57.368 +00:00] [INFO] [task/executor.go:436] ["unsubscribe channel..."] [taskID=1725034054323] [collectionID=452207326985453669] [replicaID=-1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [node=1] [source=channel_checker]
[2024/08/30 16:07:57.369 +00:00] [INFO] [querynodev2/services.go:348] ["received unsubscribe channel request"] [traceID=933268ddc01f22f9998834f7015c5610] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [currentNodeID=1]
[2024/08/30 16:07:57.369 +00:00] [INFO] [delegator/delegator.go:742] ["updateTSafe quit"] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [replicaID=452207327133564929]
[2024/08/30 16:07:57.369 +00:00] [INFO] [querynodev2/services.go:538] ["received release segment request"] [traceID=86836f1754dcda81ab63cecddb95b877] [collectionID=452207326985453669] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentIDs="[452207326985653680]"] [currentNodeID=1] [scope=Historical] [needTransfer=false]
[2024/08/30 16:07:57.369 +00:00] [INFO] [msgdispatcher/dispatcher.go:177] ["get signal"] [pchannel=by-dev-rootcoord-dml_0] [signal=pause] [isMain=true]
[2024/08/30 16:07:57.369 +00:00] [INFO] [querynodev2/services.go:568] ["start to release segments"] [traceID=86836f1754dcda81ab63cecddb95b877] [collectionID=452207326985453669] [shard=by-dev-rootcoord-dml_0_452207326985453669v0] [segmentIDs="[452207326985653680]"] [currentNodeID=1]
[2024/08/30 16:07:57.369 +00:00] [INFO] [msgdispatcher/dispatcher.go:210] ["stop working"] [pchannel=by-dev-rootcoord-dml_0] [isMain=true]
[2024/08/30 16:07:57.369 +00:00] [INFO] [msgdispatcher/dispatcher.go:200] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_0] [signal=pause] [isMain=true]
[2024/08/30 16:07:57.369 +00:00] [INFO] [segments/manager.go:780] ["remove segment from cache"] [traceID=86836f1754dcda81ab63cecddb95b877] [segmentID=452207326985653680]
[2024/08/30 16:07:57.369 +00:00] [INFO] [msgdispatcher/dispatcher.go:164] ["closed target"] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0] [isMain=true]
[2024/08/30 16:07:57.369 +00:00] [INFO] [msgdispatcher/dispatcher.go:177] ["get signal"] [pchannel=by-dev-rootcoord-dml_0] [signal=terminate] [isMain=true]
[2024/08/30 16:07:57.369 +00:00] [INFO] [msgstream/mq_msgstream.go:216] ["start to close mq msg stream"] ["producer num"=0] ["consumer num"=1]
[2024/08/30 16:07:57.370 +00:00] [INFO] [msgdispatcher/dispatcher.go:200] ["handle signal done"] [pchannel=by-dev-rootcoord-dml_0] [signal=terminate] [isMain=true]
[2024/08/30 16:07:57.370 +00:00] [INFO] [msgdispatcher/client.go:99] ["deregister done"] [role=querynode] [nodeID=1] [vchannel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:57.370 +00:00] [INFO] [msgdispatcher/manager.go:162] ["dispatcherManager exited"] [role=querynode] [nodeID=1] [pchannel=by-dev-rootcoord-dml_0]
[2024/08/30 16:07:57.370 +00:00] [INFO] [tsafe/manager.go:107] ["remove tSafe replica"] [traceID=933268ddc01f22f9998834f7015c5610] [vChannel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:57.370 +00:00] [INFO] [querynodev2/services.go:370] ["unsubscribed channel"] [traceID=933268ddc01f22f9998834f7015c5610] [collectionID=452207326985453669] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [currentNodeID=1]
[2024/08/30 16:07:57.370 +00:00] [INFO] [segments/segment.go:1482] ["delete segment from memory"] [traceID=86836f1754dcda81ab63cecddb95b877] [collectionID=452207326985453669] [partitionID=452207326985453670] [segmentID=452207326985653680] [segmentType=Sealed] [insertCount=1024]
[2024/08/30 16:07:57.370 +00:00] [INFO] [segments/collection.go:119] ["release collection due to ref count to 0"] [collectionID=452207326985453669]
[2024/08/30 16:07:57.370 +00:00] [INFO] [task/executor.go:449] ["unsubscribe channel done"] [taskID=1725034054323] [collectionID=452207326985453669] [replicaID=-1] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [node=1] [source=channel_checker] [taskID=1725034054323] ["time taken"=2.155658ms]
[2024/08/30 16:07:57.370 +00:00] [INFO] [task/executor.go:306] ["release segment done"] [taskID=1725034054324] [collectionID=452207326985453669] [replicaID=-1] [segmentID=452207326985653680] [node=1] [source=segment_checker] [taskID=1725034054324] ["time taken"=2.431104ms]
[2024/08/30 16:07:57.468 +00:00] [INFO] [observers/leader_cache_observer.go:76] ["receive event, trigger leader cache update"] [event=452207326985453669]
[2024/08/30 16:07:57.469 +00:00] [INFO] [proxy/impl.go:188] ["received request to invalidate shard leader cache"] [traceID=e8b409601f4efb25db5d24bf236ebb89] [module=Proxy] [role=proxy] [collectionIDs="[452207326985453669]"]
[2024/08/30 16:07:57.469 +00:00] [INFO] [proxy/meta_cache.go:1004] ["Invalidate shard cache for collections"] [collectionIDs="[452207326985453669]"]
[2024/08/30 16:07:57.469 +00:00] [INFO] [proxy/impl.go:193] ["complete to invalidate shard leader cache"] [traceID=e8b409601f4efb25db5d24bf236ebb89] [module=Proxy] [role=proxy] [collectionIDs="[452207326985453669]"]
[2024/08/30 16:07:57.470 +00:00] [INFO] [job/scheduler.go:144] ["start to post-execute job"] [traceID=50560f56676b7b376b9d3918e594ae86] [collectionID=452207326985453669]
[2024/08/30 16:07:57.470 +00:00] [INFO] [job/scheduler.go:146] ["job finished"] [traceID=50560f56676b7b376b9d3918e594ae86] [collectionID=452207326985453669]
[2024/08/30 16:07:57.470 +00:00] [INFO] [querycoordv2/services.go:283] ["collection released"] [traceID=50560f56676b7b376b9d3918e594ae86] [collectionID=452207326985453669]
[2024/08/30 16:07:57.470 +00:00] [INFO] [meta/failed_load_cache.go:107] ["FailedLoadCache removes cache"] [collectionID=452207326985453669]
[2024/08/30 16:07:57.471 +00:00] [INFO] [proxy/meta_cache.go:997] ["clearing shard cache for collection"] [collectionName=Benchmark]
[2024/08/30 16:07:57.475 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=1107dc95bd3aed5f4fa7d51a4fb3c197] [collectionID=452207326985453669] [indexName=] [timestamp=0]
[2024/08/30 16:07:57.475 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=452207326985453669] [indexID=452207326985653683] [totalRows=1024] [indexRows=1024] [pendingIndexRows=0] [state=Finished] [failReason=]
[2024/08/30 16:07:57.475 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=1107dc95bd3aed5f4fa7d51a4fb3c197] [collectionID=452207326985453669] [indexName=]
[2024/08/30 16:07:57.477 +00:00] [INFO] [proxy/impl.go:2286] ["DropIndex received"] [traceID=4fd25374274919408b3ee7172b903621] [role=proxy] [db=default] [collection=Benchmark] [field=vector] ["index name"=]
[2024/08/30 16:07:57.479 +00:00] [INFO] [querycoordv2/services.go:56] ["show collections request received"] [traceID=4fd25374274919408b3ee7172b903621] [collections="[]"]
[2024/08/30 16:07:57.479 +00:00] [INFO] [proxy/task_index.go:920] ["proxy drop index"] [traceID=4fd25374274919408b3ee7172b903621] [collID=452207326985453669] [field_name=vector] [index_name=] [db_name=default]
[2024/08/30 16:07:57.480 +00:00] [INFO] [datacoord/index_service.go:800] ["receive DropIndex request"] [traceID=4fd25374274919408b3ee7172b903621] [collectionID=452207326985453669] [partitionIDs="[]"] [indexName=] ["drop all indexes"=false]
[2024/08/30 16:07:57.480 +00:00] [INFO] [datacoord/index_meta.go:461] ["IndexCoord metaTable MarkIndexAsDeleted"] [collectionID=452207326985453669] [indexIDs="[452207326985653683]"]
[2024/08/30 16:07:57.482 +00:00] [INFO] [datacoord/index_meta.go:493] ["IndexCoord metaTable MarkIndexAsDeleted success"] [collectionID=452207326985453669] [indexIDs="[452207326985653683]"]
[2024/08/30 16:07:57.483 +00:00] [INFO] [proxy/impl.go:2316] ["DropIndex done"] [traceID=4fd25374274919408b3ee7172b903621] [role=proxy] [db=default] [collection=Benchmark] [field=vector] ["index name"=] [BeginTs=452207333204033538] [EndTs=452207333204033538]
[2024/08/30 16:07:57.487 +00:00] [INFO] [datacoord/services.go:74] ["receive flush request"] [traceID=fb0ab99a002c2f53269e0af740f1e352] [dbID=0] [collectionID=452207326985453669]
[2024/08/30 16:07:57.488 +00:00] [INFO] [datacoord/session_manager.go:308] ["SessionManagerImpl.FlushChannels start"] [traceID=fb0ab99a002c2f53269e0af740f1e352] [nodeID=1] [flushTs=2024/08/30 16:07:57.469 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:57.488 +00:00] [INFO] [datanode/services.go:389] ["DataNode receives FlushChannels request"] [traceID=fb0ab99a002c2f53269e0af740f1e352] [nodeId=1] [flushTs=452207333204033540] ["flushTs in Time"=2024/08/30 16:07:57.469 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:57.489 +00:00] [INFO] [datanode/services.go:404] ["success to FlushChannels"] [traceID=fb0ab99a002c2f53269e0af740f1e352] [nodeId=1] [flushTs=452207333204033540] ["flushTs in Time"=2024/08/30 16:07:57.469 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:57.489 +00:00] [INFO] [datacoord/session_manager.go:315] ["SessionManagerImpl.FlushChannels successfully"] [traceID=fb0ab99a002c2f53269e0af740f1e352] [nodeID=1] [flushTs=2024/08/30 16:07:57.469 +00:00] [channels="[by-dev-rootcoord-dml_0_452207326985453669v0]"]
[2024/08/30 16:07:57.489 +00:00] [INFO] [datacoord/services.go:166] ["flush response with segments"] [traceID=fb0ab99a002c2f53269e0af740f1e352] [dbID=0] [collectionID=452207326985453669] [collectionID=452207326985453669] [sealSegments="[]"] [flushedSegmentsCount=1] [timeOfSeal=2024/08/30 16:07:57.469 +00:00] [flushTs=2024/08/30 16:07:57.469 +00:00]
[2024/08/30 16:07:57.492 +00:00] [INFO] [datacoord/services.go:1311] ["GetFlushState failed, channel unflushed"] [traceID=a4356f8686ce56f82aa970a870fb1f83] [collection=452207326985453669] [flushTs=2024/08/30 16:07:57.469 +00:00] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [CP=2024/08/30 16:07:53.319 +00:00] [lag=4.15s]
[2024/08/30 16:07:57.774 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=7c31a9c479bac653e1562fe9a12b78a8] [collectionID=452207326985453669] [indexName=] [timestamp=0]
[2024/08/30 16:07:57.774 +00:00] [WARN] [datacoord/index_service.go:696] ["DescribeIndex fail"] [traceID=7c31a9c479bac653e1562fe9a12b78a8] [collectionID=452207326985453669] [indexName=] [error="index not found[indexName=]"]
[2024/08/30 16:07:57.774 +00:00] [WARN] [datacoord/metrics_info.go:69] ["failed to describe index, ignore to report index metrics"] [traceID=7c31a9c479bac653e1562fe9a12b78a8] [collection=452207326985453669] [error="index not found[indexName=]"]
[2024/08/30 16:07:57.826 +00:00] [INFO] [datacoord/meta.go:1446] ["UpdateChannelCheckpoint done"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0] [ts=452207333269569537] [time=2024/08/30 16:07:57.719 +00:00]
[2024/08/30 16:07:57.827 +00:00] [INFO] [writebuffer/manager.go:227] ["reset channel flushTs"] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:57.868 +00:00] [INFO] [task/scheduler.go:813] ["task removed"] [taskID=1725034054323] [collectionID=452207326985453669] [replicaID=-1] [status=succeeded] [channel=by-dev-rootcoord-dml_0_452207326985453669v0]
[2024/08/30 16:07:57.868 +00:00] [INFO] [task/scheduler.go:813] ["task removed"] [taskID=1725034054324] [collectionID=452207326985453669] [replicaID=-1] [status=succeeded] [segmentID=452207326985653680]
[2024/08/30 16:07:57.868 +00:00] [INFO] [task/scheduler.go:604] ["processed tasks"] [nodeID=1] [toProcessNum=0] [committedNum=0] [toRemoveNum=2]
[2024/08/30 16:07:57.868 +00:00] [INFO] [task/scheduler.go:610] ["process tasks related to node done"] [nodeID=1] [processingTaskNum=0] [waitingTaskNum=0] [segmentTaskNum=0] [channelTaskNum=0]
[2024/08/30 16:07:57.996 +00:00] [INFO] [datacoord/services.go:1319] ["GetFlushState all flushed"] [traceID=64054132052ade0b122b471379251ca1] [collection=452207326985453669] [flushTs=2024/08/30 16:07:57.469 +00:00]
[2024/08/30 16:07:58.003 +00:00] [INFO] [proxy/impl.go:1996] ["CreateIndex received"] [traceID=ef54e532f0b7e17d93cc209611f9c801] [role=proxy] [db=default] [collection=Benchmark] [field=vector] [extra_params="[{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"]
[2024/08/30 16:07:58.004 +00:00] [INFO] [proxy/impl.go:2009] ["CreateIndex enqueued"] [traceID=ef54e532f0b7e17d93cc209611f9c801] [role=proxy] [db=default] [collection=Benchmark] [field=vector] [extra_params="[{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [BeginTs=452207333335105539] [EndTs=452207333335105539]
[2024/08/30 16:07:58.004 +00:00] [INFO] [proxy/task_index.go:438] ["proxy create index"] [traceID=ef54e532f0b7e17d93cc209611f9c801] [collectionID=452207326985453669] [fieldID=101] [indexName=] [typeParams="[{\"key\":\"dim\",\"value\":\"2\"}]"] [indexParams="[{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [newExtraParams="[{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"]
[2024/08/30 16:07:58.005 +00:00] [INFO] [datacoord/index_service.go:176] ["receive CreateIndex request"] [traceID=ef54e532f0b7e17d93cc209611f9c801] [collectionID=452207326985453669] [IndexName=] [fieldID=101] [TypeParams="[{\"key\":\"dim\",\"value\":\"2\"}]"] [IndexParams="[{\"key\":\"cache_dataset_on_device\",\"value\":\"false\"},{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"nlist\",\"value\":\"128\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"]
[2024/08/30 16:07:58.007 +00:00] [INFO] [datacoord/index_meta.go:274] ["meta update: CreateIndex"] [collectionID=452207326985453669] [fieldID=101] [indexID=452207326986853722] [indexName=vector]
[2024/08/30 16:07:58.009 +00:00] [INFO] [datacoord/index_meta.go:287] ["meta update: CreateIndex success"] [collectionID=452207326985453669] [fieldID=101] [indexID=452207326986853722] [indexName=vector]
[2024/08/30 16:07:58.009 +00:00] [INFO] [datacoord/index_service.go:262] ["CreateIndex successfully"] [traceID=ef54e532f0b7e17d93cc209611f9c801] [collectionID=452207326985453669] [IndexName=vector] [fieldID=101] [IndexID=452207326986853722]
[2024/08/30 16:07:58.009 +00:00] [INFO] [datacoord/index_service.go:129] ["receive create index notify"] [collectionID=452207326985453669]
[2024/08/30 16:07:58.010 +00:00] [INFO] [datacoord/index_service.go:58] ["create index for segment"] [segmentID=452207326985653680] [indexID=452207326986853722]
[2024/08/30 16:07:58.010 +00:00] [INFO] [proxy/impl.go:2027] ["CreateIndex done"] [traceID=ef54e532f0b7e17d93cc209611f9c801] [role=proxy] [db=default] [collection=Benchmark] [field=vector] [extra_params="[{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"params\",\"value\":\"{\\\"nlist\\\":128}\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [BeginTs=452207333335105539] [EndTs=452207333335105539]
[2024/08/30 16:07:58.011 +00:00] [INFO] [datacoord/index_meta.go:314] ["meta update: adding segment index"] [collectionID=452207326985453669] [segmentID=452207326985653680] [indexID=452207326986853722] [buildID=452207326986853723]
[2024/08/30 16:07:58.012 +00:00] [INFO] [datacoord/index_meta.go:326] ["meta update: adding segment index success"] [collectionID=452207326985453669] [segmentID=452207326985653680] [indexID=452207326986853722] [buildID=452207326986853723]
[2024/08/30 16:07:58.012 +00:00] [INFO] [datacoord/index_builder.go:160] ["indexBuilder enqueue task"] [buildID=452207326986853723]
[2024/08/30 16:07:58.012 +00:00] [INFO] [datacoord/index_builder.go:194] ["index builder task schedule"] ["task num"=1]
[2024/08/30 16:07:58.012 +00:00] [INFO] [proxy/impl.go:6009] ["AllocTimestamp request receive"]
[2024/08/30 16:07:58.013 +00:00] [INFO] [indexnode/indexnode_service.go:242] ["Get Index Job Stats"] [traceID=55cefe60dfe09fb606fabd50773d492e] [unissued=0] [active=0] [slot=1]
[2024/08/30 16:07:58.013 +00:00] [INFO] [proxy/impl.go:6018] ["AllocTimestamp request success"] [timestamp=452207333335105541]
[2024/08/30 16:07:58.013 +00:00] [INFO] [datacoord/indexnode_manager.go:135] ["peek client success"] [nodeID=1]
[2024/08/30 16:07:58.014 +00:00] [INFO] [datacoord/indexnode_manager.go:148] ["peek client success"] [nodeID=1]
[2024/08/30 16:07:58.016 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=4]
[2024/08/30 16:07:58.016 +00:00] [INFO] [indexnode/indexnode_service.go:56] ["IndexNode building index ..."] [traceID=1e4c57b885b061b21258bc103860e6b0] [clusterID=by-dev] [indexBuildID=452207326986853723] [collectionID=452207326985453669] [indexID=0] [indexName=] [indexFilePrefix=files/index_files] [indexVersion=1] [dataPaths="[]"] [typeParams="[{\"key\":\"dim\",\"value\":\"2\"}]"] [indexParams="[{\"key\":\"cache_dataset_on_device\",\"value\":\"false\"},{\"key\":\"index_type\",\"value\":\"GPU_IVF_FLAT\"},{\"key\":\"nlist\",\"value\":\"128\"},{\"key\":\"metric_type\",\"value\":\"L2\"}]"] [numRows=1024] [current_index_version=4] [storepath=] [storeversion=0] [indexstorepath=] [dim=0]
[2024/08/30 16:07:58.022 +00:00] [INFO] [storage/remote_chunk_manager.go:92] ["remote chunk manager init success."] [remote=aws] [bucketname=a-bucket] [root=files]
[2024/08/30 16:07:58.022 +00:00] [INFO] [indexnode/task.go:318] ["IndexNode IndexBuilderTask Enqueue"] [buildID=452207326986853723] [segmentID=0]
[2024/08/30 16:07:58.022 +00:00] [INFO] [indexnode/indexnode_service.go:140] ["IndexNode successfully scheduled"] [traceID=1e4c57b885b061b21258bc103860e6b0] [clusterID=by-dev] [indexBuildID=452207326986853723] [indexName=]
[2024/08/30 16:07:58.022 +00:00] [INFO] [indexnode/task.go:324] ["Begin to prepare indexBuildTask"] [buildID=452207326986853723] [Collection=0] [SegmentID=0]
[2024/08/30 16:07:58.022 +00:00] [INFO] [indexnode/task.go:375] ["Successfully prepare indexBuildTask"] [buildID=452207326986853723] [Collection=0] [SegmentID=0]
[2024/08/30 16:07:58.023 +00:00] [INFO] [datacoord/index_builder.go:433] ["index task assigned successfully"] [buildID=452207326986853723] [segmentID=452207326985653680] [nodeID=1]
[2024/08/30 16:07:58.024 +00:00] [INFO] [datacoord/index_meta.go:753] ["meta update: segment index in progress success"] [buildID=452207326986853723] [segmentID=452207326985653680]
[2024/08/30 16:07:58.032 +00:00] [INFO] [indexnode/task.go:516] ["index params are ready"] [buildID=452207326986853723] ["index params"="{\"cache_dataset_on_device\":\"false\",\"dim\":\"2\",\"index_type\":\"GPU_IVF_FLAT\",\"metric_type\":\"L2\",\"nlist\":\"128\"}"]
I20240830 16:07:58.033087   143 MinioChunkManager.cpp:225] [SERVER][PreCheck][milvus] start to precheck chunk manager with configuration: [address=minio:9000, bucket_name=a-bucket, root_path=files, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=warn, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000]
I20240830 16:07:58.050968   143 ChunkManager.cpp:112] [SERVER][AwsChunkManager][milvus] init AwsChunkManager with parameter[endpoint=minio:9000][bucket_name=a-bucket][root_path=files][use_secure=false]
I20240830 16:07:58.051079   143 index_factory.cc:54] [KNOWHERE][Create][milvus] use key GPU_IVF_FLAT_fp32 to create knowhere index GPU_IVF_FLAT with version 4
I20240830 16:07:58.110211   143 time_recorder.cc:49] [KNOWHERE][PrintTimeRecord][milvus] Build index: done (50.275772 ms)
 => failed to build index, raft inner error
qwevdb commented 1 week ago

@yanliang567 Is the Milvus log I submitted acceptable?