Closed del-zhenwu closed 3 years ago
Sometimes the client is frozen during loading operations, only sift1m dataset need to be loaded As the following monitor result shows, the server is not receive any data, and memory not ricing after 4min, but the client get no response in 20 mins.
[2021-07-15 07:51:04,724] [ DEBUG] - Milvus insert run in 1.14s (milvus_benchmark.client:49) [2021-07-15 07:51:06,281] [ DEBUG] - Milvus insert run in 1.13s (milvus_benchmark.client:49) [2021-07-15 07:51:07,987] [ DEBUG] - Milvus insert run in 1.14s (milvus_benchmark.client:49) [2021-07-15 07:51:07,992] [ DEBUG] - End insert, start flush (milvus_benchmark.runners.accuracy:236) [2021-07-15 07:51:10,522] [ DEBUG] - Milvus flush run in 2.53s (milvus_benchmark.client:49) [2021-07-15 07:51:10,523] [ DEBUG] - End flush (milvus_benchmark.runners.accuracy:238) [2021-07-15 07:51:10,525] [ DEBUG] - Row count: 1000000 in collection: <sift_128_euclidean> (milvus_benchmark.client:389) [2021-07-15 07:51:10,525] [ INFO] - Table: sift_128_euclidean, row count: 1000000 (milvus_benchmark.runners.accuracy:240) [2021-07-15 07:51:10,527] [ INFO] - None (milvus_benchmark.client:276) [2021-07-15 07:51:10,528] [ INFO] - Drop index: sift_128_euclidean (milvus_benchmark.client:287) [2021-07-15 07:51:10,530] [ INFO] - Re-create index: sift_128_euclidean (milvus_benchmark.runners.accuracy:245) [2021-07-15 07:51:10,530] [ INFO] - Building index start, collection_name: sift_128_euclidean, index_type: IVF_FLAT, metric_type: L2 (milvus_benchmark.client:261) [2021-07-15 07:51:10,530] [ INFO] - {'nlist': 1024} (milvus_benchmark.client:263) [2021-07-15 07:51:49,378] [ DEBUG] - Milvus create_index run in 38.85s (milvus_benchmark.client:49) [2021-07-15 07:51:49,379] [ INFO] - None (milvus_benchmark.client:276) [2021-07-15 07:51:49,380] [ INFO] - {'index_type': 'flat', 'metric_type': None, 'index_param': None} (milvus_benchmark.runners.ac curacy:247) [2021-07-15 07:51:49,380] [ INFO] - Start load collection: sift_128_euclidean (milvus_benchmark.runners.accuracy:248)
logs:
https://kibana-dev.zilliz.cc/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2021-07-15T07:30:00.000Z',to:'2021-07-15T08:00:00.000Z'))&_a=(columns:!(log),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:ad1e1c50-c9d9-11eb-9e1a-b3dcad9693fd,key:kubernetes.pod_name,negate:!f,params:!(benchmark-2mdch-1-milvus-standalone-576ff4fc7b-ftz7q),type:phrases,value:benchmark-2mdch-1-milvus-standalone-576ff4fc7b-ftz7q),query:(bool:(minimum_should_match:1,should:!((match_phrase:(kubernetes.pod_name:benchmark-2mdch-1-milvus-standalone-576ff4fc7b-ftz7q))))))),hideChart:!f,index:ad1e1c50-c9d9-11eb-9e1a-b3dcad9693fd,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))
load operation will be finished in 3/4 min as expected
/kind bug
another issue: the given timeout set 3000s, but still no response after 1h passed
duplicate,close for now
Describe the bug:
Sometimes the client is frozen during loading operations, only sift1m dataset need to be loaded As the following monitor result shows, the server is not receive any data, and memory not ricing after 4min, but the client get no response in 20 mins.
logs:
https://kibana-dev.zilliz.cc/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2021-07-15T07:30:00.000Z',to:'2021-07-15T08:00:00.000Z'))&_a=(columns:!(log),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:ad1e1c50-c9d9-11eb-9e1a-b3dcad9693fd,key:kubernetes.pod_name,negate:!f,params:!(benchmark-2mdch-1-milvus-standalone-576ff4fc7b-ftz7q),type:phrases,value:benchmark-2mdch-1-milvus-standalone-576ff4fc7b-ftz7q),query:(bool:(minimum_should_match:1,should:!((match_phrase:(kubernetes.pod_name:benchmark-2mdch-1-milvus-standalone-576ff4fc7b-ftz7q))))))),hideChart:!f,index:ad1e1c50-c9d9-11eb-9e1a-b3dcad9693fd,interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))
Expected behavior:
load operation will be finished in 3/4 min as expected
Steps/Code to reproduce:
Environment:
Configuration file:
Additional context: