search

milvus-io / milvus-lite

A lightweight version of Milvus
Apache License 2.0
269 stars 30 forks source link

unable to run default_server on AMD #79

Open tommykoctur opened 9 months ago

tommykoctur commented 9 months ago

Hi,

I would like to use milvus-lite to test our indexing pipeline in gitlab CICD pipeline. We are using AMD runners. I don't know what is the issue but It crashes on AVX512.

Can you please help me ?

Thanks

[2024/01/05 14:11:15.416 +00:00] [WARN] [sessionutil/session_util.go:296] ["Session Txn unsuccessful"] [key=id] [2024/01/05 14:11:15.416 +00:00] [INFO] [sessionutil/session_util.go:299] ["Session get serverID success"] [key=id] [ServerId=1] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max send size"] [role=datacoord] [grpc.clientMaxSendSize=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init client max recv size"] [role=datacoord] [grpc.clientMaxRecvSize=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init dial timeout"] [role=datacoord] [grpc.client.dialTimeout=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive timeout"] [role=datacoord] [grpc.client.keepAliveTimeout=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init keep alive time"] [role=datacoord] [grpc.client.keepAliveTime=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max attempts"] [role=datacoord] [grpc.client.maxMaxAttempts=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init initial back off"] [role=datacoord] [grpc.client.initialBackOff=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init max back off"] [role=datacoord] [grpc.client.maxBackoff=104857600] [2024/01/05 14:11:15.417 +00:00] [DEBUG] [paramtable/grpc_param.go:238] ["Init back off multiplier"] [role=datacoord] [grpc.client.backoffMultiplier=104857600] [2024/01/05 14:11:15.417 +00:00] [INFO] [rootcoord/service.go:209] ["RootCoord start to create IndexCoord client"] [2024/01/05 14:11:15.417 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"] [2024/01/05 14:11:15.417 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2024/01/05 14:11:15.417 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2024/01/05 14:11:15.417 +00:00] [INFO] [datacoord/service.go:155] ["network port"] [port=40005] [2024/01/05 14:11:15.422 +00:00] [INFO] [paramtable/quota_param.go:769] ["init disk quota"] [diskQuota(MB)=+inf] [2024/01/05 14:11:15.422 +00:00] [INFO] [paramtable/quota_param.go:784] ["init disk quota per DB"] [diskQuotaPerCollection(MB)=1.7976931348623157e+308] [2024/01/05 14:11:15.422 +00:00] [INFO] [paramtable/component_param.go:1568] ["init segment max idle time"] [value=10m0s] [2024/01/05 14:11:15.423 +00:00] [INFO] [paramtable/component_param.go:1573] ["init segment min size from idle to sealed"] [value=16] [2024/01/05 14:11:15.423 +00:00] [INFO] [paramtable/component_param.go:1583] ["init segment max binlog file to sealed"] [value=32] [2024/01/05 14:11:15.423 +00:00] [INFO] [paramtable/component_param.go:1578] ["init segment expansion rate"] [value=1.25] [2024/01/05 14:11:15.424 +00:00] [INFO] [paramtable/base_table.go:143] ["cannot find etcd.endpoints"] [2024/01/05 14:11:15.424 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] 2024-01-05 14:11:15,424 INFO [default] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 2024-01-05 14:11:15,425 INFO [default] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 2024-01-05 14:11:15,425 INFO [default] [KNOWHERE][SetStatisticsLevel][milvus] Set knowhere::STATISTICS_LEVEL to 0 ASSERTION FAILURE FROM EASYLOGGING++ (LINE: 307) [(assertionPassed = base::utils::File::pathExists(configurationFile.c_str(), true)) == true] WITH MESSAGE "Configuration file [/root/.milvus.io/milvus-server/2.2.13/configs/easylogging.yaml] does not exist!" 2024-01-05 14:11:15,425 DEBUG [default] [SERVER][operator()][milvus] Config easylogging with yaml file: /root/.milvus.io/milvus-server/2.2.13/configs/easylogging.yaml 2024-01-05 14:11:15,425 DEBUG [default] [SEGCORE][SegcoreSetSimdType][milvus] set config simd_type: auto 2024-01-05 14:11:15,425 INFO [default] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO 2024-01-05 14:11:15,425 INFO [default] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512 2024-01-05 14:11:15,425 DEBUG [default] [SEGCORE][SetIndexSliceSize][milvus] set config index slice size(byte): 16777216 2024-01-05 14:11:15,425 DEBUG [default] [SEGCORE][SetThreadCoreCoefficient][milvus] set thread pool core coefficient: 10 2024-01-05 14:11:15,437 INFO [default] [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO 2024-01-05 14:11:15,437 INFO [default] [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512 Traceback (most recent call last): File "/builds/rnai/voicecast/tools/nir2/test_indexer.py", line 12, in <module> debug_server.start() File "/usr/local/lib/python3.10/site-packages/milvus/__init__.py", line 427, in start self.wait_started() File "/usr/local/lib/python3.10/site-packages/milvus/__init__.py", line 391, in wait_started raise TimeoutError(f'Milvus not startd in {timeout/1000} seconds') TimeoutError: Milvus not startd in 30.0 seconds

panic: All attempts results: attempt #1:err: find no available rootcoord, check rootcoord state , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #2:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #3:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #4:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #5:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #6:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #7:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #8:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #9:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #10:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run attempt #11:err: failed to connect 172.16.2.48:40000, reason: context deadline exceeded , /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/trace/stack_trace.go:51 github.com/milvus-io/milvus/internal/util/trace.StackTrace /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/grpcclient/client.go:352 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/rootcoord/client/client.go:130 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:75 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates.func1 /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/retry/retry.go:42 github.com/milvus-io/milvus/internal/util/retry.Do /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:99 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentStates /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/util/funcutil/func.go:114 github.com/milvus-io/milvus/internal/util/funcutil.WaitForComponentHealthy /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:149 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run goroutine [376](https://gitlab.devops.telekom.de/rnai/voicecast/tools/nir2/-/jobs/146166332#L376) [running]: github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).init(0xc00085d300) /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:152 +0x11e5 github.com/milvus-io/milvus/internal/distributed/indexcoord.(*Server).Run(0xc000e90708?) /__w/milvus-lite/milvus-lite/milvus_binary/milvus/internal/distributed/indexcoord/service.go:82 +0x25 github.com/milvus-io/milvus/cmd/components.(*IndexCoord).Run(0x[601](https://gitlab.devops.telekom.de/rnai/voicecast/tools/nir2/-/jobs/146166332#L601)a3a0?) /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/components/index_coord.go:51 +0x2e github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/roles/roles.go:120 +0x182 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] /__w/milvus-lite/milvus-lite/milvus_binary/milvus/cmd/roles/roles.go:104 +0x18a

matrixji commented 9 months ago

Not very sure if this is a problem with AVX512 instructions, could you have a try with milvus==2.2.16?