yanliang567
commented
3 months ago I am not sure what release are you running, please retry on milvus 2.3.15 or 2.4.1 /assign @Storm0921 /unassign
Closed Storm0921 closed 2 weeks ago
yanliang567
commented
3 months ago I am not sure what release are you running, please retry on milvus 2.3.15 or 2.4.1 /assign @Storm0921 /unassign
xiaofan-luan
commented
3 months ago seems that your etcd is running. please check the etcd state and the network connection between milvus and etcd
njjry
commented
2 months ago Hello,
I am facing the same problem with a self-built milvus (I build milvus following the instruction here: https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md#docker--docker-compose), the milvus version is 2.4.2. I used command line sudo ./scripts/start_standalone.sh. I am pasting my error message here:
"/tmp/standalone.log" [readonly] 1189L, 85523B 1,2 Top
[2024/05/22 20:35:46.536 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
2024/05/22 20:35:46 run subprocess with cmd: [../bin/milvus run standalone]
[2024/05/22 20:35:46.842 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
2024/05/22 20:35:46 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined
__ _________ _ ____ ______
/ |/ / _/ /| | / / / / / __/
/ /|_/ // // /_| |/ / /_/ /\ \
/_/ /_/___/____/___/\____/___/
Welcome to use Milvus!
Version: v2.4.2-70-ge1bafd710
Built: Wed May 22 18:45:19 UTC 2024
GitCommit: e1bafd710
GoVersion: go version go1.21.0 linux/amd64
TotalMem: 135048794112
UsedMem: 35815424
open pid file: /run/milvus/standalone.pid
lock pid file: /run/milvus/standalone.pid
[2024/05/22 20:35:46.869 +00:00] [INFO] [roles/roles.go:306] ["starting running Milvus components"]
[2024/05/22 20:35:46.883 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/05/22 20:35:46.885 +00:00] [DEBUG] [config/etcd_source.go:50] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"localhost:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/05/22 20:35:46.885 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.886 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"]
[2024/05/22 20:35:51.895 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/05/22 20:35:51.895 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer]
[2024/05/22 20:35:51.895 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/05/22 20:35:51.895 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2024/05/22 20:35:51.895 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091]
[2024/05/22 20:35:51.895 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."]
[2024/05/22 20:35:51.895 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.886 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"]
[2024/05/22 20:35:51.895 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/05/22 20:35:51.895 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer]
[2024/05/22 20:35:51.895 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/05/22 20:35:51.895 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2024/05/22 20:35:51.895 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091]
[2024/05/22 20:35:51.895 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."]
[2024/05/22 20:35:51.895 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.898 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.899 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."]
[2024/05/22 20:35:51.899 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.899 +00:00] [DEBUG] [querynode/service.go:104] [QueryNode] [port=21123]
[2024/05/22 20:35:51.900 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.900 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:51.901 +00:00] [DEBUG] [indexnode/indexnode.go:115] ["New IndexNode ..."]
[2024/05/22 20:35:51.901 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=10.168.113.68:21121] ["network port: "=21121]
[2024/05/22 20:35:51.901 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"]
[2024/05/22 20:35:51.902 +00:00] [INFO] [runtime/asm_amd64.s:1650] ["Start check query node health loop"]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:226] ["RateLimiter register for rateType"] [rateType=DMLInsert] [rateLimit=+inf] [burst=1.7976931348623157e+308]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:226] ["RateLimiter register for rateType"] [rateType=DMLDelete] [rateLimit=+inf] [burst=1.7976931348623157e+308]
[2024/05/22 20:35:51.902 +00:00] [INFO] [hookutil/hook.go:46] ["empty so path, skip to load plugin"]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:122] ["create a new Proxy instance"] [state=2]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:423] ["init Proxy server"]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:454] ["Proxy init service's parameter table done"]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:456] ["Proxy init http server's parameter table done"]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:463] ["init Proxy's parameter table done"] [internalAddress=10.168.113.68:19529] [externalAddress=10.168.113.68:19530]
[2024/05/22 20:35:51.902 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"]
[2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:470] ["init Proxy's tracer done"] ["service name"="Proxy ip: 10.168.113.68, port: 19530"]
[2024/05/22 20:35:51.902 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:52.002 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3]
[2024/05/22 20:35:56.896 +00:00] [DEBUG] [rootcoord/service.go:198] ["RootCoord connect to etcd failed"] [error="context deadline exceeded"]
[2024/05/22 20:35:56.896 +00:00] [ERROR] [components/root_coord.go:55] ["RootCoord starts error"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/home/lisa/milvus/milvus_code/milvus/cmd/components/root_coord.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:113"]
panic: context deadline exceeded
goroutine 230 [running]:
panic({0x53975c0?, 0x8584b40?})
/usr/local/go/src/runtime/panic.go:1017 +0x3ac fp=0xc0017ddf70 sp=0xc0017ddec0 pc=0x1dec88c
github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1()
/home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:114 +0x108 fp=0xc0017ddfe0 sp=0xc0017ddf70 pc=0x4d399a8
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0017ddfe8 sp=0xc0017ddfe0 pc=0x1e260c1
created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] in goroutine 1
/home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:105 +0x138
goroutine 1 [semacquire]:
runtime.gopark(0x0?, 0xc0017a4c70?, 0x60?, 0x49?, 0xc0010922c0?)
/usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc001a1f630 sp=0xc001a1f610 pc=0x1df08ae
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:404
runtime.semacquire1(0xc0019a4598, 0x60?, 0x1, 0x0, 0x1?)May I know how to solve the problem? @xiaofan-luan mentioned about the etcd, may I know how to check if my etcd is running?
Thank you for the help!
xiaofan-luan
commented
2 months ago Hello,
I am facing the same problem with a self-built milvus (I build milvus following the instruction here: https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md#docker--docker-compose), the milvus version is 2.4.2. I used command line
sudo ./scripts/start_standalone.sh. I am pasting my error message here:"/tmp/standalone.log" [readonly] 1189L, 85523B 1,2 Top [2024/05/22 20:35:46.536 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"] 2024/05/22 20:35:46 run subprocess with cmd: [../bin/milvus run standalone] [2024/05/22 20:35:46.842 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"] 2024/05/22 20:35:46 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined __ _________ _ ____ ______ / |/ / _/ /| | / / / / / __/ / /|_/ // // /_| |/ / /_/ /\ \ /_/ /_/___/____/___/\____/___/ Welcome to use Milvus! Version: v2.4.2-70-ge1bafd710 Built: Wed May 22 18:45:19 UTC 2024 GitCommit: e1bafd710 GoVersion: go version go1.21.0 linux/amd64 TotalMem: 135048794112 UsedMem: 35815424 open pid file: /run/milvus/standalone.pid lock pid file: /run/milvus/standalone.pid [2024/05/22 20:35:46.869 +00:00] [INFO] [roles/roles.go:306] ["starting running Milvus components"] [2024/05/22 20:35:46.883 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/05/22 20:35:46.885 +00:00] [DEBUG] [config/etcd_source.go:50] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"localhost:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"] [2024/05/22 20:35:46.885 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.886 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"] [2024/05/22 20:35:51.895 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}] [2024/05/22 20:35:51.895 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/05/22 20:35:51.895 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal] [2024/05/22 20:35:51.895 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."] [2024/05/22 20:35:51.895 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.886 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"] [2024/05/22 20:35:51.895 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}] [2024/05/22 20:35:51.895 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/05/22 20:35:51.895 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal] [2024/05/22 20:35:51.895 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."] [2024/05/22 20:35:51.895 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.898 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.899 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."] [2024/05/22 20:35:51.899 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.899 +00:00] [DEBUG] [querynode/service.go:104] [QueryNode] [port=21123] [2024/05/22 20:35:51.900 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.900 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.901 +00:00] [DEBUG] [indexnode/indexnode.go:115] ["New IndexNode ..."] [2024/05/22 20:35:51.901 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=10.168.113.68:21121] ["network port: "=21121] [2024/05/22 20:35:51.901 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"] [2024/05/22 20:35:51.902 +00:00] [INFO] [runtime/asm_amd64.s:1650] ["Start check query node health loop"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:226] ["RateLimiter register for rateType"] [rateType=DMLInsert] [rateLimit=+inf] [burst=1.7976931348623157e+308] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:226] ["RateLimiter register for rateType"] [rateType=DMLDelete] [rateLimit=+inf] [burst=1.7976931348623157e+308] [2024/05/22 20:35:51.902 +00:00] [INFO] [hookutil/hook.go:46] ["empty so path, skip to load plugin"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:122] ["create a new Proxy instance"] [state=2] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:423] ["init Proxy server"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:454] ["Proxy init service's parameter table done"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:456] ["Proxy init http server's parameter table done"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:463] ["init Proxy's parameter table done"] [internalAddress=10.168.113.68:19529] [externalAddress=10.168.113.68:19530] [2024/05/22 20:35:51.902 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:470] ["init Proxy's tracer done"] ["service name"="Proxy ip: 10.168.113.68, port: 19530"] [2024/05/22 20:35:51.902 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:52.002 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:56.896 +00:00] [DEBUG] [rootcoord/service.go:198] ["RootCoord connect to etcd failed"] [error="context deadline exceeded"] [2024/05/22 20:35:56.896 +00:00] [ERROR] [components/root_coord.go:55] ["RootCoord starts error"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/home/lisa/milvus/milvus_code/milvus/cmd/components/root_coord.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:113"] panic: context deadline exceeded goroutine 230 [running]: panic({0x53975c0?, 0x8584b40?}) /usr/local/go/src/runtime/panic.go:1017 +0x3ac fp=0xc0017ddf70 sp=0xc0017ddec0 pc=0x1dec88c github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() /home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:114 +0x108 fp=0xc0017ddfe0 sp=0xc0017ddf70 pc=0x4d399a8 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0017ddfe8 sp=0xc0017ddfe0 pc=0x1e260c1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[...] in goroutine 1 /home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:105 +0x138 goroutine 1 [semacquire]: runtime.gopark(0x0?, 0xc0017a4c70?, 0x60?, 0x49?, 0xc0010922c0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc001a1f630 sp=0xc001a1f610 pc=0x1df08ae runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.semacquire1(0xc0019a4598, 0x60?, 0x1, 0x0, 0x1?)May I know how to solve the problem? @xiaofan-luan mentioned about the etcd, may I know how to check if my etcd is running?
Thank you for the help!
etcd seems not running as expect. did you check the etcd docker ?
njjry
commented
2 months ago May I know how to check the etcd docker? I just ran the milvus locally, didn’t start docker containers.
On Wed, May 22, 2024 at 22:48 Xiaofan @.***> wrote:
Hello,
I am facing the same problem with a self-built milvus (I build milvus following the instruction here: https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md#docker--docker-compose https://urldefense.com/v3/__https://github.com/milvus-io/milvus/blob/master/DEVELOPMENT.md*docker--docker-compose__;Iw!!IKRxdwAv5BmarQ!dVJQi6nxsnNwMFk69x2gcI6Wx1QDksqBR-gwW0uOwoLixu-EwRw5D9yk5OGRnbFs_BdZGkE0T1ws0wzG_Zl61qYfLFQ$), the milvus version is 2.4.2. I used command line sudo ./scripts/start_standalone.sh. I am pasting my error message here:
"/tmp/standalone.log" [readonly] 1189L, 85523B 1,2 Top [2024/05/22 20:35:46.536 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"] 2024/05/22 20:35:46 run subprocess with cmd: [../bin/milvus run standalone] [2024/05/22 20:35:46.842 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"] 2024/05/22 20:35:46 maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined
__ _________ _ ____ ______/ |/ / _/ /| | / / / / / / / /|/ // // /| |/ / // /\ \ // ///__/_/__/_/
Welcome to use Milvus! Version: v2.4.2-70-ge1bafd710 Built: Wed May 22 18:45:19 UTC 2024 GitCommit: e1bafd710 GoVersion: go version go1.21.0 linux/amd64
TotalMem: 135048794112 UsedMem: 35815424
open pid file: /run/milvus/standalone.pid lock pid file: /run/milvus/standalone.pid [2024/05/22 20:35:46.869 +00:00] [INFO] [roles/roles.go:306] ["starting running Milvus components"] [2024/05/22 20:35:46.883 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/05/22 20:35:46.885 +00:00] [DEBUG] [config/etcd_source.go:50] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"localhost:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"] [2024/05/22 20:35:46.885 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.886 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"] [2024/05/22 20:35:51.895 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}] [2024/05/22 20:35:51.895 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/05/22 20:35:51.895 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal] [2024/05/22 20:35:51.895 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."] [2024/05/22 20:35:51.895 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.886 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"] [2024/05/22 20:35:51.895 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}] [2024/05/22 20:35:51.895 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource] [2024/05/22 20:35:51.895 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal] [2024/05/22 20:35:51.895 +00:00] [INFO] [http/server.go:112] ["management listen"] [addr=:9091] [2024/05/22 20:35:51.895 +00:00] [DEBUG] [rootcoord/service.go:184] ["init params done.."] [2024/05/22 20:35:51.895 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.898 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.899 +00:00] [INFO] [components/index_coord.go:38] ["IndexCoord running ..."] [2024/05/22 20:35:51.899 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.899 +00:00] [DEBUG] [querynode/service.go:104] [QueryNode] [port=21123] [2024/05/22 20:35:51.900 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.900 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:51.901 +00:00] [DEBUG] [indexnode/indexnode.go:115] ["New IndexNode ..."] [2024/05/22 20:35:51.901 +00:00] [DEBUG] [indexnode/service.go:87] [IndexNode] ["network address"=10.168.113.68:21121] ["network port: "=21121] [2024/05/22 20:35:51.901 +00:00] [INFO] [proxy/lb_policy.go:78] ["use look_aside policy on replica selection"] [2024/05/22 20:35:51.902 +00:00] [INFO] [runtime/asm_amd64.s:1650] ["Start check query node health loop"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:226] ["RateLimiter register for rateType"] [rateType=DMLInsert] [rateLimit=+inf] [burst=1.7976931348623157e+308] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/simple_rate_limiter.go:226] ["RateLimiter register for rateType"] [rateType=DMLDelete] [rateLimit=+inf] [burst=1.7976931348623157e+308] [2024/05/22 20:35:51.902 +00:00] [INFO] [hookutil/hook.go:46] ["empty so path, skip to load plugin"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:122] ["create a new Proxy instance"] [state=2] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:423] ["init Proxy server"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:454] ["Proxy init service's parameter table done"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:456] ["Proxy init http server's parameter table done"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:463] ["init Proxy's parameter table done"] [internalAddress=10.168.113.68:19529] [externalAddress=10.168.113.68:19530] [2024/05/22 20:35:51.902 +00:00] [INFO] [accesslog/global.go:146] ["Init access logger success"] [2024/05/22 20:35:51.902 +00:00] [DEBUG] [proxy/service.go:470] ["init Proxy's tracer done"] ["service name"="Proxy ip: 10.168.113.68, port: 19530"] [2024/05/22 20:35:51.902 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:52.002 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[localhost:2379]"] [minVersion=1.3] [2024/05/22 20:35:56.896 +00:00] [DEBUG] [rootcoord/service.go:198] ["RootCoord connect to etcd failed"] [error="context deadline exceeded"] [2024/05/22 20:35:56.896 +00:00] [ERROR] [components/root_coord.go:55] ["RootCoord starts error"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/cmd/components.(RootCoord).Run\n\t/home/lisa/milvus/milvus_code/milvus/cmd/components/root_coord.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:113 <http://github.com/milvus-io/milvus/cmd/components.(RootCoord).Run%5Cn%5Ct/home/lisa/milvus/milvus_code/milvus/cmd/components/root_coord.go:55%5Cngithub.com/milvus-io/milvus/cmd/roles.runComponent%5B...%5D.func1%5Cn%5Ct/home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:113>"] panic: context deadline exceeded
goroutine 230 [running]: panic({0x53975c0?, 0x8584b40?}) /usr/local/go/src/runtime/panic.go:1017 +0x3ac fp=0xc0017ddf70 sp=0xc0017ddec0 pc=0x1dec88cgithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1() http://github.com/milvus-io/milvus/cmd/roles.runComponent%5B...%5D.func1() /home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:114 +0x108 fp=0xc0017ddfe0 sp=0xc0017ddf70 pc=0x4d399a8 runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1650 +0x1 fp=0xc0017ddfe8 sp=0xc0017ddfe0 pc=0x1e260c1 created by github.com/milvus-io/milvus/cmd/roles.runComponent[.. http://github.com/milvus-io/milvus/cmd/roles.runComponent%5B...] in goroutine 1 /home/lisa/milvus/milvus_code/milvus/cmd/roles/roles.go:105 +0x138
goroutine 1 [semacquire]: runtime.gopark(0x0?, 0xc0017a4c70?, 0x60?, 0x49?, 0xc0010922c0?) /usr/local/go/src/runtime/proc.go:398 +0xce fp=0xc001a1f630 sp=0xc001a1f610 pc=0x1df08ae runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:404 runtime.semacquire1(0xc0019a4598, 0x60?, 0x1, 0x0, 0x1?)
May I know how to solve the problem? @xiaofan-luan https://urldefense.com/v3/__https://github.com/xiaofan-luan__;!!IKRxdwAv5BmarQ!dVJQi6nxsnNwMFk69x2gcI6Wx1QDksqBR-gwW0uOwoLixu-EwRw5D9yk5OGRnbFs_BdZGkE0T1ws0wzG_Zl6XX49IkQ$ mentioned about the etcd, may I know how to check if my etcd is running?
Thank you for the help!
etcd seems not running as expect. did you check the etcd docker ?
— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/milvus-io/milvus/issues/33085*issuecomment-2126284047__;Iw!!IKRxdwAv5BmarQ!dVJQi6nxsnNwMFk69x2gcI6Wx1QDksqBR-gwW0uOwoLixu-EwRw5D9yk5OGRnbFs_BdZGkE0T1ws0wzG_Zl6eUggP4I$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ADMNK6TZH4YJ2E4I7DGCDF3ZDV7JPAVCNFSM6AAAAABHZKZ4UGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRWGI4DIMBUG4__;!!IKRxdwAv5BmarQ!dVJQi6nxsnNwMFk69x2gcI6Wx1QDksqBR-gwW0uOwoLixu-EwRw5D9yk5OGRnbFs_BdZGkE0T1ws0wzG_Zl6WYqKq9g$ . You are receiving this because you commented.Message ID: @.***>
xiaofan-luan
commented
2 months ago njjry
commented
2 months ago Hello,
I don't have etcd docker container running. I only built Milvus and then run ./bin/milvus, does this step automatically start etcd container? Or how can I start etcd container manually? Thanks
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ceba7a018dfb hello-world "/hello" 6 weeks ago Exited (0) 6 weeks ago modest_antonelli
09f3ff05dbed hello-world "/hello" 6 weeks ago Exited (0) 6 weeks ago flamboyant_knuthHello,
are there any updates on the issue?
yanliang567
commented
2 months ago /assign @czs007 please help to take a look
uniquejava
commented
1 month ago I met the same issue on OpenShift 4
$ oc project xxxx
$ helm repo add milvus https://zilliztech.github.io/milvus-helm/
"milvus" has been added to your repositories
$ helm repo update
# 最小安装, 见: https://milvus.io/docs/install_standalone-helm.md
# helm install my-milvus milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false --set persistence.storageClass=milvus-storage
$ helm install my-milvus milvus/milvus --set cluster.enabled=false --set etcd.replicaCount=1 --set minio.mode=standalone --set pulsar.enabled=false
NAME: my-milvus
LAST DEPLOYED: Fri Jun 14 16:29:00 2024
NAMESPACE: xxxxx
STATUS: deployed
REVISION: 1
TEST SUITE: None
# 关闭 milvus standalone pod
$ oc scale deployment my-milvus-standalone --replicas=0
# 给 etcd 打补丁
$ oc scale statefulset my-milvus-etcd --replicas=0
$ oc patch sts my-milvus-etcd --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/securityContext"}]'
$ oc patch sts my-milvus-etcd --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/securityContext"}]'
$ oc scale statefulset my-milvus-etcd --replicas=1
# wait for etcd starting
$ oc get po -w
NAME READY STATUS RESTARTS AGE
my-milvus-etcd-0 1/1 Running 0 30m
my-milvus-minio-b94d9974-9z8k6 1/1 Running 0 4h34m
my-milvus-standalone-7d97c86f9b-srpb8 1/1 Running 0 27m
# 重启 milvus standalone pod
$ oc scale deployment my-milvus-standalone --replicas=1
# wait for etcd starting
$ oc get po -w
$ oc logs my-milvus-standalone-7d97c86f9b-8fn6v | grep -C 3 failed
Defaulted container "standalone" out of: standalone, config (init)
[2024/06/18 16:11:59.626 +00:00] [INFO] [distance/calc_distance_amd64.go:14] ["Hook avx for go simd distance computation"]
2024/06/18 16:11:59 maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined
Set runtime dir at /run/milvus failed, set it to /tmp/milvus directory
__ _________ _ ____ ______
/ |/ / _/ /| | / / / / / __/
--
[2024/06/18 16:11:59.643 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/06/18 16:11:59.644 +00:00] [DEBUG] [config/etcd_source.go:50] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"my-milvus-etcd:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/06/18 16:11:59.644 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3]
[2024/06/18 16:12:04.644 +00:00] [INFO] [paramtable/base_table.go:209] ["init with etcd failed"] [error="context deadline exceeded"]
[2024/06/18 16:12:04.647 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/06/18 16:12:04.647 +00:00] [INFO] [roles/roles.go:255] [setupPrometheusHTTPServer]
[2024/06/18 16:12:04.647 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
--
[2024/06/18 16:12:04.654 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3]
[2024/06/18 16:12:04.653 +00:00] [INFO] [runtime/asm_amd64.s:1598] ["Start check query node health loop"]
[2024/06/18 16:12:04.754 +00:00] [INFO] [etcd/etcd_util.go:49] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[my-milvus-etcd:2379]"] [minVersion=1.3]
[2024/06/18 16:12:09.649 +00:00] [DEBUG] [rootcoord/service.go:198] ["RootCoord connect to etcd failed"] [error="context deadline exceeded"]
[2024/06/18 16:12:09.649 +00:00] [ERROR] [components/root_coord.go:55] ["RootCoord starts error"] [error="context deadline exceeded"] [stack="github.com/milvus-io/milvus/cmd/components.(*RootCoord).Run\n\t/go/src/github.com/milvus-io/milvus/cmd/components/root_coord.go:55\ngithub.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n\t/go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:113"]
panic: context deadline exceeded
stale[bot]
commented
4 weeks ago This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.
ENg-122
commented
1 week ago root@fddfcd2e4283:/# etcdctl --endpoints=10.166.192.142:2379 get foo {"level":"warn","ts":"2024-08-09T09:52:27.952Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002d4000/10.166.192.142:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 10.166.192.142:2379: connect: connection refused\""} Error: context deadline exceeded这个报错咋解决啊？
xiaofan-luan
commented
1 week ago seems that etcd crash? did you check your etcd state?
ENg-122
commented
1 week ago 7f8b4c5248ca quay.io/coreos/etcd:v3.5.5 "etcd -advertise-c..." 16 hours ago Up 7 minutes (healthy) 2379-2380/tcp milvusEtcd docker状态正常 日志也没有报错
ENg-122
commented
1 week ago root@89b74812630b:/# etcdctl endpoint health 127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.533804ms root@89b74812630b:/# etcdctl endpoint status 127.0.0.1:2379, 8e9e05c52164694d, 3.5.5, 25 kB, true, false, 22, 314, 314, root@89b74812630b:/# etcdctl member list 8e9e05c52164694d, started, default, http://localhost:2380, http://127.0.0.1:2379, false
xiaofan-luan
commented
6 days ago 10.166.192.142 seems to be your etcd ip? maybe you can use etcdctl to test
xiaofan-luan
commented
6 days ago at least at the time you offer this log etcd is timeout. it could be a network issue or etcd crash, we don't have enough information on that
ENg-122
commented
5 days ago 解决了 我这边通过域名连接不了 换了ip连接可以了
Is there an existing issue for this?
Environment
Current Behavior
我运行下面的命令得到的日志如下，怎么解决这个问题？
nohup ./bin/milvus run standalone > running.log 2>&1 &有时候还会出现Datacord的类似问题Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
No response