search

saltstack-formulas / opensds-formula

Integration with Open Software Defined Storage - Capri release - SuSE/CentOS/Ubuntu
https://opensds.io
Other
4 stars 2 forks source link

osdslet container error on Ubuntu/Centos #17

Closed noelmcloughlin closed 5 years ago

noelmcloughlin commented 5 years ago 
root@ubuntu1804:~# docker ps -a
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                      PORTS                                             NAMES
21f3bdbfb215        opensdsio/dashboard:latest             "/bin/sh -c /opt/das…"   9 hours ago         Up 2 hours                                                                    dashboard
b7e411103870        opensdsio/opensds-dock:latest          "/usr/bin/osdsdock"      9 hours ago         Up 11 minutes                                                                 osdsdock
1083acf9f4d1        gcr.io/etcd-development/etcd:v3.2.24   "etcd -name etcd0 -a…"   9 hours ago         Up 12 minutes               2379-2380/udp, 0.0.0.0:2379-2380->2379-2380/tcp   osdsdb-etcd
3e7d23ad9355        opensdsio/opensds-controller:latest    "/usr/bin/osdslet"       10 hours ago        Exited (2) 10 minutes ago                                                     opensds-controller

root@ubuntu1804:~# docker logs 3e7d23ad9355
2018/12/11 01:43:55 [ERROR] Read configuration failed, use default value
E1211 01:43:58.623265       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 1 time(s)
E1211 01:44:01.624393       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2380: connect: connection refused, retry 2 time(s)
E1211 01:44:04.624930       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 3 time(s)
E1211 01:44:04.624954       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: dial tcp 127.0.0.1:2379: connect: connection refused

goroutine 1 [running]:
github.com/opensds/opensds/pkg/db/drivers/etcd.Init(0xc0001b0700, 0x2, 0x2, 0x2)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/client.go:70 +0x110
github.com/opensds/opensds/pkg/db/drivers/etcd.NewClient(0xc0001b0700, 0x2, 0x2, 0x1)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/etcd.go:64 +0x43
github.com/opensds/opensds/pkg/db.Init(0xc000123e70)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/db.go:45 +0x1a6
main.main()
    /root/gopath/src/github.com/opensds/opensds/cmd/osdslet/osdslet.go:50 +0x6d
2018/12/11 11:07:00 [ERROR] Read configuration failed, use default value
E1211 11:07:03.580288       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 1 time(s)
E1211 11:07:06.582396       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 2 time(s)
E1211 11:07:09.582704       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 3 time(s)
E1211 11:07:09.582767       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: dial tcp 127.0.0.1:2379: connect: connection refused

goroutine 1 [running]:
github.com/opensds/opensds/pkg/db/drivers/etcd.Init(0xc00014e020, 0x2, 0x2, 0x2)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/client.go:70 +0x110
github.com/opensds/opensds/pkg/db/drivers/etcd.NewClient(0xc00014e020, 0x2, 0x2, 0x1)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/etcd.go:64 +0x43
github.com/opensds/opensds/pkg/db.Init(0xc000121e70)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/db.go:45 +0x1a6
main.main()
    /root/gopath/src/github.com/opensds/opensds/cmd/osdslet/osdslet.go:50 +0x6d

root@ubuntu1804:~# curl 127.0.0.1:2379
curl: (56) Recv failure: Connection reset by peer

root@ubuntu1804:~# telnet 127.0.0.1 2379
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
Connection closed by foreign host.

root@ubuntu1804:~# netstat -a | grep 2379 | wc -l
15
noelmcloughlin commented 5 years ago

Related to etcd.docker.running state: saltstack-formulas/etcd-formula#29

leonwanghui commented 5 years ago

Yes, according to the error output, I didn't see any error from opensds daemon service

leonwanghui commented 5 years ago

BTW can you change the image tag to quay.io/coreos/etcd:latest?

noelmcloughlin commented 5 years ago

Thanks, done https://github.com/saltstack-formulas/etcd-formula/pull/30

noelmcloughlin commented 5 years ago

Hi @leonwanghui
That suggestion was helpful and resolved the saltstack-formulas/etcd-formula#29 issue.

There is just one problem left now. I'd appreciate any advice or if you could test.

This is CentOS.

[root@localhost salt]# netstat -tuplan | grep 2379 | head -2
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      2358/etcd           
tcp        0      0 127.0.0.1:2379          127.0.0.1:34226         TIME_WAIT   -  

[root@localhost salt]# ps -ef | egrep 'etcd|osds'
root      2272  2255 99 22:20 ?        00:10:23 /usr/bin/osdsdock
root      2358  2341 45 22:20 ?        00:03:12 etcd -name etcd0 -advertise-client-urls https://127.0.0.1:2379 -listen-client-urls https://127.0.0.1:2379 -initial-advertise-peer-urls https://127.0.0.1:2380 -listen-peer-urls https://127.0.0.1:2380 -initial-cluster-token etcd-cluster-1 -initial-cluster etcd0=https://127.0.0.1:2380 -initial-cluster-state new -data-dir /var/lib/etcd/etcd0 --auto-tls --peer-auto-tls

[root@localhost salt]# docker ps -a
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS                     PORTS               NAMES
2e178826bd50        opensdsio/dashboard:latest      "/bin/sh -c /opt/das…"   4 hours ago         Up 7 minutes                                   dashboard
020925671b22        opensdsio/opensds-dock:latest   "/usr/bin/osdsdock"      4 hours ago         Up 7 minutes                                   osdsdock
d56834f97c25        quay.io/coreos/etcd:latest      "etcd -name etcd0 -a…"   5 hours ago         Up 7 minutes                                   osdsdb-etcd
159d09f3aaa6        opensdsio/opensds-controller    "/usr/bin/osdslet"       5 hours ago         Exited (2) 7 minutes ago                       opensds-controller

[root@localhost salt]# docker logs 159d09f3aaa6
2018/12/13 17:33:14 [ERROR] Read configuration failed, use default value
E1213 17:33:17.603832       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2380: connect: connection refused, retry 1 time(s)
E1213 17:33:20.604952       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 2 time(s)
E1213 17:33:23.607281       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2380: connect: connection refused, retry 3 time(s)
E1213 17:33:23.607309       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: dial tcp 127.0.0.1:2380: connect: connection refused

main.main()
    /root/gopath/src/github.com/opensds/opensds/cmd/osdslet/osdslet.go:50 +0x6d
2018/12/13 22:20:38 [ERROR] Read configuration failed, use default value
E1213 22:20:41.007964       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 1 time(s)
E1213 22:20:44.009372       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2380: connect: connection refused, retry 2 time(s)
E1213 22:20:47.012673       1 utils.go:100] Get etcd client:dial tcp 127.0.0.1:2379: connect: connection refused, retry 3 time(s)
E1213 22:20:47.012693       1 utils.go:107] Get etcd client retry exceed the max retry times(3).
panic: dial tcp 127.0.0.1:2379: connect: connection refused

goroutine 1 [running]:
github.com/opensds/opensds/pkg/db/drivers/etcd.Init(0xc00014e020, 0x2, 0x2, 0x2)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/client.go:70 +0x110
github.com/opensds/opensds/pkg/db/drivers/etcd.NewClient(0xc00014e020, 0x2, 0x2, 0x1)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/drivers/etcd/etcd.go:64 +0x43
github.com/opensds/opensds/pkg/db.Init(0xc000113e70)
    /root/gopath/src/github.com/opensds/opensds/pkg/db/db.go:45 +0x1a6
main.main()
    /root/gopath/src/github.com/opensds/opensds/cmd/osdslet/osdslet.go:50 +0x6d
leonwanghui commented 5 years ago

Hi, good progress, if I get it right, you must have OpenSDS daemon service installed in container, then you need to guarantee the container network is working, perhaps set the mode to host: --net=host.

noelmcloughlin commented 5 years ago

Closing this as focussing on osdslet daemon instead.