search

bitpoke / mysql-operator

Asynchronous MySQL Replication on Kubernetes using Percona Server and Openark's Orchestrator.
https://www.bitpoke.io/docs/mysql-operator/getting-started/
Apache License 2.0
1.02k stars 274 forks source link

ERROR lookup mysql-operator-0.mysql-operator-orc on 10.96.0.10:53: read udp 10.88.0.32:51432->10.96.0.10:53: #795

Open CloudCourierStation opened 2 years ago

CloudCourierStation commented 2 years ago

Hello, when I create mysql-operator, one of the containers of operator cannot be started. May I ask what is the cause? The following is the error log root@master:~/mysql-operator# kubectl logs -f mysql-operator-0 -c orchestrator -n mysql-operator [martini] Started GET /api/raft-health for 10.88.0.1:58096 [martini] Completed 500 Internal Server Error in 321.501µs [martini] Started GET /api/raft-health for 10.88.0.1:58102 [martini] Completed 500 Internal Server Error in 512.13µs [martini] Started GET /api/raft-health for 10.88.0.1:58130 [martini] Completed 500 Internal Server Error in 364.787µs [martini] Started GET /api/raft-health for 10.88.0.1:58154 [martini] Completed 500 Internal Server Error in 455.678µs [martini] Started GET /api/raft-health for 10.88.0.1:58178 [martini] Completed 500 Internal Server Error in 487.137µs [martini] Started GET /api/raft-health for 10.88.0.1:58208 [martini] Completed 500 Internal Server Error in 314.755µs 2022-03-01 10:04:34 ERROR lookup mysql-operator-0.mysql-operator-orc on 10.96.0.10:53: read udp 10.88.0.32:33959->10.96.0.10:53: i/o timeout [martini] Started GET /api/raft-health for 10.88.0.1:58232 [martini] Completed 500 Internal Server Error in 314.683µs [martini] Started GET /api/raft-health for 10.88.0.1:58256 [martini] Completed 500 Internal Server Error in 303.236µs [martini] Started GET /api/raft-health for 10.88.0.1:58272 [martini] Completed 500 Internal Server Error in 382.705µs [martini] Started GET /api/raft-health for 10.88.0.1:58286 [martini] Completed 500 Internal Server Error in 338.301µs [martini] Started GET /api/raft-health for 10.88.0.1:58310 [martini] Completed 500 Internal Server Error in 348.86µs [martini] Started GET /api/raft-health for 10.88.0.1:58334 [martini] Completed 500 Internal Server Error in 543.519µs 2022-03-01 10:05:24 ERROR failed to open raft store: lookup mysql-operator-0.mysql-operator-orc on 10.96.0.10:53: read udp 10.88.0.32:51391->10.96.0.10:53: i/o timeout 2022-03-01 10:05:24 FATAL 2022-03-01 10:05:24 ERROR failed to open raft store: lookup mysql-operator-0.mysql-operator-orc on 10.96.0.10:53: read udp 10.88.0.32:51391->10.96.0.10:53: i/o timeout 2022/03/01 10:05:24 Command exited with error: exit status 1

jiuker commented 2 years ago

Turn down firewall？

CloudCourierStation commented 2 years ago

My firewall is off And I see that one of the mysql-operator containers is down

root@master:~# kubectl get po -n mysql-operator -owide
NAME                 READY   STATUS    RESTARTS        AGE    IP           NODE     NOMINATED NODE   READINESS GATES
my-cluster-mysql-0   3/4     Running   4 (8m26s ago)   2d5h   10.88.0.38   node-1   <none>           <none>
mysql-operator-0     1/2     Running   13 (93s ago)    2d5h   10.88.0.37   node-1   <none>           <none>
root@master:~# kubectl logs -f mysql-operator-0 -c orchestrator -n mysql-operator
[martini] Started GET /api/raft-health for 10.88.0.1:34950
[martini] Completed 500 Internal Server Error in 318.824µs
[martini] Started GET /api/raft-health for 10.88.0.1:34954
[martini] Completed 500 Internal Server Error in 359.305µs
[martini] Started GET /api/raft-health for 10.88.0.1:34962
[martini] Completed 500 Internal Server Error in 361.597µs
[martini] Started GET /api/raft-health for 10.88.0.1:34988
[martini] Completed 500 Internal Server Error in 420.606µs
[martini] Started GET /api/raft-health for 10.88.0.1:35016
[martini] Completed 500 Internal Server Error in 442.355µs
[martini] Started GET /api/raft-health for 10.88.0.1:35042
[martini] Completed 500 Internal Server Error in 302.116µs
[martini] Started GET /api/raft-health for 10.88.0.1:35070
[martini] Completed 500 Internal Server Error in 271.786µs
2022-03-03 15:34:14 ERROR lookup mysql-operator-0.mysql-operator-orc on 10.96.0.10:53: read udp 10.88.0.37:39614->10.96.0.10:53: i/o timeout
[martini] Started GET /api/raft-health for 10.88.0.1:35098
[martini] Completed 500 Internal Server Error in 274.799µs

How do I find out why the container is down

kangvai commented 1 year ago

@CloudCourierStation Hi，I seem have the same problem，have you solved now? Looking forward to your reply！

root@DEV-Master-Test-1:~# kubectl logs pod/mysql-operator-0 -c orchestrator
[martini] Started GET /api/raft-health for 10.2.0.8:60410
[martini] Completed 500 Internal Server Error in 1.938425ms
[martini] Started GET /api/raft-health for 10.2.0.8:60426
[martini] Completed 500 Internal Server Error in 427.906µs
[martini] Started GET /api/raft-health for 10.2.0.8:60434
[martini] Completed 500 Internal Server Error in 488.707µs
[martini] Started GET /api/raft-health for 10.2.0.8:59076
[martini] Completed 500 Internal Server Error in 515.206µs
[martini] Started GET /api/raft-health for 10.2.0.8:56612
[martini] Completed 500 Internal Server Error in 537.007µs
[martini] Started GET /api/raft-health for 10.2.0.8:40608
[martini] Completed 500 Internal Server Error in 488.807µs
[martini] Started GET /api/raft-health for 10.2.0.8:51890
[martini] Completed 500 Internal Server Error in 433.406µs
2023-06-19 05:45:47 ERROR lookup mysql-operator-0-orc-svc on 10.96.0.10:53: read udp 192.168.85.150:37290->10.96.0.10:53: i/o timeout
[martini] Started GET /api/raft-health for 10.2.0.8:42032
[martini] Completed 500 Internal Server Error in 484.907µs
[martini] Started GET /api/raft-health for 10.2.0.8:59384
[martini] Completed 500 Internal Server Error in 475.606µs
[martini] Started GET /api/raft-health for 10.2.0.8:51156
[martini] Completed 500 Internal Server Error in 379.405µs
[martini] Started GET /api/raft-health for 10.2.0.8:33864
[martini] Completed 500 Internal Server Error in 408.106µs
[martini] Started GET /api/raft-health for 10.2.0.8:33872
[martini] Completed 500 Internal Server Error in 385.305µs
[martini] Started GET /api/raft-health for 10.2.0.8:56118
[martini] Completed 500 Internal Server Error in 381.006µs
2023-06-19 05:46:37 ERROR failed to open raft store: lookup mysql-operator-0-orc-svc on 10.96.0.10:53: read udp 192.168.85.150:38465->10.96.0.10:53: i/o timeout
2023-06-19 05:46:37 FATAL 2023-06-19 05:46:37 ERROR failed to open raft store: lookup mysql-operator-0-orc-svc on 10.96.0.10:53: read udp 192.168.85.150:38465->10.96.0.10:53: i/o timeout
2023/06/19 05:46:37 Command exited with error: exit status 1