search

openebs-archive / jiva-operator

Kubernetes Operator for managing Jiva Volumes via custom resource.
https://openebs.io
Apache License 2.0
46 stars 44 forks source link

Target cannot connect to new initiator when the k8s node which the old initiator on is down #125

Closed von1994 closed 3 years ago

von1994 commented 3 years ago

What steps did you take and what happened: app pod is on node36 pvc-ctrl pod is on node34 pvc-rep0 is on node34 pvc-rep1 is on node35 pvc-pre2 is on node33 When I shutdown node36, wait a few minutes until app pod is terminated by server and force delete it from server. There is a new app pod created by controller and runs on node34. New app pod is stuck into ContainerCreating state because the volume is not ready. Check pvc-ctrl pod(target pod), some error message like this "rejecting connection: 10.244.2.1 target already connected at 10.244.4.0". CNI plugin is flannel, the 10.244.4.0/32 is the NIC flannel.1 on node36, the 10.244.2.1/24 is the NIC cni on node34.

What did you expect to happen: Target can connect to new initiator when the old initiator and old node not respond.

The output of the following commands will help us better understand what's going on: https://gist.github.com/von1994/af005cc019ab178c86dfc71bbfe25583

Anything else you would like to add: When I force deleted the pvc-ctrl pod and wait for a new one is running, pvc-ctrl works and app pod is running! So I think there is something not updated in old pvc-ctrl pod.

Environment:

shubham14bajpai commented 3 years ago

Hi @von1994 is the node CR for node36 removed from the cluster? If it is still there in notReady state then it will not attach the volume. Also can you please share the -oyaml for the jivavolume CR.

von1994 commented 3 years ago

Hi @von1994 is the node CR for node36 removed from the cluster? If it is still there in notReady state then it will not attach the volume. Also can you please share the -oyaml for the jivavolume CR.

The node CR is still exist. I know that when the node is NotReady, the volume attach will be effected, but the controll-side should do something when node is notReady and let the new app Pod rebuild with volumes on the another node? The information what you want:

image image jivavolumes -oyaml: https://gist.github.com/von1994/f38c9289669cf5b3b935c11496f140ba

shubham14bajpai commented 3 years ago

Hi @von1994 It seems there is still a connection from the ctrl pod to the old node rejecting connection: 10.244.2.1 target already connected at 10.244.4.0. We reject login from multiple nodes as jiva supports only RWO and multiple iscsi logins can result in corruption. Can you restart the ctrl pod and check if the connection gets reset?

Some information can be helpful here:

von1994 commented 3 years ago

Hi @von1994 It seems there is still a connection from the ctrl pod to the old node rejecting connection: 10.244.2.1 target already connected at 10.244.4.0. We reject login from multiple nodes as jiva supports only RWO and multiple iscsi logins can result in corruption. Can you restart the ctrl pod and check if the connection gets reset?

Some information can be helpful here:

  • was the shutdown graceful?
  • check the netstat status from the node where the ctrl is scheduled and verify if a stale tcp connection to the shutdown node?

Node: shutdown -h now Pod: Terminated by controller-manager pod-evicted and --graceful-period=0 --force by administrator.

The netstat status like bellow: the old node: image

the new node scheduled to: image

mittachaitu commented 3 years ago

time="2021-08-10T10:20:32Z" level=info msg="Target is connected to initiator: 10.244.4.0:17161" time="2021-08-10T10:20:32Z" level=info msg="Login request received from initiator: iqn.1994-05.com.redhat:97998fbd485c, Session type: Normal, Target name:iqn.2016-09.com.openebs.jiva:pvc-fdf4e113-733a-4b4c-a6cc-bc8af1cf9318, ISID: 0x23d000002" time="2021-08-10T10:20:32Z" level=error msg="rsa: 0h, sa:false not supported" time="2021-08-10T10:20:32Z" level=warning msg="opcode: a3h err: check condition" time="2021-08-10T10:22:02Z" level=info msg="rejecting connection: 10.244.2.1 target already connected at 10.244.4.0" time="2021-08-10T10:22:04Z" level=info msg="rejecting connection: 10.244.2.1 target already connected at 10.244.4.0"

Hi @von1994 ,

From the logs & above netstat output, it looks like this regression was introduced via https://github.com/gostor/gotgt/pull/106 PR(As shubham stated gotgt PR will reject multiple login requests from different nodes to avoid corruption issues).

Problem:

Observation on ubuntu(18.04 & 20.04)/centos-7: When we ran shutdown -h now it triggered the logout command and then the system will go off. Can I know OS flavor and version?

Action item:

Workaround:

von1994 commented 3 years ago

When we ran shutdown -h now it triggered the logout command and then the system will go off. Can I know OS flavor and version?

Thanks! I'll try the workaround. OS: CentOS Linux 7 (Core), ID_LIKE="rhel fedora" Kernel: Linux ncm36 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

von1994 commented 3 years ago

When we ran shutdown -h now it triggered the logout command and then the system will go off. Can I know OS flavor and version?

Thanks! I'll try the workaround. OS: CentOS Linux 7 (Core), ID_LIKE="rhel fedora" Kernel: Linux ncm36 3.10.0-1160.11.1.el7.x86_64 #1 SMP Fri Dec 18 16:34:56 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The workaround that restarting the corresponding jiva-ctrl pod works. But the annother one that wait 15minutes or more based on sysctl net.ipv4.tcp_retries2 value did not work.

kmova commented 3 years ago

With 2.12.1 version, the restart is no longer required.