Http rest api forwarding to incorrect pod from another pod

[yogiverma007]

### BUG REPORT We have one pod with one container allocation. We have Java Spring boot application. Use Case: A request which came on Kubernetes service is forwarded on one pod and a thread is created. This thread will do some work, after completing its work and this pod have to notify some other pod. So in our case we are notifying using http rest api. So my application is hitting HTTP Restful API from one pod's container to other pod's container.

• Pod1-Container-IP: 10.200.141.1
• Pod2-Container-IP: 10.200.146.1
• Pod3-Container-IP: 10.200.149.1

From Pod1-Container-IP: 10.200.141.1 The URL hit is: http://10.200.146.1:9000/peer/resolve/PTMIN00971c370baceae3c399105ff045a2

Expected: Ideally it should land on 10.200.146.1. Actual behaviour: But it landed on Pod3-Container 10.200.149.1. To verify this this issue http response of this api contains the IP address of Pod container on which the request landed. So in response of the api I am getting 10.200.149.1 IP Address. The event occurs in (2-5)% from total number of request.

How to reproduce: Hit large number of request with this scenario. Software Versions:

• weave version: 1.6.4
• docker version: 17.02
• uname -a 7.3_514 CentOS
• kubectl version: 1.9.2

[brb]

@yogiverma007 Thanks for the issue.

This thread will do some work, after completing its work and this pod have to notify some other pod. So in our case we are notifying using http rest api.

Does the thread doing the work call the HTTP Rest API via Kubernetes Service IP addr?

[brb]

Closing due to inactivity.

[yogiverma007]

@brb Apologies for delay in response, No the thread doesn't calls the rest api via Kubernetes Service IP addr. It is actual IP address of pod.

[brb]

@yogiverma007 Could you provide PCAP traces from each host captured with tcpdump -i weave -w "$(hostname).pcap" when you hit the problem?

[yogiverma007]

Will share you the traces in some time.