Specstore stopped responding requests after moving to bigger node

[zelima]

We are getting timeout errors from specstore - data push is not working

Acceptance Criteria

• [x] specstore responding as expected on https://api.datahub.io/source/core/finance-vix/successful
• [x] able to push

Tasks

• [x] Do the analysis and debug

Analysis

So in few hours after moving specstore to new, bigger node it stopped responding on requests. Seemed like nginx was not resolving spesctore. Debbuging a bit we found following:

• Nginx by itself is working fine, all other services are responding as expected.
  • Only getting timeouts from specstore

specstore by itself is and was working fine until suddenly it stopped responding on requests

• Sure it was working cause I see the logs after latest deploy and datasets have been successfully processed
• Sure it is working (as a service) cause:
  • Scheduled jobs are run and processed fine, even after it stopped responding (I can see the newest revisions)
  • ssh-ed into the container and server is up and running
  • CURLed http://localhost:8000/source/core/finance-vix/1 and is responding with expected data
  • deploy is complete and everything is green in GCE kuberntetes UI

[zelima]

As occurred we were seeing 2 different IP addresses for specstore when curling nginx.

curl from nginx:

* Connected to specstore (**.**.***.IP1) port 8000 (#0)
> GET /source/core/finance-vix/1 HTTP/1.1

error log in nginx:

2018/07/19 17:39:11 [info] 9#9: *59690 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while connecting to upstream, client: **.***.*.*, server: , request: "GET /source/core/finance-vix/1 HTTP/1.1", upstream: "http://**.**.***.IP2:8000/source/core/finance-vix/1", host: "api.datahub.io"

Meaning nginx resolved specstore and then specstore moved to new IP (due to node migration). As a solution we had to just restart nginx and it resolved specstore on new IP

[zelima]

FIXED