search

pnxtech / hydra-router

A service aware router for Hydra Services. Implements an API Gateway and can route web socket messages.
MIT License
60 stars 24 forks source link

Hydra-router / Docker Swarm / AWS: Getting SIGTERM #151

Closed mozrat closed 6 years ago

mozrat commented 6 years ago

Hello @cjus

I've tried to deploy my app into AWS this evening and I can't work out why hydra-router is constantly failing with a SIGTERM signal

Starting service hydra-router:1.4.31 on 10.0.2.8:5353
[2017-11-18T00:45:27.566Z] INFO (1 on 90b5a3641439): Starting service hydra-router:1.4.31 on 10.0.2.8:5353
ts: "2017-11-18T00:45:27.565Z"
serviceName: "hydra-router"
type: "info"
processID: 1
Detected IPv4 IPs:
* lo: 127.0.0.1 255.0.0.0
* lo: 10.0.2.8 255.255.255.255
* eth0: 10.0.2.5 255.255.255.0
* eth1: 172.18.0.3 255.255.0.0
[2017-11-18T00:45:27.580Z] INFO (1 on 90b5a3641439): Starting service hydra-router:1.4.31 on 10.0.2.8:5353
serviceName: "hydra-router"
[2017-11-18T00:45:27.611Z] INFO (1 on 90b5a3641439): HR: hydra-router adding [get]/v1/router/list/:thing
serviceName: "hydra-router"
[2017-11-18T00:45:27.616Z] INFO (1 on 90b5a3641439): HR: hydra-router adding [get]/
serviceName: "hydra-router"

[....]

[2017-11-18T00:46:59.517Z] FATAL (1 on 90b5a3641439): Received SIGTERM
serviceName: "hydra-router"
[2017-11-18T00:46:59.519Z] ERROR (1 on 90b5a3641439): Service is shutting down.
ts: "2017-11-18T00:46:59.519Z"
serviceName: "hydra-router"
type: "error"
processID: 1
[2017-11-18T00:46:59.521Z] ERROR (1 on 90b5a3641439): Service is shutting down.
serviceName: "hydra-router"
ts: "2017-11-18T00:46:59.519Z"
type: "error"

Before that, in the logs, my services were populating their routes happily.

Thanks for any pointers you can give.

cjus commented 6 years ago

@mozrat Are you able to post your config file? Also, is it running in a container? If so, can you post the Dockerfile?

mozrat commented 6 years ago

@cjus Yes - happy to post those. This works fine in Docker on my development machine started with docker-compose. I'm now trying to get it into AWS with Docker Swarm (and a compose file)

config.json

{
  "cluster": false,
  "processes": 0,
  "externalRoutes": {},
  "routerToken": "",
  "disableRouterEndpoint": false,
  "risingStack": false,
  "debugLogging": true,
  "hydra": {
    "serviceName": "hydra-router",
    "serviceDescription": "Service Router",
    "serviceIP": "",
    "servicePort": 5353,
    "serviceType": "router",
    "plugins": {
      "logger": {
        "logRequests": true,
        "redact": ["password"],
        "elasticsearch": {
          "host": "localhost",
          "port": 9200,
          "index": "hydra"
        }
      }
    },
    "redis": {
      "url": "redis://service-redis:6379/15"
    }
  }
}

Dockerfile

FROM flywheelsports/hydra-router:1.4.31
ADD config.json /usr/src/app/config/config.json

docker-compose (slightly redacted)

version: '3.1'

services:

  api_proxy:
    image: registry.gitlab.com/xxx/xxx_api_proxy:latest
    logging:
      driver: json-file
    environment:
      NODE_ENV: 'production'
      APP_NAME: '${APP_NAME}'
    networks:
      default: {}
      network:
        aliases:
          - api-proxy
    secrets:
      - demonightly.service.pub
      - demonightly.service.pem
      - demonightly.api_proxy.dbConnection.txt
    depends_on:
      - cache

  api_router:
    image: registry.gitlab.com/xxx/xxx/api_router:latest
    networks:
      network:
        aliases:
          - api-router
    depends_on:
      - cache

  auth:
    image: registry.gitlab.com/xxx/xxx_auth:latest
    environment:
      NODE_ENV: 'production'
      APP_NAME: '${APP_NAME}'
    networks:
      network:
        aliases:
          - service-auth
    secrets:
      - demonightly.service.pub
      - demonightly.service.pem
      - demonightly.auth.dbConnection.txt
    depends_on:
      - cache

  cache:
    image: registry.gitlab.com/xxx/xxx/cache:latest
    networks:
      network:
        aliases:
          - service-redis
    logging:
      driver: awslogs

  datastore:
    image: registry.gitlab.com/xxx/xxx_datastore:latest
    environment:
      NODE_ENV: 'production'
      APP_NAME: '${APP_NAME}'
    networks:
      network:
        aliases:
          - service-datastore
    secrets:
      - demonightly.service.pub
      - demonightly.service.pem
      - demonightly.datastore.dbConnection.txt
    depends_on:
      - cache

  reverse_proxy:
    image: registry.gitlab.com/xxx/xxx/reverse_proxy:latest
    networks:
      network:
        aliases:
          - reverse-proxy
    ports:
      - '${PORT}:80'
    depends_on:
      - ui
      - api_router

  ui:
    image: registry.gitlab.com/xxx/xxx_ui:latest
    environment:
      NODE_ENV: 'production'
      APP_NAME: '${APP_NAME}'
    networks:
      network:
        aliases:
          - service-ui
    secrets:
      - demonightly.service.pub
      - demonightly.service.pem
      - demonightly.ui.dbConnection.txt
    depends_on:
      - cache

networks:
  network:
    driver: overlay

secrets:
  demonightly.service.pub:
    external: true
  demonightly.service.pem:
    external: true
  demonightly.api_proxy.dbConnection.txt:
    external: true
  demonightly.auth.dbConnection.txt:
    external: true
  demonightly.datastore.dbConnection.txt:
    external: true
  demonightly.ui.dbConnection.txt:
    external: true
cjus commented 6 years ago

I haven't personally used docker-compose. I know I should.

@emadum any ideas? I know you're a big fan of docker-compose ;-)

To get a better understanding of the problem have you tried launching your services in AWS without a docker compose file?

mozrat commented 6 years ago

Hi @cjus. I have deployed the services manually to the Docker swarm on AWS with these commands

docker network create --driver overlay cjus_network
docker service create --name service-redis --network cjus_network --with-registry-auth --hostname service-redis redis:latest
docker service create --name demonightly_api_router --network cjus_network --with-registry-auth --hostname api-router registry.gitlab.com/xxx/xxx_cloud/api_router:latest
docker service create --name demonightly_datastore --secret demonightly.service.pem --secret demonightly.service.pub --secret demonightly.datastore.dbConnection.txt --network cjus_network --with-registry-auth --hostname service-datastore --env NODE_ENV=production --env APP_NAME=demonightly registry.gitlab.com/xxx/xxx_datastore:latest

Here is the log from service-redis

_ _ _ ____ _
| | | |_ _ __| |_ __ __ _ | _ \ ___ _ _| |_ ___ _ __
| |_| | | | |/ _` | '__/ _` | | |_) / _ \| | | | __/ _ \ '__|
| _ | |_| | (_| | | | (_| | | _ < (_) | |_| | || __/ |
|_| |_|\__, |\__,_|_| \__,_| |_| \_\___/ \__,_|\__\___|_|
|___/
Starting service hydra-router:1.4.31 on 10.0.2.4:5353
[2017-11-18T17:49:44.358Z] INFO (1 on api-router): Starting service hydra-router:1.4.31 on 10.0.2.4:5353
ts: "2017-11-18T17:49:44.358Z"
serviceName: "hydra-router"
type: "info"
processID: 1
Detected IPv4 IPs:
* lo: 127.0.0.1 255.0.0.0
* lo: 10.0.2.4 255.255.255.255
* eth0: 10.0.2.5 255.255.255.0
* eth1: 172.18.0.3 255.255.0.0
[2017-11-18T17:49:44.371Z] INFO (1 on api-router): Starting service hydra-router:1.4.31 on 10.0.2.4:5353
serviceName: "hydra-router"
[2017-11-18T17:49:44.405Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/stats
serviceName: "hydra-router"
[2017-11-18T17:49:44.407Z] INFO (1 on api-router): HR: hydra-router adding [get]/
serviceName: "hydra-router"
[2017-11-18T17:49:44.407Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/clear
serviceName: "hydra-router"
[2017-11-18T17:49:44.408Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/list/:thing
serviceName: "hydra-router"
[2017-11-18T17:49:44.410Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/version
serviceName: "hydra-router"
[2017-11-18T17:49:44.410Z] INFO (1 on api-router): HR: hydra-router adding [get]/hydra-router
serviceName: "hydra-router"
[2017-11-18T17:49:44.411Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/refresh
serviceName: "hydra-router"
[2017-11-18T17:49:44.412Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/health
serviceName: "hydra-router"
[2017-11-18T17:49:44.412Z] INFO (1 on api-router): HR: hydra-router adding [get]/hydra-router/
serviceName: "hydra-router"
[2017-11-18T17:49:44.412Z] INFO (1 on api-router): HR: hydra-router adding [get]/hydra-router/:rest
serviceName: "hydra-router"
[2017-11-18T17:49:44.415Z] INFO (1 on api-router): HR: hydra-router adding [get]/v1/router/refresh/:service
serviceName: "hydra-router"
[2017-11-18T17:49:44.415Z] INFO (1 on api-router): HR: hydra-router adding [post]/v1/router/message
serviceName: "hydra-router"
[2017-11-18T17:49:55.931Z] INFO (1 on api-router): HR: Incoming channel message:
{
    "to": "hydra-router:/refresh",
    "frm": "datastore-service:/",
    "mid": "bb50903e-1208-4424-888e-c5fc04a10ac7",
    "ts": "2017-11-18T17:49:55.893Z",
    "ver": "UMF/1.4.5",
    "bdy": {
        "action": "refresh",
        "serviceName": "datastore-service"
    }
}
serviceName: "hydra-router"
[2017-11-18T17:51:15.756Z] FATAL (1 on api-router): Received SIGTERM
serviceName: "hydra-router"
[2017-11-18T17:51:15.759Z] ERROR (1 on api-router): Service is shutting down.
ts: "2017-11-18T17:51:15.759Z"
serviceName: "hydra-router"
type: "error"
processID: 1
[2017-11-18T17:51:15.760Z] ERROR (1 on api-router): Service is shutting down.
serviceName: "hydra-router"
ts: "2017-11-18T17:51:15.759Z"
type: "error"
processID: 1

Just to restate that these same containers work fine on my local docker-machine. Thanks for any thoughts you have

cjus commented 6 years ago

Looked at your config.json above. It looks old and incomplete. In particular queuerDB is missing. Can you try using a modified version of the one below?

{
  "externalRoutes": {},
  "routerToken": "",
  "disableRouterEndpoint": false,
  "debugLogging": true,
  "queuerDB": 0,
  "requestTimeout": 5,
  "forceMessageSignature": false,
  "signatureSharedSecret": "d632dd6d-fb75-fake-babf-ee1364f3716c",
  "hydra": {
    "serviceName": "hydra-router",
    "serviceDescription": "Service Router",
    "serviceIP": "",
    "servicePort": "5353",
    "serviceType": "router",
    "plugins": {
      "logger": {
        "logRequests": false,
        "noFile": true,
        "toConsole": false,
        "redact": ["password"],
        "elasticsearch": {
          "rotate": "daily",
          "host": "host",
          "port": 9200,
          "index": "hydra"
        }
      }
    },
    "redis": {
      "url": "redis://host:6379/15"
    }
  }
}
mozrat commented 6 years ago

Hi @cjus - just an interim update. I added the new config.json and rebuilt. At the moment my hydra-router isn't staying up, but is showing less in the logs

My config is exactly as yours above except redis.url and I'm only seeing this before the container stops

_ _ _ ____ _
| | | |_ _ __| |_ __ __ _ | _ \ ___ _ _| |_ ___ _ __
| |_| | | | |/ _` | '__/ _` | | |_) / _ \| | | | __/ _ \ '__|
| _ | |_| | (_| | | | (_| | | _ < (_) | |_| | || __/ |
|_| |_|\__, |\__,_|_| \__,_| |_| \_\___/ \__,_|\__\___|_|
|___/
Starting service hydra-router:1.4.31 on 10.0.1.4:5353
Detected IPv4 IPs:
* lo: 127.0.0.1 255.0.0.0
* lo: 10.0.1.4 255.255.255.255
* eth0: 10.0.1.5 255.255.255.0
* eth1: 172.18.0.3 255.255.0.0

Sorry - I wish I could be more helpful in my reporting of this issue.

cjus commented 6 years ago

@mozrat for speed in resolving this issue it might make sense to chat via the hydranpm.slack.com channel. If you're interested, send me an email to carlos.justiniano@gmail.com with your desired username.

mozrat commented 6 years ago

After a discussion on Slack we found two issues

1) Don't run hydra-router on AWS t2.micro instances else you'll get out of memory type errors 2) In your Dockerfile for your hydra-router include the following

EXPOSE 5353
HEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost:5353/v1/router/health || exit 1

(where 5353 is your service port in config.json)

The root cause of my issue was Docker swarm taking down unhealthy containers due to a missing healthcheck

Lots of thanks to @cjus for his time