help request: healthcheck

[pferrer-pdpaola]

Description

Hi,

We have Apisix installed where each upstream is a microservice for us instead of a node. The health check is performed at the node level, and we would like to perform the health check at the microservice level because it's possible that a microservice is failing due to an incorrect deployment while others on the same node are functioning correctly.

We have multiple nodes, and on each node, there are multiple microservices.

Is it possible to perform health checks separately for nodes and for individual microservices?

Best Regards

[kayx23]

There might be a misconception of what a node could be here. What is your "node"? A host? And you have microservices deployed on each host?

In APISIX, an upstream can be configured in multiple nodes. If health check is configured on upstream with upstream.checks.xxxx, each node in the upstream will be checked for health individually.

This can be examined with a request to the control API health check endpoint:

curl "http://127.0.0.1:9090/v1/healthcheck"

You get something like this for example:

[
  {
    "name": "/apisix/routes/example-healthcheck-route",
    "type": "http",
    "nodes": [
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 0,
          "success": 0
        },
        "ip": "172.24.0.5",
        "status": "healthy"
      },
      {
        "port": 80,
        "counter": {
          "http_failure": 0,
          "tcp_failure": 0,
          "timeout_failure": 3,
          "success": 0
        },
        "ip": "172.24.0.4",
        "status": "unhealthy"
      }
    ]
  },
  {
    "name": "/apisix/upstreams/example-healthcheck-route",
    "type": "http",
    "nodes": {}
  }
]

In this example, two nginx containers were deployed in a Docker network as sample upstreams. They're on the same physical host. Note how the first node is healthy and the second node is not.

[pferrer-pdpaola]

Hi Kayx23,

Thanks for the reply.

Yes, on each host, we have multiple microservices. The health check works correctly at the node level, but it would be better for us to do it at the microservice level.

My question is focused on whether we can perform the check for the microservice independently of whether it is sharing hosts with other microservices.

If two microservices share the same IP, the health check of one microservice affects the other because they share the IP. This is visible in the Apisix logs, where the health check of one impacts the other as they are on the same server.

[kayx23]

If two microservices share the same IP, the health check of one microservice affects the other because they share the IP.

Shouldn't they be deployed on different ports? then you can configure your upstream health checks as something like:

curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "example-route-with-healthcheck",
  "uri":"/",
  "upstream":{
    "type":"roundrobin",
    "nodes":{
      "192.168.4.85:5000": 1,
      "192.168.4.85:5001": 1
    },
    "checks":{
      "active": {
        "type": "http",
        "http_path": "/",
          "healthy": {
          "interval": 2,
          "successes": 1
        },
        "unhealthy": {
          "interval": 1,
          "http_failures": 2
        }
      }
    }
  }
}'

[pferrer-pdpaola]

We've considered the option of ports, but we need to work on ports 80 and 443 because we had issues with external connections that weren't on those ports before. If we configure them with different ports, would that solve the problem?

Additionally, we've thought about adding private IPs to the host, one for each microservice it has. It's an option we haven't tried yet.

We would prefer something cleaner to avoid filling the servers with IPs or having a large number of ports that make service management more complex. Would it be possible to add a check in Apisix to determine if it should handle it in one way or another?

[kayx23]

Let me get the question right. Say you have one host with an arbitrary IP 192.168.0.111:

• one microservice is listening on 80 and 443 (HTTP/HTTPS traffic for a web service)

Now, where is the second microservice listening on? They can't be 80 and 443 as well. Port conflicts.

Could you draw an architectural diagram of your setup if that's easier?

[pferrer-pdpaola]

I don't see an option to attach screenshots. I'll try to explain myself.

The requests enter through an external IP that balances to an Apisix cluster.

api.mydomain.com: 172.16.0.1

Then Apisix performs its tasks with the configured routes and upstreams.

Host 1 nginx: 192.168.0.111 Host 2 nginx: 192.168.0.112 Host 3 nginx: 192.168.0.113

UPSTREAM Microservice 1: ms1-1.mydomain.com pointing to host1 and ms1-2.mydomain.com pointing to Host2 UPSTREAM Microservice 2: ms2-1.mydomain.com pointing to host1 and ms2-2.mydomain.com pointing to Host2 UPSTREAM Microservice 3: Host3 It's not possible to configure a healtcheck in this scenario.

The health checks of UPSTREAM Microservice 1 affect UPSTREAM Microservice 2, and those of UPSTREAM Microservice 2 affect UPSTREAM Microservice 1.

What we are trying to achieve without configuring different ports or adding different IPs is to make the health checks independent even if the destination hosts are the same.

[kayx23]

I don't see an option to attach screenshots.

Either pasting from the clipboard or dragging and dropping to the input box should work.

UPSTREAM Microservice 1: ms1-1.mydomain.com pointing to host1 and ms1-2.mydomain.com pointing to Host2

Here do you mean Host 1 nginx routes to ms1-1 and Host 2 nginx routes to ms1-2? In other words, NGINX services are in front of these upstream miscroservices, correct?

If that's the case, I fail to understand how these two upstreams jinx with each other:

The health checks of UPSTREAM Microservice 1 affect UPSTREAM Microservice 2, and those of UPSTREAM Microservice 2 affect UPSTREAM Microservice 1.

Is your UPSTREAM (Microservice 1) configuration something like this:

{
  "id": "example-route-with-healthcheck",
  "uri":"/",
  "upstream":{
    "type":"roundrobin",
    "nodes":{
      "nginx1:80": 1,
      "nginx2:80": 1
    },
    "checks":{
      "active": {
        "type": "http",
        "http_path": "/",
          "healthy": {
          "interval": 2,
          "successes": 1
        },
        "unhealthy": {
          "interval": 1,
          "http_failures": 2
        }
      }
    }
  }
}

The name of the microservice won't be configured explicitly here. APISIX checks a specific path configured in upstream.checks.active.http_path (doc here). The request forwarding to specific microservices is entirely taken care of by NGINX.

If this understanding isn't quite right, please whiteboard.

[pferrer-pdpaola]

This is an example of the configuration of one upstream:

{ "nodes": [ { "host": "test-sysadmin-ha-01.domain.com", "port": 443, "weight": 1 }, { "host": "test-sysadmin-ha-02.domain.com", "port": 443, "weight": 1 } ], "timeout": { "connect": 600, "send": 600, "read": 600 }, "type": "roundrobin", "checks": { "active": { "concurrency": 2, "healthy": { "http_statuses": [ 200, 302 ], "interval": 5, "successes": 2 }, "host": "test-sysadmin.domain.com", "http_path": "/index.html", "https_verify_certificate": true, "port": 443, "timeout": 5, "type": "https", "unhealthy": { "http_failures": 5, "http_statuses": [ 429, 404, 500, 501, 502, 503, 504, 505 ], "interval": 5, "tcp_failures": 5, "timeouts": 5 } }, "passive": { "healthy": { "http_statuses": [ 200, 201, 202, 203, 204, 205, 206, 207, 208, 226, 300, 301, 302, 303, 304, 305, 306, 307, 308 ], "successes": 5 }, "type": "https", "unhealthy": { "http_failures": 5, "http_statuses": [ 429, 500, 503, 404 ], "tcp_failures": 5, "timeouts": 5 } } }, "scheme": "https", "pass_host": "pass", "name": "TEST SYSADMIN", "desc": "CON ALTA DISPONIBILIDAD", "keepalive_pool": { "idle_timeout": 60, "requests": 1000, "size": 320 } }

Here do you mean Host 1 nginx routes to ms1-1 and Host 2 nginx routes to ms1-2? In other words, NGINX services are in front of these upstream miscroservices, correct?

Yes, but ms2-1 and ms2-2 point to the same IP addresses as ms1-1 and ms2-2. The microservices are hosted on nginx (Laravel projects).

[pferrer-pdpaola]

?

[kayx23]

Hello. If this issue still persists, please provide a diagram of your setup for others to help better.

[kayx23]

Hope your issues are resolved. It wasn't entirely clear how everything was set up due to the mix use of terminologies.

There's a new doc on the topic that might be helpful: https://docs.api7.ai/apisix/how-to-guide/traffic-management/health-check

Closing this for now, but feel free to reopen if you have more questions. Thanks.