search

hashicorp / consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.
https://www.consul.io
Other
28.39k stars 4.43k forks source link

Check definitions not reflected in output from agent/health/service/id/:service-id #20409

Open Wnthr opened 9 months ago

Wnthr commented 9 months ago

Overview of the Issue

After registering a service with checks at the agent/service/register endpoint, the check definition is not reflected when using the agent health/service/id/:service-id endpoint.

Reproduction Steps

Steps to reproduce this issue, eg:

  1. Start a consul agent
  2. Register a service (example payload below)
  3. Query that service
# cat pulp-lb.json
{
  "name": "pulp",
  "id": "pulp-lb",
  "tags": ["lb"],
  "tagged_addresses": {
    "lan_ipv4": {
      "address": "10.0.0.100",
      "port": 8080
    }
  },
  "checks": [
    {
      "name": "nginx.service",
      "CheckId": "pulp-nginx-service",
      "args": ["/usr/local/bin/check-systemd-service.sh", "nginx"],
      "DeregisterCriticalServiceAfter": "15m",
      "Interval": "15s"
    },
    {
      "name": "consul.service",
      "CheckId": "pulp-consul-service",
      "args": ["/usr/local/bin/check-systemd-service.sh", "consul"],
      "DeregisterCriticalServiceAfter": "15m",
      "interval": "15s"
    },
    {
      "name": "consul-template.service",
      "CheckId": "pulp-consul-template-service",
      "args": ["/usr/local/bin/check-systemd-service.sh", "consul-template"],
      "DeregisterCriticalServiceAfter": "15m",
      "interval": "15s"
    },
    {
      "name": "pulp_tcp_lan",
      "CheckId": "pulp-tcp-lan",
      "TCP": "10.0.0.100:8080",
      "interval": "5s"
    }
  ]
}

Registering service using curl --request PUT --data @pulp-lb.json http://127.0.0.1:8500/v1/agent/service/register?replace-existing-checks=true

Output from http://127.0.0.1:8500/v1/agent/health/service/id/pulp-lb

{
  "AggregatedStatus": "passing",
  "Service": {
    "ID": "pulp-lb",
    "Service": "pulp",
    "Tags": [
      "lb"
    ],
    "Meta": {},
    "Port": 0,
    "Address": "",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.0.0.100",
        "Port": 8080
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "Checks": [
    {
      "Node": "master",
      "CheckID": "pulp-consul-service",
      "Name": "consul.service",
      "Status": "passing",
      "Notes": "",
      "Output": "",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    },
    {
      "Node": "master",
      "CheckID": "pulp-consul-template-service",
      "Name": "consul-template.service",
      "Status": "passing",
      "Notes": "",
      "Output": "",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    },
    {
      "Node": "master",
      "CheckID": "pulp-nginx-service",
      "Name": "nginx.service",
      "Status": "passing",
      "Notes": "",
      "Output": "",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    },
    {
      "Node": "master",
      "CheckID": "pulp-tcp-lan",
      "Name": "pulp_tcp_lan",
      "Status": "passing",
      "Notes": "",
      "Output": "TCP connect 10.0.0.100:8080: Success",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    }
  ]
}

Consul info for Client (no Server in test setup)

Client info ``` # consul info agent: check_monitors = 6 check_ttls = 0 checks = 10 services = 4 build: prerelease = revision = 7736539d version = 1.17.2 version_metadata = consul: acl = disabled bootstrap = true known_datacenters = 1 leader = true leader_addr = 10.0.0.100:8300 server = true raft: applied_index = 14527 commit_index = 14527 fsm_pending = 0 last_contact = 0 last_log_index = 14527 last_log_term = 10 last_snapshot_index = 0 last_snapshot_term = 0 latest_configuration = [{Suffrage:Voter ID:25010851-3a85-32f0-a6d0-fcf94dd3f0e5 Address:10.0.0.100:8300}] latest_configuration_index = 0 num_peers = 0 protocol_version = 3 protocol_version_max = 3 protocol_version_min = 0 snapshot_version_max = 1 snapshot_version_min = 0 state = Leader term = 10 runtime: arch = amd64 cpu_count = 2 goroutines = 223 max_procs = 2 os = linux version = go1.21.6 serf_lan: coordinate_resets = 0 encrypted = false event_queue = 1 event_time = 10 failed = 0 health_score = 0 intent_queue = 1 left = 0 member_time = 10 members = 1 query_queue = 0 query_time = 1 serf_wan: coordinate_resets = 0 encrypted = false event_queue = 0 event_time = 1 failed = 0 health_score = 0 intent_queue = 0 left = 0 member_time = 1 members = 1 query_queue = 0 query_time = 1 ``` Client config JSON ``` { "retry_join" : ["10.0.0.100"], "data_dir": "/opt/consul/data", "log_level": "INFO", "server": true, "node_name": "master", "addresses": { "https": "10.0.0.100" }, "bind_addr": "10.0.0.100", "ui": true, "bootstrap_expect": 1 } ```

Operating system and Environment details

NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

Log Fragments

# journalctl -fu consul
Jan 30 21:48:23 social-bonefish consul[1117]: 2024-01-30T21:48:23.606Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14501&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:40846 error="context canceled"
Jan 30 21:48:23 social-bonefish consul[1117]: 2024-01-30T21:48:23.607Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/vault?index=14509&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:40848 error="context canceled"
Jan 30 21:48:23 social-bonefish consul[1117]: 2024-01-30T21:48:23.607Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14501&passing=1&stale=&tag=lb&wait=60000ms" from=127.0.0.1:40860 error="context canceled"
Jan 30 21:50:18 social-bonefish consul[1117]: 2024-01-30T21:50:18.544Z [INFO]  agent: Synced check: check=service:pulp-backend:1
Jan 30 21:51:31 social-bonefish consul[1117]: 2024-01-30T21:51:31.494Z [INFO]  agent: Synced check: check=service:vault-backend:1
Jan 30 21:52:47 social-bonefish consul[1117]: 2024-01-30T21:52:47.808Z [INFO]  agent: Synced check: check=service:vault-lb:1
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.593Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14518&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:35612 error="context canceled"
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.594Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/vault?index=14528&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:35620 error="context canceled"
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.595Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/vault?index=14528&passing=1&stale=&tag=lb&wait=60000ms" from=127.0.0.1:35616 error="context canceled"
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.597Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14518&passing=1&stale=&tag=lb&wait=60000ms" from=127.0.0.1:35592 error="context canceled"
vijayraghav-io commented 9 months ago

@Wnthr , looking into this. The check definitions - DeregisterCriticalServiceAfter, interval, TCP are in output with default values, actual values provided during service registration are not displayed in output.