Health Check against an instance using CLI

[rdlrt]

It would be nice to have a a healthcheck ping from CLI against a specific instance (eg: -b http://127.0.0.1:8053/api/v0 , that will check and report stats below behind the scenes (currently achieved via bash script grest-poll.sh ):

• [x] Query target's tip to check that the last block known to instance is within 600 seconds, use timeout of 2 seconds to establish connection to this endpoint (anything more being too slow to accept connection)
• [ ] Be able to verify (using <baseURL>/ endpoint) that each path from spec is available at the target instance (current implementation here)
• [x] Check every record in control table to check they're relatively recent (see here )
• [x] Check Observation limit is uniform
• [x] Do a Get/POST to test a sample request
• [x] Optionally - at the end of it all, with a debug argument (eg: -d ), provide a summary of the endpoint results.
• [x] Exit with return code 1 if any of the tests fail, and 0 if they all pass.

If all of the above is achievable, this will help us use something like koios-cli healthcheck [-d] for actual healthcheck from haproxy itself, as an alternative to using bash script we currently use.

[mkungla]

Definitely good use-case to add. Should that command be top level command or belong under some new subcommand? e.g. koios-cli healthcheck ... vs koios-cli <command-group?> healthcheck ...

Since most of tasks are there. e.g. with current main koios-cli api get /control_table?key=eq.stake_distribution_lbh

I guess just some more clarification about intended behavior and In general it should be fairly easy to implement .

[rdlrt]

Sorry for the delay, I've updated the description (was originally just a place holder) - let me know if it makes sense , and if any additional clarifications would be needed

[mkungla]

Custom GET/POST/HEAD request are already possible:

common options apply to all sub commands under api

koios-cli  api --help

--port value, -p value  Set port (default: 443)
--host value            Set host (default: "api.koios.rest")
--api-version value     Set API version (default: "v0")
--scheme value          Set URL scheme (default: "https")
--origin value          Set Origin header for requests. (default: "https://github.com/cardano-community/koios-go-client")
--rate-limit value      Set API Client rate limit for outgoing requests (default: 5)

• GET e.g. koios-cli api --host api.koios.rest --port 443 get /tip

koios-cli api get --help

NAME:
   koios-cli api get - send GET request to the specified API endpoint

USAGE:
   koios-cli api get [command options] [endpoint]

• HEAD e.g. koios-cli api --host api.koios.rest --port 443 head /tip

koios-cli api head --help

NAME:
   koios-cli api head - head issues a HEAD request to the specified API endpoint

USAGE:
   koios-cli api head [command options] [endpoint]

• POST e.g. koios-cli api --host api.koios.rest --port 443 post /endpoint '{"some":"payload"}'

koios-cli api post --help

NAME:
   koios-cli api post - send POST request to the specified API endpoint

USAGE:
   koios-cli api post [command options] [endpoint] [payload]

[mkungla]

Optionally - at the end of it all, with a debug argument (eg: -d ), provide a summary of the endpoint results.

opted right now for --quiet flag when provided will not print nothing to std out

[mkungla]

@rdlrt so you can test out v0.2.0 with preliminary implementation of healthcheck.

koios-cli healthcheck is basically with default options equivalent

koios-cli healthcheck --timeout 2  --age 600 --port 443 --host "api.koios.rest" --api-version v0 --scheme "https" 

default output is something like this

2022/03/17 11:09:05 [  OK  ]:  check-tip  -  tip age 18s
2022/03/17 11:09:05 [  OK  ]:  check-cache-status(eq.stake_distribution_lbh)  -
2022/03/17 11:09:06 [  OK  ]:  check-cache-status(eq.pool_history_cache_last_updated)  -  time diff 45s
2022/03/17 11:09:06 [  OK  ]:  check-cache-status(eq.last_active_stake_validated_epoch)  -  time diff 45s
2022/03/17 11:09:06 [  OK  ]:  check-limit  -  PostgREST config limit is 999
2022/03/17 11:09:07 [  OK  ]:  check-endpoint(/tx_metalabels)  -  got valid response
2022/03/17 11:09:07 [  OK  ]:  check-rpcs  -  not implemented
2022/03/17 11:09:08 [  OK  ]:  check-endpoint(/epoch_info?_epoch_no=326)  -  got valid response

for haproxy you probably would like to use --quiet flag which will not output nothing and exits respectively with statuscode 0 or 1

for more detailed json output you can use --json flag which outputs something like this

{
 "chk_tip": {
  "request_url": "https://api.koios.rest:443/api/v0/tip",
  "request_method": "GET",
  "status_code": 200,
  "status": "200 OK",
  "date": "Thu, 17 Mar 2022 09:12:31 GMT",
  "content_range": "0-0/*",
  "stats": {
   "req_started_at": "2022-03-17T09:12:30.777363806Z",
   "req_dns_lookup_dur": 1075552,
   "tls_hs_dur": 318729829,
   "est_cxn_dur": 169426580,
   "ttfb": 678905795,
   "req_dur": 679180386,
   "req_dur_str": "679.180386ms"
  },
  "data": {
   "abs_slot": 55942047,
   "block_no": 7007224,
   "block_time": "2022-03-17T09:12:18",
   "epoch_no": 327,
   "epoch_slot": 41247,
   "hash": "356784d5b20d3a648f6ffb29388d309fc42d9513428a044d39c5a4e4a8abb28e"
  },
  "last_block_age_ns": 13000000000,
  "last_block_age_str": "13s",
  "tip_timeout_ns": 2000000000,
  "tip_timeout_str": "2s"
 },
 "chk_rpcs": {},
 "chk_cache_status": [
  {
   "task": "eq.stake_distribution_lbh",
   "status": "ok",
   "message": "block diff 110"
  },
  {
   "task": "eq.pool_history_cache_last_updated",
   "status": "ok",
   "message": "time diff 17s"
  },
  {
   "task": "eq.pool_history_cache_last_updated",
   "status": "ok",
   "message": "time diff 17s"
  }
 ],
 "chk_limit": {
  "task": "check-limit",
  "status": "ok",
  "message": "PostgREST config limit is 999"
 },
 "chk_endpt_get": [
  {
   "task": "check-endpoint(/tx_metalabels)",
   "status": "ok",
   "message": ""
  },
  {
   "task": "check-endpoint(/epoch_info?_epoch_no=326)",
   "status": "ok",
   "message": ""
  }
 ]
}

With this you should be able to healthcheck individual instances, so next step would be to discuss how the implementation of api compare and open api schema checksum checks should be performed and should these be in same check queue or as separate commands!?