Health Endpoint

[reqlez]

Oracle needs an endpoint where somebody can check it's health. I propose /isHealthy for the endpoint name. Please suggest others, if there is a better standard to use.

The health of the oracle is good to know, for different applications, example:

• failover / redundancy. If somebody wants to add a few oracles to a load balancer, their LB needs to check health somehow.
• monitoring / alerts so that the operator know that there is an issue with the oracle that needs to be addressed.
• Other examples? Maybe to be used in a kubernetes / docker compose setup ?

Currently, Ergo Node does not have a Health endpoint either, I also made a comment about a similar endpoint for node here: https://github.com/ergoplatform/ergo/issues/1755#issuecomment-1436074163

Some things to discuss:

1. Should oracle check the node it connects to ? if several nodes are present ( saw a new issue opened with multi node support ), in that case, check all and return healthy if at least one is okay? Fanta mentioned some Kung Fu regex to check this:

"GET /info and if /lH.(\d+),[\s\S]P.*\1,/gm matches, fullHeight is equal to maxPeerHeight, which would mean the node is synced"

However, I don't think EQUAL is the way to go here, I would do at least 1 block behind to be considered synced. To add, additional comparison would have to be implemented as well, example: "lastIncomingMessage": 1656365850283 VS "currentSystemTime": 1656365854007 to make sure the node is actually communicating with the network.

IMO, the best way to solve above, is add a Health endpoint to Ergo Node ( addressing the above issue linked ), so that Oracle can just check that endpoint instead of adding a bunch of extra code to Oracle.

2. What would trigger an unhealthy state, otherwise? If ergonode is healthy? Some examples:
   • Has oracle posted a datapoint in the last X amount of time? what should be X?
   • Another indicator, that would catch the oracle being down quicker than above? Maybe look at last error message thrown?

Any other ideas?

[greenhat]

I agree that implementing complex logic to determine a node's health on the oracle side is not ideal. Instead, the node should provide a simple endpoint indicating its status as either green or red.

The ultimate metric for determining oracle's health is whether it has posted a data point within the last X blocks. However, we must first define what "posted" means. While the Oracle sends transactions containing data points, it does not know if these transactions are included in the chain.

To ensure complete accuracy, we need to verify that transactions with data point boxes are indeed included in the chain. Internally, we have our last posted data point box from the node wallet scan.

Do the remaining question is how to determine X? Ideally, we want it to be no older than the last epoch, but realistically we probably should leave some leeway (2 epochs old?) to weather out mempool hiccups, etc.

[greenhat]

ref #53