fix(manager): use rpc to check if a node is running or not

[RolandSherwin]

• After a reboot, the PID of the service might be taken by some other process and if we use the PID to check if our node is active, then we'd get positive there, but would fail when we try to access the RPC endpoint.
• So just try to connect to the RPC endpoint to see if the node is currently active or not
• Does this in parallel, as a connection failure will take 3 seconds to return.

[jacderida]

Hey Roland,

Good point about the restarts, but people in the community have noted issues with excessive CPU usage when using the RPC service. We may want to just think about this one. I seem to remember I made changes for looking up the location of the exe for the process, which was unique, because each exe has its own service directory. We might be able to do this after restart?

[RolandSherwin]

Hey @jacderida, that sounds good too. So do we know for sure that RPC is causing issues? It is just a tiny webserver that we rarely send requests to.

[jacderida]

I've had quite a few messages saying that it seems to spike CPU usage, and in turn that may be what is causing the node manager to take so long to refresh a large number of nodes. I have never had time to properly investigate and determine whether it is the real cause of the spikes, so I can't say for sure.

I have just had another thought though: if we are simply just checking the connection here, as opposed to running one of the commands, maybe that might be OK.

[RolandSherwin]

Right I see.

I'm okay with either approach. Just one thing to keep in mind is the time it'd take to refresh many nodes. The current method by checking the connection take 3 seconds to respond back if the node is down, i.e., rpc server is inactive. This is without any retry and by just checking for connection. So I had to do perform the checks in parallel for 50 nodes at a time.

But with your approach, I think even if we do it sequentially, we'd finish within a couple ms. In that case, we can go with yours.