Solview improvements - feedback from @ruuda 2021q3

[kucharskim]

https://chorusone.slack.com/archives/CV89K1VB2/p1632925506230000?thread_ts=1632912320.217500&cid=CV89K1VB2

It registers variables for the gauges that it serves, and then updates the gauges. Not sure if these gauges include timestamps, but it doesn't appear so. This means that:

• Currently there is no problem because there is no error handling, the application will just crash if any RPC call fails, but I suppose you want it to continue running because the RPC (especially the public one) is flaky
• In that case, if updating a metric fails, then the old value will remain, and the /metrics endpoint will serve stale data
• This means that in Prometheus, the data will still be present, only it will stop changing, but you can't tell in Prometheus that it stopped changing because the collector is failing. Then when you get misleading metrics, you might stare at the metrics and think that something on the network halted, while really the collector is failing.

Also:

• There is double polling here: solview polls the RPC, and Prometheus polls solview. Without timestamped metrics, Prometheus will create data points at the time where it polled, but the data was retrieved from the RPC earlier. There's a time.sleep(5) in the update loop, so the data might be 5 seconds + time it took to do all the polls old by the time Prometheus sees it.
• Data is polled in individual RPC calls, but then presented to Prometheus in a single request. Without timestamps, this means Prometheus will create data points at the same time, even though the data was collected at different times. For watch_accounts for example, what might happen is that we own accounts X with 100 SOL, and Y with 100 SOL, so you watch accounts X and Y. solview queries X and finds 100 SOL. Then we do a transfer from X to Y. Then solview queries Y and finds 200 SOL. Then Prometheus queries solview, and it gets X=100, Y=200. You build a nice dashboard that shows our total balance with sum(solview_account_balance). And now your dashboard shows that we briefly owned 300 SOL. (You also get this problem if two solview instances run at the same time, like we ran into with solido .)
• I think even timestamping the metrics will not fix this, because Prometheus just holds the value of a sample until the next value.

There is no easy way to fix this, it requires atomically reading multiple accounts, something like read transactions. There exists only a single RPC call that can do this, getMultipleAccounts, but it is very low-level. If you want to read the balances of e.g. SPL token accounts, then you have to manually parse the data. (I complained about this many times to the Solana team but they don't seem to understand the value of read transactions ...)

Some Python footguns:

• The requests default timeout is infinite, so you if the server accepts the TCP connection but doesn't respond, the application will hang indefinitely
• From just time.sleep(5) you can't tell how long it sleeps unless you happen to know that time.sleep takes seconds. But in Python it takes seconds, in Java it takes milliseconds, and in Haskell it takes microseconds, so I am never quite sure without consulting the documentation. It's easy to remove the ambiguity: write sleep_seconds = 5 time.sleep(sleep_seconds). (Usually it's nicer to use keyword args, i.e. time.sleep(seconds=5), but unfortunately for the particular case of time.sleep this is not possible because it's implemented in C and doesn't handle keyword args.)

[kucharskim]

Another one from @ruuda, in pull request #1:

Also if you want to handle the exceptions instead of crashing the app, try this one weird trick to make your application more reliable.