saveStats store incomplete results if test runs overlap

[michielbdejong]

testAll first wipes all stats, then tests limit and balance, then tests route broadcasts, and then saves the new stats.

If two testAll runs overlap, and the new one is testing limit&balance while the older one arrives at saving the new stats, then empty values for limit&balance will be stored.

In the course of fixing this, we could also start keeping a full log of all tests performed, and then have a separate stats collection process. that might be cleaner.

[michielbdejong]

this would then also be a good time to classify test types. so far, testHost and testPeer have been overlapping a bit, and testHost is actually specific to webfinger-based peering.

test script 1 (single-host, proactive):

// log peer test start
get_balance
get_limit
send_request:broadcast_routes
send_request:quote_request:curve // for the ledger we just announced ourselves
send_request:quote_request:by-dest // for the ledger we just announced ourselves
send_request:quote_request:by-src // for the ledger we just announced ourselves
send_transfer // to the ledger we just announced ourselves

test script 2 (in response to send_transfer):

// log incoming transfer
cancel_transfer // if it is timed out
fulfill_condition // otherwise, if we have the preimage
reject_incoming_transfer // otherwise, if we do not

test script 3 (in response to fulfill_condition):

// log incoming fulfill

test script 4 (in response to reject_incoming_transfer):

// log incoming reject

test script 5 (in response to cancel_transfer):

// log incoming cancel

test script 6 (in response to send_request:broadcast_routes):

// log incoming route announce
send_transfer // if it's a loop route

test script 7 (in response to send_request:quote:curve):

// log incoming curve quote request
// respond if we have the curve, reject if not

test script 8 (in response to send_request:quote:by-dest):

// log incoming dest quote request
// respond if we have the curve, reject if not

test script 9 (in response to send_request:quote:by-src):

// log incoming src quote request
// respond if we have the curve, reject if not

[michielbdejong]

I see a refactor coming up... :)

ilp-node can log everything it sees, keeping one log per peer. Log entries would always have the peerHostname and the timestamp, an incoming/outgoing flag, and the exact RPC call contents. Optionally, it would have a 'measurement', for instance latency, errorType, etc.

connectorland would then receives these logs, and 1) writes them to stdout, and 2) displays a truncated version on per-host analysis pages, and 3) produces accumulative stats for display in the stats tables.

Sometimes, connectorland tells ilp-node to add a peer, to announce a testLedger, to get a remote quote, and to send a payment.

Actions like fulfilling a payment if ilp-node has the preimage, don't require interaction, they are handled inside the ilp-node instance.