Node Chokes Intermittently

[adamkrellenstein]

[adamkrellenstein]

This is causing issues...

[adamkrellenstein]

Able to reproduce with Locust tests against dev.counterparty.io:4000. When the latency spikes there are no logs for slow requests (at DEBUG) on the node and there are just fewer logs being captured! hm....

https://console.cloud.google.com/logs/query;query=resource.type%3D%22gce_instance%22%0Aresource.labels.instance_id%3D%228878289658435634410%22;cursorTimestamp=2024-11-15T22:42:08.663543005Z;startTime=2024-11-15T22:38:20.000Z;endTime=2024-11-15T22:44:20.000Z?inv=1&invt=AbhlpA&project=noted-extension-407918

[adamkrellenstein]

Seems like this is probably insufficient CPU, esp. with the asset conservation check...

[adamkrellenstein]

https://console.cloud.google.com/logs/query;query=resource.labels.instance_id%3D%228878289658435634410%22;cursorTimestamp=2024-11-18T01:25:08.948325060Z;startTime=2024-11-18T00:59:00.000Z;endTime=2024-11-18T01:26:00.000Z?project=noted-extension-407918&inv=1&invt=AbhxUQ

[adamkrellenstein]

Three strategies for dealing with this:

1. profile sqlite3
2. add / update tracing for API watcher procesS
3. record queries that trigger the problem for playback

[adamkrellenstein]

Also seeing this with the mempool and asset conservation check disabled

[adamkrellenstein]

there could be lots of obvious errors in the Gunicorn logs that we're not seeing https://github.com/CounterpartyXCP/counterparty-core/issues/2515

[adamkrellenstein]

Weirdly enough, the load test on commit 657d5c09 causes the choking but on develop does not.