Read/Write lock timeout when performing lots of API calls

[bitphage]

Tried to switch one of my applications to golos 0.17.0 and I immediately got node stuck. Later I will try to provide a testcase. Originally discovered by vik, so I'm confirming the issue.

744044ms            p2p_plugin.cpp:143            handle_block         ] Got 9 transactions on block 15180724 by primus -- latency: 44 ms
747383ms            p2p_plugin.cpp:143            handle_block         ] Got 9 transactions on block 15180725 by anyx -- latency: 383 ms
750040ms            p2p_plugin.cpp:143            handle_block         ] Got 15 transactions on block 15180726 by steepshot -- latency: 40 ms
Write lock timeoutRead lock timeout

Read lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Read lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Read lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout
Write lock timeout

[Vik-BC]

https://youtu.be/8y3MGcqsdMc

seed-node = 5.9.18.213:4243
seed-node = 52.32.75.69:4243
seed-node = 52.57.156.202:4243
seed-node = 88.99.13.48:4243
seed-node = golos-seed.arcange.eu:4243
seed-node = golos-seed.esteem.ws:4243
seed-node = golosnode.com:4243
seed-node = 138.68.101.115:4243
seed-node = golos.imcoins.org:2001
seed-node = 178.62.224.148:4242
webserver-thread-pool-size = 10
webserver-http-endpoint = 127.0.0.1:9090
webserver-ws-endpoint = 127.0.0.1:9091
rpc-endpoint = 0.0.0.0:2001
shared-file-dir = 'blockhain'
shared-file-size = 100G
plugin = chain p2p json_rpc webserver network_broadcast_api witness test_api database_api private_message follow social_network market_history account_by_key account_history chain_stats block_info raw_block
bcd-trigger = [[0,10],[85,300]]
follow-max-feed-size = 500
bucket-size = [15,60,300,3600,86400]
history-per-size = 5760
enable-stale-production = false
required-participation = 0

[log.console_appender.stderr]
stream=std_error
[log.file_appender.p2p]
filename=logs/p2p/p2p.log
[logger.default]

[afalaleev]

Added config-parameters for read/write locks:

# maximum microseconds for trying to get read lock
read-wait-micro = 1000000

# maximum number of retries to get read lock
#  may be small, because it is an request to get data
max-read-wait-retries = 5

# maximum microseconds for trying to get write lock
write-wait-micro = 1000000

# maximum number of retries to get write lock
max-write-wait-retries = 10000

[afalaleev]

Added strong and weak strategies for write lock.

For example, you have a high load node, which is used to get data from the blockchain. Such nodes should in any case accept signed blocks, but it can skip transactions for broadcasting, because they may come to the node in a signed block.

An example of configuration for a heavy load node:

read-wait-micro = 10000
max-read-wait-retries = 5

write-wait-micro = 1000000
max-write-wait-retries = 2

[afalaleev]

Added option 'single_write_thread' - all write requests (push_block and push_transaction) are added to queue, which is handled by a single thread.

An example of configuration for a heavy load node:

read-wait-micro = 10000
max-read-wait-retries = 5

write-wait-micro = 1000000 # it may be less, because this timeout only on accepting transaction without signed block
max-write-wait-retries = 2

single-write-thread = true

[afalaleev]

The following settings shows a good result:

read-wait-micro = 10000
max-read-wait-retries = 500

write-wait-micro = 10000
max-write-wait-retries = 200

single-write-thread = true

webserver-thread-pool-size = <number of CPU> - 1

[afalaleev]

The following settings shows better result:

read-wait-micro = 500000
max-read-wait-retries = 2

write-wait-micro = 500000
max-write-wait-retries = 3

single-write-thread = true

webserver-thread-pool-size = <number of CPU> - 1

[kotbegemot]

https://github.com/GolosChain/golos/pull/500

[kotbegemot]

depends https://github.com/GolosChain/golos/issues/501