Block intervals increase over time, even with as few as 10 validators

[DrZoltanFazekas]

In the devnet with only 8 validators producing empty blocks, the average block time was 1.2s at the beginning, but by the time the network reached 30k blocks, it increased noticeably:

  view height timestmp                                   proposer      gas txns highqc
--------------------------------------------------------------------------------------
 29843  29362 23:32:53 0xb2d01a549f84d04346f4bb454ef6ae2553129526        0    0  29842 
 29844  29363 23:32:55 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  29843 2s since last block
 29845  29364 23:32:58 0xa598a7946e88196f9aa6c05837a554d2318e4c12        0    0  29844 3s since last block
 29846  29365 23:33:00 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  29845 2s since last block
 29847  29366 23:33:03 0x35293af1338a53ee22ef3a784e6f42e4d05702b5        0    0  29846 3s since last block
 29848  29367 23:33:05 0x5e579ddd88a40c0738453d284c25adc0a5178cbc        0    0  29847 2s since last block
 29849  29368 23:33:06 0xa598a7946e88196f9aa6c05837a554d2318e4c12        0    0  29848 
 29850  29369 23:33:08 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  29849 2s since last block
 29851  29370 23:33:12 0x35293af1338a53ee22ef3a784e6f42e4d05702b5        0    0  29850 4s since last block
 29852  29371 23:33:14 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  29851 2s since last block

For comparison, the prototestnet with 11 validators has produced over 3.5 million blocks yet maintained an average block time of 1.2s

Since individual block times are 5s or even longer, the devnet has a lot of missed views (timeouts).

It's unclear whether the block times are increasing over time i.e. with the number of blocks produced by the network.

[DrZoltanFazekas]

A few more examples. This sample illustrates 2.5s average block time:

  view height timestmp                                   proposer      gas txns highqc
--------------------------------------------------------------------------------------
 53331  51581 10:21:07 0xb2d01a549f84d04346f4bb454ef6ae2553129526        0    0  53330 
 53332  51582 10:21:10 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53331 3s since last block
 53333  51583 10:21:12 0x64d6698287492627cf5e000ce89ee9eed1160ad5        0    0  53332 2s since last block
 53334  51584 10:21:16 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53333 4s since last block
 53335  51585 10:21:20 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53334 4s since last block
 53336  51586 10:21:23 0xb2d01a549f84d04346f4bb454ef6ae2553129526        0    0  53335 3s since last block
 53337  51587 10:21:24 0xa598a7946e88196f9aa6c05837a554d2318e4c12        0    0  53336 
 53338  51588 10:21:27 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53337 3s since last block
 53339  51589 10:21:30 0x64d6698287492627cf5e000ce89ee9eed1160ad5        0    0  53338 3s since last block
 53340  51590 10:21:33 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53339 3s since last block

2.1s average here:

  view height timestmp                                   proposer      gas txns highqc
--------------------------------------------------------------------------------------
 53492  51736 10:26:08 0xb2d01a549f84d04346f4bb454ef6ae2553129526        0    0  53491 
 53493  51737 10:26:11 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53492 3s since last block
 53494  51738 10:26:13 0x35293af1338a53ee22ef3a784e6f42e4d05702b5        0    0  53493 2s since last block
 53495  51739 10:26:16 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53494 3s since last block
 53496  51740 10:26:19 0xa598a7946e88196f9aa6c05837a554d2318e4c12        0    0  53495 3s since last block
 53497  51741 10:26:20 0xa598a7946e88196f9aa6c05837a554d2318e4c12        0    0  53496 
 53498  51742 10:26:23 0x977a797a0546cdb1082c2368921af455dfdc7a5a        0    0  53497 3s since last block
 53499  51743 10:26:25 0xe1af288953c3314c00c4130c05c60b8a78d0e779        0    0  53498 2s since last block
 53500  51744 10:26:26 0x5e579ddd88a40c0738453d284c25adc0a5178cbc        0    0  53499 
 53501  51745 10:26:28 0x5e579ddd88a40c0738453d284c25adc0a5178cbc        0    0  53500 2s since last block
 53502  51746 10:26:29 0xb2d01a549f84d04346f4bb454ef6ae2553129526        0    0  53501

[shawn-zil]

This is still observed, even after #1813 fix i.e. the 45 validator perftest network block-times have gone from 1.2s (<1k blocks) to 1.3s (>48k blocks).

Feels like a DB issue, as the DB grows with time. I'll take a quick look into it, as it seems like a 'small-ish' issue.