Figure out mempool bug that we often find in CI

[mariari]

I often find the following issue in CI.

I can sometimes get this to happen in my REPL when it's in a weird state, I'm not sure how to reproduce this

  1) test Transaction print (AnomaTest.Node.Mempool)
Error:      test/node/mempool_test.exs:90
     ** (exit) exited in: GenServer.call(:mempool_mempool_com, {:tx, [[1, 666 | 0], 0, [[8, [1, 0, 0, 0, 0, 0, 0 | 0], [1, 6, [5, [1 | 1], 8, [9, 1406, 0 | 127], 9, 2, 10, [6, 0 | 58], 0 | 2], [6, [5, [1 | 1], 8, [9, 1406, 0 | 127], 9, 2, 10, [6, 0 | 118], 0 | 2], [6, [5, [1 | 1], 8, [9, 1406, 0 | 127], 9, 2, 10, [6, 0 | 478], 0 | 2], [6, [5, [1 | 0], 0 | 222], [0 | 0], 6, [0 | 1778], [1 | 0], 1 | 1], 1 | 1], 1 | 1], 1 | 1], 0 | 1], 8, [1, 0, 0, 0, 0, 0, 0 | 0], [1, 5, [1 | 0], 0 | 222], 0 | 1], [[1 | 0], [8, [1, 0, 0, 0, 0, 0, 0 | 0], [1, 8, [[8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 8, [1 | 0], [1, 8, [6, [6, [3, 0 | 6], [1 | 1], 1 | 0], [0 | 6], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 28], 0 | 2], [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 9, 181, 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 58], 0 | 2], [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 9, 44, 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 118], 0 | 2], [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 9, 180, 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 238], 0 | 2], [8, [7, [0 | 7], 9, 46, 0 | 1], 9, 2, 10, [6, 0 | 478], 0 | 2], [6, [6, [3, 0 | 446], [1 | 1], 1 | 0], [0 | 446], 0 | 0], 6, [5, [1 | 0], 0 | 447], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [[8, [1 | 0], [1, 8, [6, [6, [3, 0 | 6], [1 | 1], 1 | 0], [0 | 6], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [[8, [1, 0, 0, 0, 0, 0, 0, 0 | 0], [1, 8, [8, [7, [0 | 7], 9, 91, 0 | 1], 9, 2, 10, [6, 0 | 14], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1 | 0], [1, 8, [6, [6, [3, 0 | 6], [1 | 1], 1 | 0], [0 | 6], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1, 0, 0, 0, 0, 0, 0, 0 | 0], [1, 8, [[8, [7, [0 | 7], 9, 47, 0 | 1], 9, 2, 10, [6, 0 | 28], 0 | 2], [6, [6, [3, 0 | 26], [1 | 1], 1 | 0], [0 | 26], 0 | 0], [6, [6, [3, 0 | 54], [1 | 1], 1 | 0], [0 | 54], 0 | 0], [6, [6, [3, 0 | 110], [1 | 1], 1 | 0], [0 | 110], 0 | 0], [6, [5, [1 | 0], 0 | 222], [1 | 0], 6, [5, [1 | 1], 0 | 222], [1 | 1], 0 | 0], [6, [6, [3, 0 | 446], [1 | 1], 1 | 0], [0 | 446], 0 | 0], [6, [6, [3, 0 | 894], [1 | 1], 1 | 0], [0 | 894], 0 | 0], 6, [6, [3, 0 | 895], [1 | 1], 1 | 0], [0 | 895], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [8, [1 | 0], [1, 8, [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 8, [1, 0, 0 | 0], [1, 8, [[6, [6, [3, 0 | 12], [1 | 1], 1 | 0], [0 | 12], 0 | 0], [6, [5, [1 | 0], 0 | 26], [1 | 0], 6, [5, [1 | 1], 0 | 26], [1 | 1], 0 | 0], 6, [6, [3, 0 | 27], [1 | 1], 1 | 0], [0 | 27], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 14], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1 | 0], [1, 8, [7, [[7, [0 | 7], 7, [9, 10, 0 | 1], 9, 2, 0 | 1], 1 | 0], 8, [0 | 2], [1, 0 | 15], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [[8, [1 | 0], [1, 6, [5, [1 | 0], 0 | 6], [1 | 1], 8, [9, 4, 0 | 63], 9, 2, 10, [6, [7, [0 | 3], 1 | 2], 7, [0 | 3], 9, 2, 10, [6, 8, [9, 342, 0 | 63], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 1], 0 | 2], 0 | 1], [8, [1 | 0], [1, [8, [1, 0 | 0], [1, 8, [9, 20, 0 | 255], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 90, 0 | 7], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 190, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 0 | 29], 0 | 2], 0 | 28], 0 | 2], 0 | 1], [8, [1 | 0], [1, 8, [9, 367, 0 | 7], 9, 2, 10, [6, [7, [0 | 3], 1 | 1], 0 | 14], 0 | 2], 0 | 1], [[8, [1, 0 | 0], [1, 8, [1 | 0], 7, [10, [29, 8, [9, 10, 0 | 15], 9, 2, 10, [6, 0 | 61], 0 | 2], 0 | 1], 8, [0 | 29], 8, [1, 6, [5, [0 | 14], 0 | 124], [8, [9, 767, 0 | 63], 9, 2, 10, [6, [7, [0 | 3], 1 | 1], 0 | 14], 0 | 2], 9, 2, 10, [14, 4, 0 | 14], 10, [6, 8, [9, 20, 0 | 2047], 9, 2, 10, [6, [0 | 253], 7, [...], ...], 0 | 2], 0 | 1], 9, 2, 0 | 1], 0 | 1], [8, [1, 0 | 0], [1, 8, [9, 4, 0 | 255], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 255], 9, 2, 10, [6, [7, ...], 0 | 28], 0 | 2], 0 | 2], 0 | 2], 0 | 1], [8, [9, 4, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], [8, [1 | 0], [1, 8, [9, 47, 0 | 255], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 342, 0 | 255], 9, 2, 10, [6, 7, [0 | 3], 9, 382, 0 | 7], 0 | 2], 7, [0 | 3], 8, [9, 10, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 2], 0 | 1], 8, [1, 0 | 0], [1, 8, [9, 46, 0 | 255], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 255], 9, 2, 10, [6, ...], 0 | 2], 0 | 2], 0 | 2], 0 | 1], [8, [1, 0 | 0], [1, 8, [9, 10, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 20, 0 | 255], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1, [0 | 0] | 0], [1, 8, [9, 367, 0 | 7], 9, 2, 10, [6, [0 | 57], 7, [0 | 3], 8, [9, 767, 0 | 7], 9, 2, 10, [6, [0 | 56], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1 | 0], [1, 8, [1 | 0], 8, [1, 6, [5, [1 | 0], 0 | 30], [0 | 6], 9, 2, 10, [30, 8, [9, 767, 0 | 31], 9, 2, 10, [6, [7, [0 | 3], 1 | 1], 0 | 62], 0 | 2], 10, [6, 4, 0 | 6], 0 | 1], 9, 2, 0 | 1], 0 | 1], [8, [9, 4, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 2], [8, [1 | 0], [1, 8, [9, 46, 0 | 7], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 366, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 7, [0 | 3], 1 | 1], 0 | 2], 0 | 1], 8, [1, 0 | 0], [1, 8, [9, 170, 0 | 255], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 255], 9, 2, 10, [...], ...], 0 | 2], 0 | 2], 0 | 1], 0 | 1], 8, [1 | 0], [1, 8, [8, [9, 10, 0 | 7], 9, 190, 10, [6, 7, [0 | 3], 1 | 0], 0 | 2], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 1], [[7, [8, [1, 0 | 0], [1, 6, [5, [1 | 0], 0 | 13], [1 | 1], 8, [9, 4, 0 | 31], 9, 2, 10, [6, [0 | 28], 7, [0 | 3], 9, 2, 10, [13, 8, [9, 342, 0 | 31], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 1], 0 | 2], 0 | 1], 11, [1953718630, 1, 7827312, [0 | 7] | 0], 0 | 1], 8, [1 | 0], [1, [8, [1, 1 | 1], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 127], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [[8, [1, 0 | 0], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 20, 0 | 127], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1 | 0], [1, 8, [9, 46, 0 | 127], 9, 2, 10, [6, [0 | 14], 0 | 62], 0 | 2], 0 | 1], [8, [1, 1 | 1], [1, 8, [9, 4, 0 | 7], 9, 2, 10, [6, [0 | 28], 7, [0 | 3], 8, [9, 174, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1 | 0], [1, 0 | 0], 0 | 1], 8, [1 | 0], [1, 8, [9, 47, 0 | 127], 9, 2, 10, [6, [0 | 62], 7, [0 | 3], 8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 2], 0 | 1], [8, [1, 0 | 0], [1, 5, [8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 1], [8, [1, 0 | 0], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], 8, [1, 0 | 0], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 47, 0 | 127], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 20, 0 | 127], 9, 2, 10, [6, [0 | 62], 0 | 28], 0 | 2], 7, [0 | 3], 8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 2], 0 | 2], 0 | 1], 0 | 1], [[8, [1, [[0 | 15], [0 | 0], [0 | 0] | 0] | 0], [1, 8, [0 | 101], [1, 8, [0 | 60], 9, 2, 10, [6, [0 | 125], 0 | 14], 0 | 2], 0 | 1], 0 | 1], [[8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [7, [0 | 7], [1, 7, [0 | 14], 9, 2, 0 | 1], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [6, [3, 0 | 6], [[6, [5, [1 | 0], 0 | 12], [1 | 0], 0 | 0], 8, [0 | 30], 9, 2, 10, [6, 0 | 29], 0 | 2], 6, [5, [1 | 0], 0 | 6], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [[0 | 26], 7, [8, [9, 47, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 6], [1, 8, [[8, [0 | 30], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [7, [0 | 7], 8, [9, 47, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 29], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [1 | 0], [1, 8, [7, [1, 0 | 0], 8, [0 | 2], [1, 0 | 15], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [8, [[8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [[1 | 0], 0 | 54], [1, 8, [6, [5, [1 | 1], 0 | 12], [[6, [5, [0 | 12], 1 | 1], [1 | 1], 0 | 0], 8, [0 | 60], 9, 2, 10, [6, 0 | 29], 0 | 2], [6, [5, [0 | 12], 1 | 0], [1 | 0], 0 | 0], 8, [0 | 61], 9, 2, 10, [6, 0 | 29], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [6, [3, 0 | 6], [[8, [0 | 30], 9, 2, 10, [6, 0 | 28], 0 | 2], [8, [7, [0 | 7], 8, [9, 702, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 58], 0 | 2], 8, [7, [0 | 7], 8, [9, 702, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 59], 0 | 2], 6, [5, [1 | 0], 0 | 6], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [1 | 0], [1, 6, [5, [1 | 0], 0 | 6], [1 | 0], 4, 9, 2, 10, [6, 0 | 13], 0 | 1], 0 | 1], 8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [8, [1, 0, [0 | 15], 0, 0 | 0], [1, 8, [1, 6, [5, [1 | 0], 0 | 28], [1 | 0], [8, [0 | 29], 9, 2, 10, [6, 0 | 120], 0 | 2], 9, 2, 10, [28, 0 | 57], 0 | 1], 9, 2, 0 | 1], 0 | 1], [8, [1, 0, [0 | 13], [0 | 0] | 0], [1, 8, [1, 6, [5, [1 | 0], 0 | 28], [0 | 237], 8, [0 | 29], 9, 2, 10, [6, [0 | 120], 7, [0 | 3], 9, 2, 10, [28, 0 | 57], 0 | 1], 0 | 2], 9, 2, 0 | 1], 0 | 1], 8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [6, [3, 0 | 6], [[8, [0 | 30], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [7, [0 | 7], 8, [9, 47, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 29], 0 | 2], 6, [5, [1 | 0], 0 | 6], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [[7, [8, [1, 1 | 1], [1, 8, [1 | 0], 8, [1, 6, [5, [1 | 0], 0 | 60], [0 | 6], 9, 2, 10, [60, 8, [9, 342, 0 | 31], 9, 2, 10, [6, 0 | 124], 0 | 2], 10, [6, 8, [9, 20, 0 | 31], 9, 2, 10, [6, [0 | 125], 0 | 14], 0 | 2], 0 | 1], 9, 2, 0 | 1], 0 | 1], 11, [1953718630, 1, 7107949, [0 | 7] | 0], 0 | 1], [[7, [8, [1, 0 | 0], [1, 6, [5, [1 | 0], 0 | 12], [0 | 13], 9, 2, 10, [6, [8, [9, 342, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 4, 0 | 13], 0 | 1], 0 | 1], 11, [1953718630, 1, 6579297, [0 | 7] | 0], 0 | 1], [[7, [8, [1, 0 | 0], [1, 6, [5, [0 | 12], 0 | 13], [1 | 0], 6, [8, [9, 343, 0 | 7], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], [1 | 0], 1 | 1], 0 | 1], 11, [1953718630, 1, 6648940, [0 | 7] | 0], 0 | 1], [7, [8, [1, 1 | 1], [1, 6, [5, [1 | 0], 0 | 13], [0 | 0], 8, [1 | 0], 8, [1, 6, [8, [9, 343, 0 | 31], 9, 2, 10, [6, [0 | 124], 0 | 125], 0 | 2], [0 | 6], 9, 2, 10, [60, 8, [9, 47, 0 | 31], 9, 2, 10, [6, [0 | 124], 0 | 125], 0 | 2], 10, [6, 4, 0 | 6], 0 | 1], 9, 2, 0 | 1], 0 | 1], 11, [1953718630, 1, 7760228, [0 | 7] | 0], 0 | 1], [7, [8, [1 | 0], [1, 6, [5, [1 | 0], 0 | 6], [0 | 0], 8, [1 | 0], 8, [1, 6, [5, [0 | 30], 4, 0 | 6], [0 | 6], 9, 2, 10, [6, 4, 0 | 6], 0 | 1], 9, 2, 0 | 1], 0 | 1], 11, [1953718630, 1, 6514020, [0 | 7] | 0], 0 | 1], 7, [8, [1, 0 | 0], [1, 6, [6, [5, [0 | 12], 0 | 13], [1 | 1], 1 | 0], [6, [8, [1, 6, [5, [1 | 0], 0 | 28], [1 | 0], 6, [6, [6, [5, [1 | 0], 0 | 29], [1 | 1], 1 | 0], [6, [9, 2, 10, [14, [8, [9, ...], 9, ...], 8, [9, ...], 9, ...], 0 | 1], [1 | 0], 1 | 1], 1 | 1], [1 | 0], 1 | 1], 9, 2, 0 | 1], [1 | 0], 1 | 1], 1 | 1], 0 | 1], 11, [1953718630, 1, 6845548, [0 | 7] | 0], 0 | 1], 7, [8, [1, 0 | 0], [1, 6, [8, [9, 84, 0 | 7], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], [1 | 1], 1 | 0], 0 | 1], 11, [1953718630, 1, 6845543, [0 | 7] | 0], 0 | 1], [7, [8, [1, 0 | 0], [1, 6, [8, [9, 343, 0 | 7], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], [1 | 1], 1 | 0], 0 | 1], 11, [1953718630, 1, 6648935, [0 | 7] | 0], 0 | 1], [7, [8, [1, 1 | 1], [1, 6, [5, [1 | 0], 0 | 13], [0 | 0], 8, [9, 47, 0 | 7], 9, 2, 10, [6, [0 | 28], 7, [0 | 3], 8, [9, 4, 0 | 7], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 170, 0 | 7], 9, 2, 10, [6, ...], 0 | 2], 0 | 2], 0 | 2], 0 | 1], 11, [1953718630, 1, 6582125, [0 | 7] | 0], 0 | 1], 7, [8, [1, 0 | 0], [1, 6, [5, [1 | 0], 0 | 13], [0 | 12], 9, 2, 10, [6, [8, [9, 342, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [9, 342, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 1], 0 | 1], 11, [1953718630, 1, 6452595, [0 | 7] | 0], 0 | 1], [0 | 3] | 909]}, 10000)
         ** (EXIT) shutdown
     code: zero_tx = Mcom.tx(node.mempool, zero_counter(key))
     stacktrace:
       (elixir 1.15.6) lib/gen_server.ex:1074: GenServer.call/3
       test/node/mempool_test.exs:95: (test)

[agureev]

I would add that the Pinger sometimes also bugs out yet less frequently.

     test/node/pinger_test.exs:32
     Assertion failed, no matching message after 100ms
     The following variables were pinned:
       pid_zero = #PID<0.1508.0>
     The process mailbox is empty.
     code: assert_receive {:"$gen_cast", {:process_done, ^pid_zero}}
     stacktrace:
       test/node/pinger_test.exs:40: (test)

Adding here since this seems connected to the broadcast functionality as well.

[mariari]

Interesting, I can look at this then

[mariari]

I can confirm the following:

It dies much faster with the broadcast but still dies without it. I'm running the modified test

  test "death", %{node: node} do
    zero_tx = Mcom.tx(node.mempool, zero_counter(666))
  end

And I get the following output

iex(mariari@Gensokyo)91> Enum.each(0..99, fn(x) -> {IO.inspect("number:#{inspect(x)}"), AnomaTest.Node.Mempool."test death"(%{node: node})} end)
"number:0"
"number:1"
"number:2"
"number:3"
"number:4"
"number:5"
"number:6"
"number:7"
"number:8"
"number:9"
"number:10"
"number:11"
"number:12"
"number:13"
"number:14"
"number:15"
"number:16"
"number:17"
"number:18"
"number:19"
"number:20"
"number:21"
"number:22"
"number:23"
"number:24"
"number:25"
"number:26"
"number:27"
"number:28"
"number:29"
"number:30"
"number:31"
"number:32"
"number:33"
"number:34"
"number:35"
"number:36"
"number:37"
"number:38"
"number:39"
"number:40"
"number:41"
"number:42"
"number:43"
"number:44"
"number:45"
"number:46"
"number:47"
"number:48"
"number:49"
"number:50"
"number:51"
"number:52"
"number:53"
"number:54"
"number:55"
"number:56"
"number:57"
"number:58"
"number:59"
"number:60"
"number:61"
** (exit) exited in: GenServer.call(:mempool_mempool, {:tx, [[1, 666 | 0], 0, [[8, [1, 0, 0, 0, 0, 0, 0 | 0], [1, 6, [5, [1 | 1], 8, [9, 1406, 0 | 127], 9, 2, 10, [6, 0 | 58], 0 | 2], [6, [5, [1 | 1], 8, [9, 1406, 0 | 127], 9, 2, 10, [6, 0 | 118], 0 | 2], [6, [5, [1 | 1], 8, [9, 1406, 0 | 127], 9, 2, 10, [6, 0 | 478], 0 | 2], [6, [5, [1 | 0], 0 | 222], [0 | 0], 6, [0 | 1778], [1 | 0], 1 | 1], 1 | 1], 1 | 1], 1 | 1], 0 | 1], 8, [1, 0, 0, 0, 0, 0, 0 | 0], [1, 5, [1 | 0], 0 | 222], 0 | 1], [[1 | 0], [8, [1, 0, 0, 0, 0, 0, 0 | 0], [1, 8, [[8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 8, [1 | 0], [1, 8, [6, [6, [3, 0 | 6], [1 | 1], 1 | 0], [0 | 6], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 28], 0 | 2], [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 9, 181, 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 58], 0 | 2], [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 9, 44, 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 118], 0 | 2], [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 9, 180, 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 238], 0 | 2], [8, [7, [0 | 7], 9, 46, 0 | 1], 9, 2, 10, [6, 0 | 478], 0 | 2], [6, [6, [3, 0 | 446], [1 | 1], 1 | 0], [0 | 446], 0 | 0], 6, [5, [1 | 0], 0 | 447], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [[8, [1 | 0], [1, 8, [6, [6, [3, 0 | 6], [1 | 1], 1 | 0], [0 | 6], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [[8, [1, 0, 0, 0, 0, 0, 0, 0 | 0], [1, 8, [8, [7, [0 | 7], 9, 91, 0 | 1], 9, 2, 10, [6, 0 | 14], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1 | 0], [1, 8, [6, [6, [3, 0 | 6], [1 | 1], 1 | 0], [0 | 6], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1, 0, 0, 0, 0, 0, 0, 0 | 0], [1, 8, [[8, [7, [0 | 7], 9, 47, 0 | 1], 9, 2, 10, [6, 0 | 28], 0 | 2], [6, [6, [3, 0 | 26], [1 | 1], 1 | 0], [0 | 26], 0 | 0], [6, [6, [3, 0 | 54], [1 | 1], 1 | 0], [0 | 54], 0 | 0], [6, [6, [3, 0 | 110], [1 | 1], 1 | 0], [0 | 110], 0 | 0], [6, [5, [1 | 0], 0 | 222], [1 | 0], 6, [5, [1 | 1], 0 | 222], [1 | 1], 0 | 0], [6, [6, [3, 0 | 446], [1 | 1], 1 | 0], [0 | 446], 0 | 0], [6, [6, [3, 0 | 894], [1 | 1], 1 | 0], [0 | 894], 0 | 0], 6, [6, [3, 0 | 895], [1 | 1], 1 | 0], [0 | 895], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [8, [1 | 0], [1, 8, [8, [7, [0 | 7], 8, [9, 47, 0 | 15], 9, 2, 10, [6, 7, [0 | 3], 8, [1, 0, 0 | 0], [1, 8, [[6, [6, [3, 0 | 12], [1 | 1], 1 | 0], [0 | 12], 0 | 0], [6, [5, [1 | 0], 0 | 26], [1 | 0], 6, [5, [1 | 1], 0 | 26], [1 | 1], 0 | 0], 6, [6, [3, 0 | 27], [1 | 1], 1 | 0], [0 | 27], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 2], 9, 2, 10, [6, 0 | 14], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1 | 0], [1, 8, [7, [[7, [0 | 7], 7, [9, 10, 0 | 1], 9, 2, 0 | 1], 1 | 0], 8, [0 | 2], [1, 0 | 15], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [[8, [1 | 0], [1, 6, [5, [1 | 0], 0 | 6], [1 | 1], 8, [9, 4, 0 | 63], 9, 2, 10, [6, [7, [0 | 3], 1 | 2], 7, [0 | 3], 9, 2, 10, [6, 8, [9, 342, 0 | 63], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 1], 0 | 2], 0 | 1], [8, [1 | 0], [1, [8, [1, 0 | 0], [1, 8, [9, 20, 0 | 255], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 90, 0 | 7], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 190, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 0 | 29], 0 | 2], 0 | 28], 0 | 2], 0 | 1], [8, [1 | 0], [1, 8, [9, 367, 0 | 7], 9, 2, 10, [6, [7, [0 | 3], 1 | 1], 0 | 14], 0 | 2], 0 | 1], [[8, [1, 0 | 0], [1, 8, [1 | 0], 7, [10, [29, 8, [9, 10, 0 | 15], 9, 2, 10, [6, 0 | 61], 0 | 2], 0 | 1], 8, [0 | 29], 8, [1, 6, [5, [0 | 14], 0 | 124], [8, [9, 767, 0 | 63], 9, 2, 10, [6, [7, [0 | 3], 1 | 1], 0 | 14], 0 | 2], 9, 2, 10, [14, 4, 0 | 14], 10, [6, 8, [9, 20, 0 | 2047], 9, 2, 10, [6, [0 | 253], 7, [...], ...], 0 | 2], 0 | 1], 9, 2, 0 | 1], 0 | 1], [8, [1, 0 | 0], [1, 8, [9, 4, 0 | 255], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 255], 9, 2, 10, [6, [7, ...], 0 | 28], 0 | 2], 0 | 2], 0 | 2], 0 | 1], [8, [9, 4, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], [8, [1 | 0], [1, 8, [9, 47, 0 | 255], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 342, 0 | 255], 9, 2, 10, [6, 7, [0 | 3], 9, 382, 0 | 7], 0 | 2], 7, [0 | 3], 8, [9, 10, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 2], 0 | 1], 8, [1, 0 | 0], [1, 8, [9, 46, 0 | 255], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 255], 9, 2, 10, [6, ...], 0 | 2], 0 | 2], 0 | 2], 0 | 1], [8, [1, 0 | 0], [1, 8, [9, 10, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 20, 0 | 255], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1, [0 | 0] | 0], [1, 8, [9, 367, 0 | 7], 9, 2, 10, [6, [0 | 57], 7, [0 | 3], 8, [9, 767, 0 | 7], 9, 2, 10, [6, [0 | 56], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1 | 0], [1, 8, [1 | 0], 8, [1, 6, [5, [1 | 0], 0 | 30], [0 | 6], 9, 2, 10, [30, 8, [9, 767, 0 | 31], 9, 2, 10, [6, [7, [0 | 3], 1 | 1], 0 | 62], 0 | 2], 10, [6, 4, 0 | 6], 0 | 1], 9, 2, 0 | 1], 0 | 1], [8, [9, 4, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 2], [8, [1 | 0], [1, 8, [9, 46, 0 | 7], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 366, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 7, [0 | 3], 1 | 1], 0 | 2], 0 | 1], 8, [1, 0 | 0], [1, 8, [9, 170, 0 | 255], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 255], 9, 2, 10, [...], ...], 0 | 2], 0 | 2], 0 | 1], 0 | 1], 8, [1 | 0], [1, 8, [8, [9, 10, 0 | 7], 9, 190, 10, [6, 7, [0 | 3], 1 | 0], 0 | 2], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 1], [[7, [8, [1, 0 | 0], [1, 6, [5, [1 | 0], 0 | 13], [1 | 1], 8, [9, 4, 0 | 31], 9, 2, 10, [6, [0 | 28], 7, [0 | 3], 9, 2, 10, [13, 8, [9, 342, 0 | 31], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 1], 0 | 2], 0 | 1], 11, [1953718630, 1, 7827312, [0 | 7] | 0], 0 | 1], 8, [1 | 0], [1, [8, [1, 1 | 1], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 127], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [[8, [1, 0 | 0], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 20, 0 | 127], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1 | 0], [1, 8, [9, 46, 0 | 127], 9, 2, 10, [6, [0 | 14], 0 | 62], 0 | 2], 0 | 1], [8, [1, 1 | 1], [1, 8, [9, 4, 0 | 7], 9, 2, 10, [6, [0 | 28], 7, [0 | 3], 8, [9, 174, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 2], 0 | 1], [8, [1 | 0], [1, 0 | 0], 0 | 1], 8, [1 | 0], [1, 8, [9, 47, 0 | 127], 9, 2, 10, [6, [0 | 62], 7, [0 | 3], 8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 2], 0 | 1], [8, [1, 0 | 0], [1, 5, [8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 1], [8, [1, 0 | 0], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 4, 0 | 31], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], 0 | 2], 0 | 1], 8, [1, 0 | 0], [1, 8, [9, 42, 0 | 7], 9, 2, 10, [6, 7, [0 | 3], 8, [9, 47, 0 | 127], 9, 2, 10, [6, [7, [0 | 3], 8, [9, 20, 0 | 127], 9, 2, 10, [6, [0 | 62], 0 | 28], 0 | 2], 7, [0 | 3], 8, [9, 42, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 2], 0 | 2], 0 | 1], 0 | 1], [[8, [1, [[0 | 15], [0 | 0], [0 | 0] | 0] | 0], [1, 8, [0 | 101], [1, 8, [0 | 60], 9, 2, 10, [6, [0 | 125], 0 | 14], 0 | 2], 0 | 1], 0 | 1], [[8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [7, [0 | 7], [1, 7, [0 | 14], 9, 2, 0 | 1], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [6, [3, 0 | 6], [[6, [5, [1 | 0], 0 | 12], [1 | 0], 0 | 0], 8, [0 | 30], 9, 2, 10, [6, 0 | 29], 0 | 2], 6, [5, [1 | 0], 0 | 6], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [[0 | 26], 7, [8, [9, 47, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 0 | 6], [1, 8, [[8, [0 | 30], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [7, [0 | 7], 8, [9, 47, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 29], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [1 | 0], [1, 8, [7, [1, 0 | 0], 8, [0 | 2], [1, 0 | 15], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [8, [[8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [[1 | 0], 0 | 54], [1, 8, [6, [5, [1 | 1], 0 | 12], [[6, [5, [0 | 12], 1 | 1], [1 | 1], 0 | 0], 8, [0 | 60], 9, 2, 10, [6, 0 | 29], 0 | 2], [6, [5, [0 | 12], 1 | 0], [1 | 0], 0 | 0], 8, [0 | 61], 9, 2, 10, [6, 0 | 29], 0 | 2], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [6, [3, 0 | 6], [[8, [0 | 30], 9, 2, 10, [6, 0 | 28], 0 | 2], [8, [7, [0 | 7], 8, [9, 702, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 58], 0 | 2], 8, [7, [0 | 7], 8, [9, 702, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 59], 0 | 2], 6, [5, [1 | 0], 0 | 6], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [8, [1 | 0], [1, 6, [5, [1 | 0], 0 | 6], [1 | 0], 4, 9, 2, 10, [6, 0 | 13], 0 | 1], 0 | 1], 8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [8, [1, 0, [0 | 15], 0, 0 | 0], [1, 8, [1, 6, [5, [1 | 0], 0 | 28], [1 | 0], [8, [0 | 29], 9, 2, 10, [6, 0 | 120], 0 | 2], 9, 2, 10, [28, 0 | 57], 0 | 1], 9, 2, 0 | 1], 0 | 1], [8, [1, 0, [0 | 13], [0 | 0] | 0], [1, 8, [1, 6, [5, [1 | 0], 0 | 28], [0 | 237], 8, [0 | 29], 9, 2, 10, [6, [0 | 120], 7, [0 | 3], 9, 2, 10, [28, 0 | 57], 0 | 1], 0 | 2], 9, 2, 0 | 1], 0 | 1], 8, [8, [1 | 0], [1, 8, [0 | 6], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], [1, 8, [1 | 0], [1, 8, [6, [3, 0 | 6], [[8, [0 | 30], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [7, [0 | 7], 8, [9, 47, 0 | 7], 9, 2, 10, [6, 0 | 14], 0 | 2], 9, 2, 10, [6, 0 | 29], 0 | 2], 6, [5, [1 | 0], 0 | 6], [1 | 0], 0 | 0], 8, [5, [0 | 14], 0 | 2], 0 | 6], 0 | 1], 0 | 1], [[7, [8, [1, 1 | 1], [1, 8, [1 | 0], 8, [1, 6, [5, [1 | 0], 0 | 60], [0 | 6], 9, 2, 10, [60, 8, [9, 342, 0 | 31], 9, 2, 10, [6, 0 | 124], 0 | 2], 10, [6, 8, [9, 20, 0 | 31], 9, 2, 10, [6, [0 | 125], 0 | 14], 0 | 2], 0 | 1], 9, 2, 0 | 1], 0 | 1], 11, [1953718630, 1, 7107949, [0 | 7] | 0], 0 | 1], [[7, [8, [1, 0 | 0], [1, 6, [5, [1 | 0], 0 | 12], [0 | 13], 9, 2, 10, [6, [8, [9, 342, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 4, 0 | 13], 0 | 1], 0 | 1], 11, [1953718630, 1, 6579297, [0 | 7] | 0], 0 | 1], [[7, [8, [1, 0 | 0], [1, 6, [5, [0 | 12], 0 | 13], [1 | 0], 6, [8, [9, 343, 0 | 7], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], [1 | 0], 1 | 1], 0 | 1], 11, [1953718630, 1, 6648940, [0 | 7] | 0], 0 | 1], [7, [8, [1, 1 | 1], [1, 6, [5, [1 | 0], 0 | 13], [0 | 0], 8, [1 | 0], 8, [1, 6, [8, [9, 343, 0 | 31], 9, 2, 10, [6, [0 | 124], 0 | 125], 0 | 2], [0 | 6], 9, 2, 10, [60, 8, [9, 47, 0 | 31], 9, 2, 10, [6, [0 | 124], 0 | 125], 0 | 2], 10, [6, 4, 0 | 6], 0 | 1], 9, 2, 0 | 1], 0 | 1], 11, [1953718630, 1, 7760228, [0 | 7] | 0], 0 | 1], [7, [8, [1 | 0], [1, 6, [5, [1 | 0], 0 | 6], [0 | 0], 8, [1 | 0], 8, [1, 6, [5, [0 | 30], 4, 0 | 6], [0 | 6], 9, 2, 10, [6, 4, 0 | 6], 0 | 1], 9, 2, 0 | 1], 0 | 1], 11, [1953718630, 1, 6514020, [0 | 7] | 0], 0 | 1], 7, [8, [1, 0 | 0], [1, 6, [6, [5, [0 | 12], 0 | 13], [1 | 1], 1 | 0], [6, [8, [1, 6, [5, [1 | 0], 0 | 28], [1 | 0], 6, [6, [6, [5, [1 | 0], 0 | 29], [1 | 1], 1 | 0], [6, [9, 2, 10, [14, [8, [9, ...], 9, ...], 8, [9, ...], 9, ...], 0 | 1], [1 | 0], 1 | 1], 1 | 1], [1 | 0], 1 | 1], 9, 2, 0 | 1], [1 | 0], 1 | 1], 1 | 1], 0 | 1], 11, [1953718630, 1, 6845548, [0 | 7] | 0], 0 | 1], 7, [8, [1, 0 | 0], [1, 6, [8, [9, 84, 0 | 7], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], [1 | 1], 1 | 0], 0 | 1], 11, [1953718630, 1, 6845543, [0 | 7] | 0], 0 | 1], [7, [8, [1, 0 | 0], [1, 6, [8, [9, 343, 0 | 7], 9, 2, 10, [6, [0 | 28], 0 | 29], 0 | 2], [1 | 1], 1 | 0], 0 | 1], 11, [1953718630, 1, 6648935, [0 | 7] | 0], 0 | 1], [7, [8, [1, 1 | 1], [1, 6, [5, [1 | 0], 0 | 13], [0 | 0], 8, [9, 47, 0 | 7], 9, 2, 10, [6, [0 | 28], 7, [0 | 3], 8, [9, 4, 0 | 7], 9, 2, 10, [6, [0 | 29], 7, [0 | 3], 8, [9, 170, 0 | 7], 9, 2, 10, [6, ...], 0 | 2], 0 | 2], 0 | 2], 0 | 1], 11, [1953718630, 1, 6582125, [0 | 7] | 0], 0 | 1], 7, [8, [1, 0 | 0], [1, 6, [5, [1 | 0], 0 | 13], [0 | 12], 9, 2, 10, [6, [8, [9, 342, 0 | 7], 9, 2, 10, [6, 0 | 28], 0 | 2], 8, [9, 342, 0 | 7], 9, 2, 10, [6, 0 | 29], 0 | 2], 0 | 1], 0 | 1], 11, [1953718630, 1, 6452595, [0 | 7] | 0], 0 | 1], [0 | 3] | 909]}, 10000)
    ** (EXIT) shutdown
    (elixir 1.15.5) lib/gen_server.ex:1074: GenServer.call/3
    test/node/mempool_test.exs:101: AnomaTest.Node.Mempool."test death"/1
    iex:91: (file)
    iex:91: (file)

[mariari]

I figured out why it fails

  test "Transaction print", %{node: node} do
    key = 666
    Ccom.subscribe(node.executor, self())
    Mcom.hard_reset(node.mempool)

-   zero_tx = Mcom.tx(node.mempool, zero_counter(key))
+   zero_tx = Mcom.tx(node.mempool, {:kv, zero_counter(666)})
    assert Mcom.pending_txs(node.mempool) == [zero_tx]
    Mcom.soft_reset(node.mempool)
  end

[mariari]

At least I believe that is what is happening, basically we have:

• A genserver, sending a mesage to another genserver, who adds a child to a Task.Supervisor via Task.Supervisor.async. This async task should always die
• The Task should always die, as it's malformed, and it won't pattern match
• However it sometimes takes over 80 attempts to realize the error upwards

Not running this via the indirection, it dies like I'd expect

iex(mariari@Gensokyo)75> Ccom.new_transaction(node.executor, id, zero)
****************** 1: nil
****************** 2: nil
%Task{
  mfa: {Anoma.Node.Executor.Worker, :run, 3},
  owner: #PID<0.1850.0>,
  pid: #PID<0.2518.0>,
  ref: #Reference<0.0.236803.1133210647.4232642567.89771>
}
** (EXIT from #PID<0.1850.0>) shell process exited with reason: an exception was raised:
    ** (FunctionClauseError) no function clause matching in Anoma.Node.Executor.Worker.run/3

Mind you what is really being run and why it's dying is

Ccom.fire_new_transaction(node.executor, id, zero)

which shows the weird behavior

[mariari]

Okay I figured out why we didn't get the report.

We reset the transactions too fast, before the errors can propagate up, we end up kiling the process in the soft_reset

[mariari]

The

config :logger,
  level: :warning,
  handle_otp_reports: true,
  handle_sasl_reports: true

options make it obvious so we will see it even if it doesn't happen