Pull based mempool

[dnkolegov]

In most Mine functions we use polling in the default clause of the select statement. This leads to the following million lines in the logs and probably big inefficiency.

2022-08-27T16:09:33.424Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000081971}
2022-08-27T16:09:33.424Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000091091}
2022-08-27T16:09:33.424Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000077593}
2022-08-27T16:09:33.424Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000106447}
2022-08-27T16:09:33.425Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.0001174}
2022-08-27T16:09:33.425Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000089388}
2022-08-27T16:09:33.426Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000085548}
2022-08-27T16:09:33.426Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000086867}
2022-08-27T16:09:33.427Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000084708}
2022-08-27T16:09:33.427Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000087421}
2022-08-27T16:09:33.427Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000087834}
2022-08-27T16:09:33.427Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000084959}
2022-08-27T16:09:33.428Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000081306}
2022-08-27T16:09:33.428Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000129739}
2022-08-27T16:09:33.429Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000084564}
2022-08-27T16:09:33.429Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000087551}
2022-08-27T16:09:33.429Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.00008116}
2022-08-27T16:09:33.429Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000089143}
2022-08-27T16:09:33.430Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.00007906}
2022-08-27T16:09:33.430Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.00007827}
2022-08-27T16:09:33.431Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000081545}
2022-08-27T16:09:33.431Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000085226}
2022-08-27T16:09:33.431Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000080777}
2022-08-27T16:09:33.431Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000088139}
2022-08-27T16:09:33.432Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.000077117}
2022-08-27T16:09:33.432Z    INFO    messagepool messagepool/selection.go:472    message selection done  {"took": 0.0000788}

Could we improve it at least from a usability perspective:

• do not log these message selections with INFO level
• replace select-default to ticker-based approach <-receiveFromMempool.C, but in this case, it is not obvious how often we should read from mempool
• add notifications to mempool and use pulling in the Mine

The first suggestion is the most trivial one.

@adlrocha What do you think?

[adlrocha]

This is a great point, and I agree we should get it fixed.

The first option is the friction-less one (and the most obvious).

I think the the other two may have their trade-offs:

• A ticker-based approach may not be a bad idea, but there's a risk that we may not be able to choose the right ticker time. I think this should be consensus-specific and configurable.
• A push-based mempool is the most efficient alternative, but in this case we may still need some kind of ticker for consensus algorithms that support empty blocks (we may still need to propose new blocks even if there are no messages in the mempool).

I'd suggest to start with 1, document 2 and 3 in an issue, and decide after we re-architect the code. WDYT?

[dnkolegov]

The first option added - https://github.com/filecoin-project/eudico/pull/234/commits/6fb898b5a42ab39b075c404d2c003b55e2e36777