Reuse a `rayon` thread pool in `Pevm` between runs

[hai-rise]

See https://github.com/risechain/pevm/issues/309#issuecomment-2401775949.

[hai-rise]

@byhashdong Here it is, sir!

[byhashdong]

• A thread pool (of parked threads) may work better. Early experiments with rayon were very poor, so we may need to roll a specialized implementation.
  • after trying rayon, get same result, the performance is worse than original thread pool implementaion
• We should start post-processing as soon as the scheduler flags completion, instead of waiting for the scoped threads to join (very sad tail-end latency).
  • having a little test, found time gap between all threads completed and scoped completed normally less than 100us.
• We should re-consider exiting idle threads early like having no new work after 100 iterations.
  • confused, while task.is_some() { as the code, thread should exit if there is no new work to do. @hai-rise

[hai-rise]

after trying rayon, get same result, the performance is worse than original thread pool implementation

Hopefully one day we understand why!

having a little test, found time gap between all threads completed and scoped completed normally less than 100us.

Ah yes, the scope should end immediately when all threads join. The original idea is more about starting to post-process when Scheduler announces that execution has been done, so even before all threads have completed joining!

confused, while task.is_some() { as the code, thread should exit if there is no new work to do.

While looking for a Scheduler::next_task, we currently loop until execution is done or aborted: https://github.com/risechain/pevm/blob/bed6f220c343d03e6bef0dc7540c531444450d57/src/scheduler.rs#L118-L119

We can add a counter here to return a None to join a thread early when it fails to find a job after X iterations.