AP (Add Piece) multi core performance issues / not always desirable

[benjaminh83]

Checklist

• [X] This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
• [X] This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
• [X] I have a specific, actionable, and well motivated feature request to propose.

Lotus component

• [ ] lotus daemon - chain sync
• [ ] lotus miner - mining and block production
• [X] lotus miner/worker - sealing
• [ ] lotus miner - proving(WindowPoSt)
• [ ] lotus miner/market - storage deal
• [ ] lotus miner/market - retrieval deal
• [ ] lotus miner/market - data transfer
• [ ] lotus client
• [ ] lotus JSON-RPC API
• [ ] lotus message management (mpool)
• [ ] Other

What is the motivation behind this feature request? Is your feature request related to a problem? Please describe.

We are now seeing good progress on the market node and the distribution of deals to AP workers. This highlights the next bottleneck in the deal handling flow, which is the AP process. On my AP/PC1 worker specced with Epyc 7F72, 8xNVMe raid0, 880GiB memory, I observe the following behavior while staging full 32GiB deals (running the AP):

• 1 deal AP time: 4m7s
• 4 deals in parallel AP: 18m20s
• 5 deals in parallel AP: 31m30s

CPU usage is all cores 2-10% - also when running 4 APs on the same time. I think the multicore AP is kind of broken.

Describe the solution you'd like

1. Maybe we should get that variable in there to be able to switch back to single core AP. At least, that gave us a super predictable AP time of 20m, but we could do 2-4-8-12 AP processes in parallel, and not much difference.
2. Alternative could be a variable to control how many cores multi core AP is allowed to use per job.
3. Alternative could be to fix the performance of multi core AP, so it does not as this drastically diminishing performance when running multiple jobs.

Describe alternatives you've considered

No response

Additional context

I think all storage providers will need to be able to do multiple APs, due to the nature of how deals are desirably published in batches, and will start up AP jobs in batches.

[neondragon]

The scheduler also schedules one AP task per available CPU core on a worker, left-over behavior from when AP was single-threaded. It schedules e.g. 16 multi-core AP tasks on a 16 CPU worker. On a 64-core worker, it will schedule maximum 64 AP tasks running in parallel.

As running parallel AP appears to decrease overall throughput, I think we either need performance improvements to AP such that they scale linearly, or a way to limit the maximum number of parallel AP tasks per-worker.