scheduler logic bug

[dafsic]

Checklist

• [X] This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• [X] This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• [X] This is not a new feature request. If it is, please file a feature request instead.
• [X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
• [X] I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• [X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• [x] I did not make any code changes to lotus.

Lotus component

• [x] lotus daemon - chain sync
• [x] lotus miner - mining and block production
• [X] lotus miner/worker - sealing
• [ ] lotus miner - proving(WindowPoSt)
• [x] lotus miner/market - storage deal
• [ ] lotus miner/market - retrieval deal
• [ ] lotus miner/market - data transfer
• [ ] lotus client
• [ ] lotus JSON-RPC API
• [ ] lotus message management (mpool)
• [ ] Other

Lotus Version

lotus version 1.15.2+mainnet+git.d1eeb7ca3.dirty
lotus-miner version 1.15.2+mainnet+git.d1eeb7ca3.dirty

Describe the Bug

00000000 53799 4fb43665 worker-185 C2 assigned(1) 15m19.4s 00000000 53804 4fb43665 worker-185 PC2 prepared 14m30.9s 6f1b0aad 53803 4fb43665 worker-185 PC2 running 14m30.9s 00000000 53798 54346696 worker-186 C2 prepared 16m16.5s a0f8b7fa 53797 54346696 worker-186 C2 running 16m23.6s 00000000 53815 54346696 worker-186 PC1 prepared 24.1s 8063bf87 53810 54346696 worker-186 PC1 running 1h21m52.5s 02064e90 53811 54346696 worker-186 PC1 running 1h5m51.2s 8125858b 53809 54346696 worker-186 PC1 running 2h32m22.3s ce327ba8 53808 54346696 worker-186 PC1 running 2h57m24.8s e8f83bf5 53807 54346696 worker-186 PC1 running 2h57m24.8s 5eba0687 53813 54346696 worker-186 PC1 running 30m10s 58aaa875 53812 54346696 worker-186 PC1 running 50m55.5s f8bc5cc4 53814 54346696 worker-186 PC1 running 8m0.9s

All PC1s is on worker-186, some of these should in worker-185, right?

These machines have the exact same hardware configuration

Logging Information

00000000 53799 4fb43665 worker-185 C2 assigned(1) 15m19.4s
00000000 53804 4fb43665 worker-185 PC2 prepared 14m30.9s
6f1b0aad 53803 4fb43665 worker-185 PC2 running 14m30.9s
00000000 53798 54346696 worker-186 C2 prepared 16m16.5s
a0f8b7fa 53797 54346696 worker-186 C2 running 16m23.6s
00000000 53815 54346696 worker-186 PC1 prepared 24.1s
8063bf87 53810 54346696 worker-186 PC1 running 1h21m52.5s
02064e90 53811 54346696 worker-186 PC1 running 1h5m51.2s
8125858b 53809 54346696 worker-186 PC1 running 2h32m22.3s
ce327ba8 53808 54346696 worker-186 PC1 running 2h57m24.8s
e8f83bf5 53807 54346696 worker-186 PC1 running 2h57m24.8s
5eba0687 53813 54346696 worker-186 PC1 running 30m10s
58aaa875 53812 54346696 worker-186 PC1 running 50m55.5s
f8bc5cc4 53814 54346696 worker-186 PC1 running 8m0.9s

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...' ...

[Reiers]

Hi @dafsic

Thanks for the report. Looks like you are running Lotus with custom code or adjustment .dirty

Please upgrade to stock lotus, make clean all !

Im unable to reproduce the issue here, did you upgrade all workers? Did turn off dealmaking during the upgrade? Did you upgrade with sectors in the pipeline?

If the issue persist - leave a comment on here with new logs and repro steps.

Thank you !

[dafsic]

Thanks Reply

[dafsic]

I know the reason, when there is a new AP or P1 task is coming, the scheduler will compare which worker has lower resource usage. But when a worker has a P2 or C2 task, even if the worker has no other tasks, its usage is 100% because its GPU is used (only one GPU). So even if other workers already have multiple P1 tasks, as long as the GPU is not being used, the AP or P1 task will still be assigned to it. So when assigning AP or P1 tasks, you should only compare CPU utilization.

[dafsic]

@Reiers hope to enhance