search

filecoin-project / lotus

Reference implementation of the Filecoin protocol, written in Go
https://lotus.filecoin.io/
Other
2.85k stars 1.27k forks source link

failed to wait for porep inclusion: found message with equal nonce as the one we are looking for that is NOT a valid replacement message #8866

Closed licornduan closed 2 years ago

licornduan commented 2 years ago

Checklist

Lotus component

Lotus Version

Daemon:  1.14.0+mainnet+git.47d77c933.dirty+api1.5.0
Local: lotus version 1.14.0+mainnet+git.47d77c933.dirty

Daemon:  1.14.0+mainnet+git.47d77c933.dirty+api1.3.0
Local: lotus-miner version 1.14.0+mainnet+git.47d77c933.dirty

Describe the Bug

SectorID: 122982 Status: CommitWait CIDcommD: baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq CIDcommR: bagboea4b5abcakyd73w4glpiya3kewiiwjxhycwzqi7rrct3x7cx6tabgdwqq5to Ticket: b2391c9a41280d4c8f401d776230f0429d899c10d4865ee3b513eeafd81ac9b4 TicketH: 1877169 Seed: ea597bce66331e02f10aab31bb7a599e9c71d4b93dccf717281df7b0165cfe66 SeedH: 1878852 Precommit: bafy2bzacedwo3xrqn65qd2seezaiaxrnfn7flu3dy55rdqvf2o2t5ofzpl3hk Commit: bafy2bzacedgaggotn27fd4jutieo3u7hklxaewdpdmdj2sbjunpiamt3slcoi Deals: [0] Retries: 0

Event Log:

  1. 2022-06-08 08:11:56 +0800 CST: [event;sealing.SectorStartCC] {"User":{"ID":122982,"SectorType":8}}
  2. 2022-06-08 08:35:48 +0800 CST: [event;sealing.SectorPacked] {"User":{"FillerPieces":[{"Size":34359738368,"PieceCID":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}]}}
  3. 2022-06-08 08:35:48 +0800 CST: [event;sealing.SectorTicket] {"User":{"TicketValue":"sjkcmkEoDUyPQB13YjDwQp2JnBDUhl7jtRPur9gaybQ=","TicketEpoch":1877169}}
  4. 2022-06-08 12:20:48 +0800 CST: [event;sealing.SectorPreCommit1] {"User":{"PreCommit1Out":"eyJfbG90dXNfU2VhbFJhbmRvbW5lc3MiOiJzamtjbWtFb0RVeVBRQjEzWWpEd1FwMkpuQkRVaGw3anRSUHVyOWdheWJRPSIsImNvbW1fZCI6WzcsMTI2LDk1LDIyMiw1MywxOTcsMTAsMTQ3LDMsMTY1LDgwLDksMjI3LDczLDEzOCw3OCwxOTAsMjIzLDI0MywxNTYsNjYsMTgzLDE2LDE4Myw0OCwyMTYsMjM2LDEyMiwxOTksMTc1LDE2Niw2Ml0sImNvbmZpZyI6eyJpZCI6InRyZWUtZCIsInBhdGgiOiIvd29ya2VyL2xvY2FsMTU5L3NlYWwvY2FjaGUvcy10MDIzMjE5LTEyMjk4MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MjE0NzQ4MzY0N30sImxhYmVscyI6eyJTdGFja2VkRHJnMzJHaUJWMSI6eyJfaCI6bnVsbCwibGFiZWxzIjpbeyJpZCI6ImxheWVyLTEiLCJwYXRoIjoiL3dvcmtlci9sb2NhbDE1OS9zZWFsL2NhY2hlL3MtdDAyMzIxOS0xMjI5ODIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci0yIiwicGF0aCI6Ii93b3JrZXIvbG9jYWwxNTkvc2VhbC9jYWNoZS9zLXQwMjMyMTktMTIyOTgyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMyIsInBhdGgiOiIvd29ya2VyL2xvY2FsMTU5L3NlYWwvY2FjaGUvcy10MDIzMjE5LTEyMjk4MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTQiLCJwYXRoIjoiL3dvcmtlci9sb2NhbDE1OS9zZWFsL2NhY2hlL3MtdDAyMzIxOS0xMjI5ODIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci01IiwicGF0aCI6Ii93b3JrZXIvbG9jYWwxNTkvc2VhbC9jYWNoZS9zLXQwMjMyMTktMTIyOTgyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItNiIsInBhdGgiOiIvd29ya2VyL2xvY2FsMTU5L3NlYWwvY2FjaGUvcy10MDIzMjE5LTEyMjk4MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTciLCJwYXRoIjoiL3dvcmtlci9sb2NhbDE1OS9zZWFsL2NhY2hlL3MtdDAyMzIxOS0xMjI5ODIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9LHsiaWQiOiJsYXllci04IiwicGF0aCI6Ii93b3JrZXIvbG9jYWwxNTkvc2VhbC9jYWNoZS9zLXQwMjMyMTktMTIyOTgyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItOSIsInBhdGgiOiIvd29ya2VyL2xvY2FsMTU5L3NlYWwvY2FjaGUvcy10MDIzMjE5LTEyMjk4MiIsInJvd3NfdG9fZGlzY2FyZCI6Nywic2l6ZSI6MTA3Mzc0MTgyNH0seyJpZCI6ImxheWVyLTEwIiwicGF0aCI6Ii93b3JrZXIvbG9jYWwxNTkvc2VhbC9jYWNoZS9zLXQwMjMyMTktMTIyOTgyIiwicm93c190b19kaXNjYXJkIjo3LCJzaXplIjoxMDczNzQxODI0fSx7ImlkIjoibGF5ZXItMTEiLCJwYXRoIjoiL3dvcmtlci9sb2NhbDE1OS9zZWFsL2NhY2hlL3MtdDAyMzIxOS0xMjI5ODIiLCJyb3dzX3RvX2Rpc2NhcmQiOjcsInNpemUiOjEwNzM3NDE4MjR9XX19LCJyZWdpc3RlcmVkX3Byb29mIjoiU3RhY2tlZERyZzMyR2lCVjFfMSJ9"}}
  5. 2022-06-08 12:50:32 +0800 CST: [event;sealing.SectorPreCommit2] {"User":{"Sealed":{"/":"bagboea4b5abcakyd73w4glpiya3kewiiwjxhycwzqi7rrct3x7cx6tabgdwqq5to"},"Unsealed":{"/":"baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq"}}}
  6. 2022-06-08 12:50:32 +0800 CST: [event;sealing.SectorPreCommitBatch] {"User":{}}
  7. 2022-06-08 13:50:34 +0800 CST: [event;sealing.SectorPreCommitBatchSent] {"User":{"Message":{"/":"bafy2bzacedwo3xrqn65qd2seezaiaxrnfn7flu3dy55rdqvf2o2t5ofzpl3hk"}}}
  8. 2022-06-08 13:54:33 +0800 CST: [event;sealing.SectorPreCommitLanded] {"User":{"TipSet":"AXGg5AIgGa/j6SneH+zlOzYTIwOyvFJ7cJnjZz0fGSFTkIrGMkgBcaDkAiBwB5aXDbG2aUesCZQVpDsJw3xPTjtVUgFkzaCfLvTEYAFxoOQCIPEcLcVpZaxqJ0/ivwy0d4xX8bZRhB20IZaaz6xYex+sAXGg5AIgRbGPKZWLt1tJYq+L1ehyJJEaLZxd79N1ws9crvRQzYQBcaDkAiAeH0PZ4coDY1venDe67xmBvvByyTu4km7KavYInAn1ZgFxoOQCIM3CbwCIU9izCl16otTq4aKy2VkcEq0LLB2MwXJqkCsu"}}
  9. 2022-06-08 15:09:44 +0800 CST: [event;sealing.SectorSeedReady] {"User":{"SeedValue":"6ll7zmYzHgLxCqsxu3pZnpxx1Lk9zPcXKB33sBZc/mY=","SeedEpoch":1878852}}
  10. 2022-06-08 15:30:52 +0800 CST: [event;sealing.SectorCommitted] {"User":{"Proof":"jJEE9lI7MzLk3jC5jV7LieuZhFii3oBtRi+sUJurOc7LJMXhQp3GvtgQVZGnO5h7gYvfG4szzQYYblnfOYpVexv4bn2IXyNinkVYobdvzGYCkyxcZEwwQpplVFo7ZUkxFoj1dYKrTnXR/TN38HYTjIoZeA8k1yMpCh+BLDXX+F/RVTgX0LdVXo1t1uoQbqzAiLPltStRc8epCgTa7gyBsQZfU5Lwnxwyt0ocsX6Imcr/e3wWeJHYnlHBNmMsDCpOsID2KIH75+tlDqoM0/D/+cfjJVmHcCBuctd5XWmBhPZM/9k5nmwI4hjeFBdSTrq7hDDUyyDs30uNJnXOK9nu5CAbFxMSCVGpx+xyL7XYFePm6jRJ+LpWQ+z5S2jOnaCHF56Vilxc+OBemsjbLfzJOcQyCH5FuPLVVxKtmQdzheV6Cdf2s9WJniMvmOjbwGDoj9BoLZrogWZnnhIt8qGpRJJ13L/SWeQkMihPjqYSsn9QruoBW0PGEn7R5rtawHB5tdMF/Y4QLeviMMmSPhzXno/QneExjpK3OPnPyywEUPKJiq8n8H+unETqaVINrQxUtQB1M8uULrZPM3ZRbLkcXSLY7MtbsH2GraI4HJRtoC2o4CE6K9bOyUZTWzSYo4RWDAwc0rQX/fVbtMsrCvvPdYW59afseZ2UHN0fnnzpOpyPrDP00J5TJG/1FqYsfi1gjqWYdIjJU/pofCWlF4cIRpovRp0gbvS/Wq81pkN1OB5zsgRlzV1tUr8lD8TYU8RAimOkwvAxd0JlLc3ZUCbUZQ5cQmR0YAGr6Syr1H1vgROaZdQEQRNpavutyCaRSJqQt7QncKwhikYSn4+A0WdqGyXjTN+aaR+xmHKXFLCGwlIfCVYOKjBRB/agUSu7DvNWA4euc+QAvPgD/frHzAD3ZOvHUJwoeP+w3vlGEBbNVUZT2ZT8ogZLBO1+/HfS7U66sRzOVIj2mTh5xOOXEAxNoJyJhH9No7I62hIC0J6bU+4bAc2/8CyjUtOyeD+wDuB+ge+EYaYWH1SCzEAvyjtCFKy1N6g68TvFRhoe70fF5yqow5n12ShtZzbGFsrL5KYgtvmFYZKS75eyTd7ZPfdhJdVdp9w0/Zv1zLIOe/FdQYPRf5c17BamJF9rrnv/sSyXGfuz3MRfgiN0rqMu1lmKzI05B94rILpuIvLZI8cXpF3xRCr7P6LxnzIU+FSmftPViDmLZcOghMDjBXLB4N7FomUlRjWYrzYujVM5oQTknJZY2DzYyqDPU7Zy2cJ7g2E9oY5sKTEltD5Bp8eq1qqY7BsANr8naOdqUNwonu2sFmtNGYJmmw2PFH/ZzUYGdi9jpD3Rw218G7RIrpGepuqS1Wqgm1mY7c2Do+iMc4q4EtLFWA+OREUtmTyAAb3KGuGdBW4JmM7GQDQIHzyMyTABmndrIX/QxYL325aQYvQjUdfllZQDUa8mjUFLtq14CjQTlvoF5/tJ9HK+XXhQHiQxgMzxXqcoknArxrjeYHbQlsNxN3zIXKNnxyRIitJBKF6WmH204j9tgaMOIxEu+di9SzCJBcwXn7z++XW6lD3a8rvYxMc/E3/jaIWddLIFQaYZjs8VKG/bpc2TKxpgGDiWVet/mNC8izykWBDWJR/yGzxHokc7P6/xvD9xaG+6q+kzEnw497F28fiqZSXFElkQdXOu4udvzTNzlEziOkf5wgiqvQxpBwN4K8yYwKJ6tweumLwO75ey6zF8xEVf++V5KUyMXz6gKc2Qyt47eMPsxEDTStrGcjs5K9TqTqJDoQY0qrYH3qYUD7e2IjNh8OizyIHrZBGzWhzqXrr4xbK0F14IEKNgA0Bdb7Bte9JLmp7kqlR10kZzOc3J+nHFvGtHRbxkmT9XylpNFV3nyAl775/TtUUpMRR8v+Jd/2uZbXQrClGuGhDLQGMeOowErx9OM9X+wKr4ecmJHi4fAgkgLwxDe5sqQmrn5BZdclwe1ZeUkCG1Ae0/X3ybjnqHXGPlKtfIhDKBbToQ+SojBdN4NTe6w6RqfIrEpYG8he7Vm+ctszBR8i1FQl982dE+iCWWdluJjYqIpwDZRXoqmb44IA8NSC+LViuqZiLYeDjlgcMhlWklmxLtTQ8ALfGTO4pkFX1NnjSZzbCZf02yrFOpBnlxwY16qmYBK8IH5tEV1S7pCqbZzyZb+L1DQuhTaFekvVsXB+NYQZOIt5ebxqJ9PYSc2Dq2Yq56+GY31s31pV3ohluHH/iFGtN6aJ6V50UGlW8/vAVRZxhvx/LNgmwGYRogMmR0FvsGH9UgWMrF5GbBp+QQUJHuXvgCvkng9I4MEwo/zhIjh8tI71xseR20lYad6dTQbBERL6C1bhTPDWMVmJxlIvmYIhAsQN/AfANAjT3lv+SKqFIXf8Y2ox/UbWNfLsFqydwMGrobz+vIutRgAO5uDO1Y/jEd63D2iKk2YyyLGYKLkmqu4TnlijGz5vE4SOkgmMT1BsIkmmlTPCcVq22Pen7nsUdDt8o9+B+foAbdQxw5nhpizbblXBCTk62fWNJAm1rRK8gmaxOhCMyk"}}
  11. 2022-06-08 15:30:52 +0800 CST: [event;sealing.SectorCommitSubmitted] {"User":{"Message":{"/":"bafy2bzacedgaggotn27fd4jutieo3u7hklxaewdpdmdj2sbjunpiamt3slcoi"}}}
  12. 2022-06-08 15:31:39 +0800 CST: [event;sealing.SectorCommitFailed] {"User":{}} failed to wait for porep inclusion: found message with equal nonce as the one we are looking for that is NOT a valid replacement message (F:bafy2bzacedgaggotn27fd4jutieo3u7hklxaewdpdmdj2sbjunpiamt3slcoi n 67491, TS: bafy2bzaceabcgqcyiabelywgfs2kbp37b4nhm7qpvgblx4kmxsqjbfizeg5lu n67491)
  13. 2022-06-08 15:32:39 +0800 CST: [event;sealing.SectorRetryCommitWait] {"User":{}}

Logging Information

10. 2022-06-08 15:30:52 +0800 CST:  [event;sealing.SectorCommitSubmitted]   {"User":{"Message":{"/":"bafy2bzacedgaggotn27fd4jutieo3u7hklxaewdpdmdj2sbjunpiamt3slcoi"}}}
11. 2022-06-08 15:31:39 +0800 CST:  [event;sealing.SectorCommitFailed]  {"User":{}}
    failed to wait for porep inclusion: found message with equal nonce as the one we are looking for that is NOT a valid replacement message (F:bafy2bzacedgaggotn27fd4jutieo3u7hklxaewdpdmdj2sbjunpiamt3slcoi n 67491, TS: bafy2bzaceabcgqcyiabelywgfs2kbp37b4nhm7qpvgblx4kmxsqjbfizeg5lu n67491)
12. 2022-06-08 15:32:39 +0800 CST:  [event;sealing.SectorRetryCommitWait]   {"User":{}}

Repo Steps

1.14.0+mainnet+git.47d77c933.dirty

2022-06-08T13:50:34.102+0800 INFO sectors storage-sealing/states_sealing.go:470 Sector precommitted: 122982 2022-06-08T13:54:33.518+0800 INFO sectors storage-sealing/states_sealing.go:490 precommit message landed on chain: 122982 2022-06-08T15:09:44.692+0800 INFO sectors storage-sealing/states_sealing.go:570 KOMIT 122982 b2391c9a41280d4c8f401d776230f0429d899c10d4865ee3b513eeafd81ac9b4(1877169); ea597bce66331e02f10aab31bb7a599e9c71d4b93dccf717281df7b0165cfe66(1878852); [{34359738368 baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq}]; r:bagboea4b5abcakyd73w4glpiya3kewiiwjxhycwzqi7rrct3x7cx6tabgdwqq5to; d:baga6ea4seaqao7s73y24kcutaosvacpdjgfe5pw76ooefnyqw4ynr3d2y6x2mpq 2022-06-08T15:09:44.929+0800 DEBUG advmgr sector-storage/sched.go:765 SCHED ASSIGNED sqi:1 sector 122982 task seal/v0/commit/1 to window 101 2022-06-08T15:09:44.929+0800 INFO advmgr sector-storage/sched.go:788 wp test sector 122982,task:seal/v0/commit/1 sqi:1 to wid:a7b097ea-4434-4cda-836f-9bc4e5baf47e,hostname:worker27-pc2_1 2022-06-08T15:09:44.929+0800 DEBUG advmgr sector-storage/sched_worker.go:439 assign worker sector 122982 (ready) 2022-06-08T15:09:54.787+0800 DEBUG advmgr sector-storage/sched.go:765 SCHED ASSIGNED sqi:2 sector 122982 task seal/v0/commit/2 to window 3 2022-06-08T15:09:54.787+0800 INFO advmgr sector-storage/sched.go:788 wp test sector 122982,task:seal/v0/commit/2 sqi:2 to wid:c6756d3b-92fc-4a66-866f-2515f66e2a49,hostname:workerc31-c2-2 2022-06-08T15:09:54.788+0800 DEBUG advmgr sector-storage/sched_worker.go:439 assign worker sector 122982 (ready) 2022-06-08T15:31:39.186+0800 WARN sectors storage-sealing/fsm.go:733 sector 122982 got error event sealing.SectorCommitFailed: failed to wait for porep inclusion: found message with equal nonce as the one we are looking for that is NOT a valid replacement message (F:bafy2bzacedgaggotn27fd4jutieo3u7hklxaewdpdmdj2sbjunpiamt3slcoi n 67491, TS: bafy2bzaceabcgqcyiabelywgfs2kbp37b4nhm7qpvgblx4kmxsqjbfizeg5lu n67491) 2022-06-08T15:31:39.217+0800 INFO sectors storage-sealing/states_failed.go:28 CommitFailed(122982), waiting 59.78269733s before retrying

rjan90 commented 2 years ago

Hey @licornduan! From this error message: failed to wait for porep inclusion: found message with equal nonce as the one we are looking for that is NOT a valid replacement message it seems like you are trying to send an entirely different message with the same nonce as message that has already been sent. Are you implementing some kind of custom message-logic?

https://github.com/filecoin-project/lotus/blob/d48b629b4666d5659892611d4163268e68a94588/chain/stmgr/searchwait.go#L254-L259

github-actions[bot] commented 2 years ago

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 24 hours.

github-actions[bot] commented 2 years ago

This issue was closed because it is missing author input.