Problems with IPNIsync

[cryptowhizzard]

Checklist

• [X] This is not a question or a support request. If you have any boost related questions, please ask in the discussion forum.
• [X] This is not a new feature request. If it is, please file a feature request instead.
• [X] This is not an enhancement request. If it is, please file a improvement suggestion instead.
• [X] I have searched on the issue tracker and the discussion forum, and there is no existing related issue or discussion.
• [X] I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• [X] I did not make any code changes to boost.

Boost component

• [X] boost daemon - storage providers
• [ ] boost client
• [ ] boost UI
• [ ] boost data-transfer
• [X] boost index-provider
• [ ] Other

Boost Version

Latest master of 13 nov 2023

Describe the Bug

Boost generates an error regenerated link from multihash lister did not match the original link; multihashes returned by the lister for the same key are not consistent causing IPNIsync to halt on Indexer side.

Logging Information

2023-11-13T12:46:10.588Z    DEBUG   index-provider-wrapper  indexprovider/wrapper.go:575    returning piece iterator    {"propCid": "bafyreicpfl4qsu4y723rhdu3njkrvvx3uh74tlpxmwr56v4mzehphz242e", "piece": "baga6ea4seaqkljwlwpcjsyol3bxudck5wvzwo3zsnxfocg6ctbgk3xdlljyseiy", "err": null}
2023-11-13T12:46:17.534Z    ERROR   provider/engine engine/linksystem.go:129    Regeneration of entries link from multihash iterator did not match the original link. Check that multihash iterator consistently returns the same entries for the same key. {"want": "baguqeera4v4fm56ddi4hrytmkwn2e64hd6wgwmcze4s6fph36ldv55gsvnya", "got": "baguqeeraclox2bkja373gehvxzflg2nvere6dtjagf4arhfdlzlq6mas57gq"}
2023-11-13T12:46:17.535Z    ERROR   dagsync/ipnisync    ipnisync/publisher.go:235   Failed to load requested block  {"err": "regenerated link from multihash lister did not match the original link; multihashes returned by the lister for the same key are not consistent", "cid": "baguqeera4v4fm56ddi4hrytmkwn2e64hd6wgwmcze4s6fph36ldv55gsvnya"}

Repo Steps

1. Run '...'
2. Do '...'
3. See error '...' ...

[cryptowhizzard]

Apart from this problem, there is a second problem emerging.

Not all deals are getting announced to the network indexer.

error while ingesting ad baguqeerazpknqo4o4ok6oax7tmpib4ebekuahg2i35gvubbpex7rwy55chhq: syncEntriesErr: failed to sync first entry while checking entries type: sync handler failed: failed to traverse requested dag: failed to load node for root cid baguqeeramden5stz3kbgzb7abpl3kastbenkruh36uo372lhrv5mrpek4rta: failed to fetch block for cid baguqeeramden5stz3kbgzb7abpl3kastbenkruh36uo372lhrv5mrpek4rta: fetch request failed: Get "http://212.6.53.92:3104/ipni/v1/ad/baguqeeramden5stz3kbgzb7abpl3kastbenkruh36uo372lhrv5mrpek4rta": net/http: request canceled (Client.Timeout exceeded while awaiting headers) (19h ago)

These timeout are blocking further ingestion of ads.

[masih]

Latest master of 13 nov 2023

Can you provide the concrete commit ID please?

[cryptowhizzard]

To provide some more context:

__Edit removed the screenshot due sensitive data. Please refer to the ASCII below.

This example has not made it to the indexer. Checking bafkreigjnu72cdjpzfjs55xurvif2dqip62zfbwpypohcvpe4kgettvtc4 on cid.contact gives no record for the machine with ip ending on 53.92.

[cryptowhizzard]

In ASCII for the record:

Deal 3e500432-453f-4ea8-9b6c-75c31351d831

CreatedAt	2023-11-12 21:50:01.830 (3 days ago)

f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i 12D3KooWF3msa2qui4j5TMcK4ua7ebTscGGXmZSeG1UizLcWVnqM bafyreiak6opod5caczdl7kgtz5vl2beypxngzx7q4gz4rumonjbd2o5an4 bafkreigjnu72cdjpzfjs55xurvif2dqip62zfbwpypohcvpe4kgettvtc4 Yes Yes Yes baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa 32 GiB (34,359,738,368 bytes) 0 atto 0.007 FIL 0 atto 3,391,587 3,404,020 (in 4 days) 4,918,900 (in a year) 1,514,880 (526 days) 0 atto (Price per epoch x Piece Size x Duration) Online http 16.5 GiB (17,723,430,495 bytes) URL: https://XXXXXXXXXdataset.twinquasar.io/nasanex/baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa.car 0 B (0 bytes) /root/.boost/incoming/3e500432-453f-4ea8-9b6c-75c31351d831.download Yes 38763 0 34,359,738,368 bafy2bzacebyqozi2pcto3kr3k33egtka6qq4ukht4htujktsp2mb6f2cvkdkm 62,966,623 IndexedAndAnnounced (3 days ago) Sealer: Proving

Deal 3e500432-453f-4ea8-9b6c-75c31351d831 CreatedAt 2023-11-12 21:50:01.830 (3 days ago) Client Address f1zo3n62q6j5b3rfc4drj67d5e7hr5pqtygxy6s6i Client Peer ID 12D3KooWF3msa2qui4j5TMcK4ua7ebTscGGXmZSeG1UizLcWVnqM Signed Proposal CID bafyreiak6opod5caczdl7kgtz5vl2beypxngzx7q4gz4rumonjbd2o5an4 Label bafkreigjnu72cdjpzfjs55xurvif2dqip62zfbwpypohcvpe4kgettvtc4 Verified Yes Keep Unsealed Copy Yes Announce To IPNI Yes Piece CID baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa Piece Size 32 GiB (34,359,738,368 bytes) Client Collateral 0 atto Provider Collateral 0.007 FIL Storage Price / epoch / GiB 0 atto Current Epoch 3,391,587 Start Epoch 3,404,020 (in 4 days) End Epoch 4,918,900 (in a year) Duration 1,514,880 (526 days) Storage Fee 0 atto (Price per epoch x Piece Size x Duration) Transfer Mode Online Transfer Type http Transfer Size 16.5 GiB (17,723,430,495 bytes) Transfer Params URL: https://XXXXXXXXXX.twinquasar.io/nasanex/baga6ea4seaqivaftkm4c2ls4ikpmrnskm7oeclnzlgcetcnpiokvfq47cxgcwpa.car Transferred 0 B (0 bytes) Inbound File Path /root/.boost/incoming/3e500432-453f-4ea8-9b6c-75c31351d831.download Delete After Add Piece Yes Sector ID 38763 Sector Data Offset 0 Sector Data Length 34,359,738,368 Publish Message CID bafy2bzacebyqozi2pcto3kr3k33egtka6qq4ukht4htujktsp2mb6f2cvkdkm Chain Deal ID 62,966,623 Checkpoint IndexedAndAnnounced (3 days ago) Status Sealer: Proving