ipni / index-provider

📢 Index Provider
Other
36 stars 17 forks source link

verify / list seems not to work when rm advertisements present #179

Closed willscott closed 2 years ago

willscott commented 2 years ago

fails trying to get a block that doesn't exist

willscott commented 2 years ago
./cmd/provider/provider list --provider-addr-info /ip4/59.138.253.69/tcp/55555/p2p/12D3KooWSPtnggJL5wkoAmtgGCosViEC6Z2heTmt7ZTF98GYPyKq --aerl 1
ID:          baguqeeqqs5l2anifxmmb45hsedz326az24
PreviousID:  baguqeeqqvtphlli7xhee7y5dtzukdohm4a
ProviderID:  12D3KooWSPtnggJL5wkoAmtgGCosViEC6Z2heTmt7ZTF98GYPyKq
Addresses:   [/ip4/59.138.253.69/tcp/55555]
Is Remove:   true
Entries:
  Chunk Count: 0
  Total Count: 0
Note: More entries are available but not synced due to the configured entries recursion limit.
 ./cmd/provider/provider list --provider-addr-info /ip4/59.138.253.69/tcp/55555/p2p/12D3KooWSPtnggJL5wkoAmtgGCosViEC6Z2heTmt7ZTF98GYPyKq --aerl 2
2022-02-18T15:13:45.927+0100    ERROR   go-legs-dtsync  dtsync/sync.go:170  datatransfer failed {"cid": "baguqeeqqs5l2anifxmmb45hsedz326az24", "peer": "12D3KooWSPtnggJL5wkoAmtgGCosViEC6Z2heTmt7ZTF98GYPyKq", "message": "data transfer channel QmcqHP5wTfg9QcWDqxYfJ97YdLMv6tTrBi9AAe2c5zF6tD-12D3KooWSPtnggJL5wkoAmtgGCosViEC6Z2heTmt7ZTF98GYPyKq-1645193623922261001 failed to transfer data: channel QmcqHP5wTfg9QcWDqxYfJ97YdLMv6tTrBi9AAe2c5zF6tD-12D3KooWSPtnggJL5wkoAmtgGCosViEC6Z2heTmt7ZTF98GYPyKq-1645193623922261001: graphsync request failed to complete: remote peer is missing block: bafkreehdwdcefgh4dqkjv67uzcmw7oje"}
datatransfer failed
masih commented 2 years ago

This is caused by removal advertisements that include a list of entries, which are no longer available on the SP side.

Note that the indexing protocol initially intended to support explicit removal of multihashes in removal ads. But later we decided to only support remove all here.

This also is a known bug where removal ads are not retrievable due to an internal mapping issue.

The priority of that ticket was reduced since the assumption was that removal ads are not integrated in lotus yet. But looks like the lotus integration indeed publishes removal ads when a deal expires or slashed.

Path forward:

  1. Fix #174
  2. In CLI for backward compatibility purposes, ignore entries in removal advertisement and treat them all as "remove all", i.e. do not attempt to list entries in a removal ad.

Note that removal ad cannot be used as a source of multihashes to perform verify, because its originally advertised entries are most likely no longer traversable to list the multihashes for verification.