Evergreen deals often show commP mismatch

stuberman commented 2 years ago

Here is one of many examples:

Found eligible file and downloaded: lotus client retrieve --provider f019551 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car
File list on boost node: 2.0G Jul 17 21:37 'baga6e~olkmuuoy__bafyaa~427co6am.car'
Import file boostd import-data bafyreifpprhtw7ttqx3kbmpbgw3mcvirdc65xxfodrzalz63wov6sg62gm /market/deals/baga6e~olkmuuoy__bafyaa~427co6am.car

Error: couldnt import v1.1.0 deal, or find boost deal: given data does not match expected commP (got: baga6ea4seaqgz2tjx4yqn64kzv5ofvksyiepveggschbnxjl6mj6w3cn3kbywjq, expected baga6ea4seaqbw4426sr4yg6zgyhsmfjdojjypuoedotryivagz3m65volkmuuoy)

Lotus versions lotus-miner version

Daemon: 1.16.1+mainnet+git.a2fa07d35+api1.5.0 Local: lotus-miner version 1.16.1+mainnet+git.a2fa07d35

boostd --version

boostd version 1.2.0-rc3+git.7665e14

Details below

Found example eligible small file (2 GiB) curl -sLH "Authorization: $( ./fil-spid.bash f01278 )" https://api.evergreen.filecoin.io/eligible_pieces/anywhere

 {
  "piece_cid": "baga6ea4seaqbw4426sr4yg6zgyhsmfjdojjypuoedotryivagz3m65volkmuuoy",
  "dataset": "ghtorrent-project",
  "padded_piece_size": 2147483648,
  "payload_cids": [
    "bafyaa6qsgafcmalqudsaeifcfvxkuxvlm3hwgkzlqcfvmx266kb57cl2rk67d6sfus6hif4diijaagelucbyabasgafcmalqudsaeidw2vv43h4xixdsahbewrwuxrlvzt5i4xhw3vrwqw6zu4vmynvdcejaaghiwsgn4aykcqeaeghzv6e54bzaqcaibaaeed427co6am"
  ],
  "sources": [
    {
      "source_type": "Filecoin",
      "provider_id": "f0165400",
      "deal_id": 3194795,
      "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D",
      "deal_expiration": "2023-05-28T21:33:00Z",
      "is_filplus": true,
      "sector_id": null,
      "sector_expires": null,
      "sample_retrieve_cmd": "lotus client retrieve --provider f0165400 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car"
    },
    {
      "source_type": "Filecoin",
      "provider_id": "f0694396",
      "deal_id": 3194576,
      "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D",
      "deal_expiration": "2023-05-28T21:21:30Z",
      "is_filplus": true,
      "sector_id": null,
      "sector_expires": null,
      "sample_retrieve_cmd": "lotus client retrieve --provider f0694396 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car"
    },
    {
      "source_type": "Filecoin",
      "provider_id": "f01091840",
      "deal_id": 3195393,
      "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D",
      "deal_expiration": "2023-05-28T20:44:30Z",
      "is_filplus": true,
      "sector_id": null,
      "sector_expires": null,
      "sample_retrieve_cmd": "lotus client retrieve --provider f01091840 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car"
    },
    {
      "source_type": "Filecoin",
      "provider_id": "f02301",
      "deal_id": 3194249,
      "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D",
      "deal_expiration": "2023-05-28T17:33:30Z",
      "is_filplus": true,
      "sector_id": null,
      "sector_expires": null,
      "sample_retrieve_cmd": "lotus client retrieve --provider f02301 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car"
    },
    {
      "source_type": "Filecoin",
      "provider_id": "f066596",
      "deal_id": 3194961,
      "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D",
      "deal_expiration": "2023-05-28T08:39:00Z",
      "is_filplus": true,
      "sector_id": null,
      "sector_expires": null,
      "sample_retrieve_cmd": "lotus client retrieve --provider f066596 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car"
    },
    {
      "source_type": "Filecoin",
      "provider_id": "f019551",
      "deal_id": 3196046,
      "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D",
      "deal_expiration": "2023-05-27T23:32:00Z",
      "is_filplus": true,
      "sector_id": null,
      "sector_expires": null,
      "sample_retrieve_cmd": "lotus client retrieve --provider f019551 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car"
    }
  ],
  "sample_request_cmd": "echo curl -sLH \"Authorization: $( ./fil-spid.bash f01278 )\" https://api.evergreen.filecoin.io/request_piece/baga6ea4seaqbw4426sr4yg6zgyhsmfjdojjypuoedotryivagz3m65volkmuuoy | sh"

echo curl -sLH \"Authorization: $( ./fil-spid.bash f01278 )\" https://api.evergreen.filecoin.io/pending_proposals | sh

"pending_proposals": [ { "deal_proposal_cid": "bafyreifpprhtw7ttqx3kbmpbgw3mcvirdc65xxfodrzalz63wov6sg62gm", "hours_remaining": 65, "piece_size": 2147483648, "piece_cid": "baga6ea4seaqbw4426sr4yg6zgyhsmfjdojjypuoedotryivagz3m65volkmuuoy", "root_cid": "bafyaa6qsgafcmalqudsaeifcfvxkuxvlm3hwgkzlqcfvmx266kb57cl2rk67d6sfus6hif4diijaagelucbyabasgafcmalqudsaeidw2vv43h4xixdsahbewrwuxrlvzt5i4xhw3vrwqw6zu4vmynvdcejaaghiwsgn4aykcqeaeghzv6e54bzaqcaibaaeed427co6am", "deal_start_time": "2022-07-20T16:00:00Z", "deal_start_epoch": 2000880, "sample_import_cmd": "lotus-miner storage-deals import-data bafyreifpprhtw7ttqx3kbmpbgw3mcvirdc65xxfodrzalz63wov6sg62gm baga6e~olkmuuoybafyaa~427co6am.car", "sources": [ { "source_type": "Filecoin", "provider_id": "f0165400", "deal_id": 3194796, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T21:33:00Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f0165400 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f0165400", "deal_id": 3194795, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T21:33:00Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f0165400 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoybafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f0694396", "deal_id": 3194576, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T21:21:30Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f0694396 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoybafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f0694396", "deal_id": 3194577, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T21:21:30Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f0694396 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f01091840", "deal_id": 3195393, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T20:44:30Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f01091840 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoybafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f02301", "deal_id": 3194249, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T17:33:30Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f02301 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoybafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f066596", "deal_id": 3194961, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-28T08:39:00Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f066596 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoy__bafyaa~427co6am.car" }, { "source_type": "Filecoin", "provider_id": "f019551", "deal_id": 3196046, "original_payload_cid": "mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D", "deal_expiration": "2023-05-27T23:32:00Z", "is_filplus": true, "sector_id": null, "sector_expires": null, "sample_retrieve_cmd": "lotus client retrieve --provider f019551 --maxPrice 0 --allow-local --car 'mAXAAehIwCiYBcKDkAiCiLW6qXqtmz2MrK4CLVl9e8oPfiXqKvfH6RaS8dBeDQhIAGIugg4AEEjAKJgFwoOQCIHbVa82fl0XHIBwktG1LxXXM+o5c9t1jaFvZpyrMNqMREgAY6LSM3gMKFAgCGPmvid4HIICAgIAEIPmvid4D' $(pwd)/baga6e~olkmuuoybafyaa~427co6am.car"

ribasushi commented 2 years ago

As investigated at length in this slack thread, this is an unfortunate deficiency in how lotus client retrieve operates today. Specifically it dropping some of the blocks present in the original car results in the behavior you see.

A cumbersome workaround for some of the issues, which is available to you right now is to pass the downloads through kubo ( formerly go-ipfs ). This is not a scalable option, I am only mentioning it as something that came out of the investigation.

The real solution is one or more of:

beef up lotus client retrieve to preserve more fidelity ( although this is a losing battle that might not fix all the issues you are observing )
have more SPs deploy booster-http / have it advertised properly as part of supported protocols / have a client ( filc ? ) transparently pull that

I do not have insight into timelines for the above, tagging @LexLuthr to share any knowledge he has.

@stuberman unfortunately, there isn't much I can do to address this from evergreen's side itself... Nevertheless, keeping issue open to track related efforts

ribasushi commented 6 months ago

No longer relevant with graphsync finally being retired.

ribasushi / spade

Evergreen deals often show commP mismatch #3