SP Retrieval Issues for expired deal content

[davidd8]

This is a tracking ticket for issues related to SPs serving retrievals for expired deal content as seen in autoretrieve:

• Why is autoretrieve requesting content from expired deals? Is this expected behavior? @marinakosti to dig into this.
• What should be the Boost behavior when receiving a request for expired content? It currently rejects those requests at the query phase. @jacobheun mentions that maybe Boost shouldn’t reject right away if the SP still has the data but this may be challenging to resolve for expired deals until the indexing lifecycle improvements land. We can potentially patch the rejection sooner rather than later, but getting the announced index metadata on indexers showing correct sealing/unsealing status is going to take longer.
• Golden Retriever metrics are affected by these requests, bringing down the success rate for SPs in the program. What constitutes a "valid request" for data in Golden Retriever? @rvagg points out that we may want revisit changes to autoretrieve behavior for failed requests

Below is the SP feedback:

0 retrievals success for the last 10 days Autoretrieve is requesting expired deals, boost is rejecting the deals rightaway. our #golden-retriever stats are dropping from 83% to 0% The problem has been highlighted a week ago here : https://filecoinproject.slack.com/archives/C03CKDLEWG1/p1675856817927119 --> I'm getting flooded by the following message (like 10 per second!):

2023-02-08T22:41:28.078+1100    ERROR   retrieval       impl/provider.go:434    Retrieval query: GetAsk: GetRetrievalPricingInput: failed to fetch storage deal state: 1 error occurred:
        * deal 2421142 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed

Checked out the deal on CID Checker, and it expired a few days ago (3rd February). Not sure why or who is requesting a retrieval pricing for an expired deal and why does boost floods the logs with this error (my daemon is also getting errors about it)

[rvagg]

Specifically I think this difference is likely one of the changes impacting throughput:

• Old: https://github.com/application-research/autoretrieve/blob/f94b5ca8d79d58c2243dcbc023e550fada0a938e/filecoin/retriever.go#L130-L132
• New: https://github.com/filecoin-project/lassie/blob/9c0866a55a53fbb9589d310ea09bda7139c72a67/pkg/retriever/sptracker.go#L49-L51

We even increased SP concurrency from 1 to 2 and it hasn't really lifted much, so we seem to be bottlenecked on the number of "available" SPs - that is the number not already at max-concurrency minus those who have been timed out from failures. Since the number of failures is quite high, and we only penalise failures and don't do the reverse for successes, then 5 failures in a window of 30 seconds is going to get you removed from the pool of available SPs for 10 minutes.

We could consider:

• Removing penalisation entirely
• Backing off the penalty timeout, decreasing the window and/or increasing the number of failures needed to timeout
• Cancelling an SP's failure record on success (i.e. 1 success wipes out record of failures in the failure window)

[davidd8]

Linking https://github.com/filecoin-project/boost/issues/1200 for the Boost side of the issue.