filecoin-saturn / L1-node

Filecoin Saturn L1 Node • The edge cache layer of Filecoin's decentralized CDN 🪐
Other
143 stars 49 forks source link

Stop wrapping single block requests in CARs and graph requests #289

Open lidel opened 1 year ago

lidel commented 1 year ago

Problem

iiuc L1 always asks Lassie for a CAR:

https://github.com/filecoin-saturn/L1-node/blob/9be51940935ade50a0634d7e02761b26cf405a77/container/shim/src/fetchers/lassie.js#L78

https://github.com/filecoin-saturn/L1-node/blob/9be51940935ade50a0634d7e02761b26cf405a77/container/shim/src/fetchers/lassie.js#L135-L136

This is very unfortunate when the original request was for a single block adds cost of unnecessary CAR serialization and serialization, and creates surface for amplification bugs which are then worked around with things like https://github.com/filecoin-saturn/L1-node/pull/288 (and was the real reason why we did amplification attack against elastic IPFS last week).

Proposed fix

L1 knows if a request was for a single block (inspecting Accept: application/vnd.ipld.raw or ?format=raw), and in such case should ask Lassie the same response type. There should be no CAR.

guanzo commented 1 year ago

Lassie only returns CARs.

AFAIK there's no way to explicitly request single blocks, like you can with the IPFS Gateway's "raw" endpoint. Rather, you can specify a "shallow" depth which gets translated into a MatchUnixFSPreloadSelector, which may or may not return a single block, depending on if the CID is a directory or not.

288 (now running in prod) defaults requests to a shallow depth, before it was full depth. That's the minimal request the L1 can make with the current Lassie API. Lassie metrics show that download bandwidth has reduced by 4x for P90, and up to 32x!!! for P95 requests.

image