search

storacha / w3link

🪐 The IPFS gateway for web3.storage is not "another gateway", but a caching layer that sits on top of existing IPFS public gateways.
Other
24 stars 9 forks source link

w3link's cache didn't handle `accept: application/vnd.ipld.raw` correctly #18

Open Xuanwo opened 2 years ago

Xuanwo commented 2 years ago

Hi, I'm the maintainer of OpenDAL: A rust lib for accessing data freely, painlessly, and efficiently. OpenDAL added ipfs support recently. During our test, we found that w3link didn't cache the correct data if the accept header is set.

For example:

QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ is a directory of opendal's test data: https://ipfs.io/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ

Background: OpenDAL use accept: application/vnd.ipld.raw to get the raw block of the directory for all child links. In this way, we don't need to parse the returning HTML.

However, w3s.link returns raw block even I didn't set accept header:

Shell Log ```shell :( curl -L https://w3s.link/ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ | cat % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 488 100 488 0 0 824 0 --:--:-- --:--:-- --:--:-- 0 ───────┬─────────────────────────────────────────────────────────────────────────────── │ STDIN ───────┼─────────────────────────────────────────────────────────────────────────────── 1 │ ^R2 2 │ "^R �T�xj^^ :^Z+c�$���^?!�ș�bs^Fh,͗��c^R 3 │ normal_dir^X<^R5 4 │ "^R ߀^Q���^Xe�h��@{�+<)!^P��N�Ie L�c�^B^R^Knormal_file^X��^P^R4 5 │ "^R )�~�b$^P��zv��'1^Y^QQ�^L^C^B�^Z��γ�b^R 6 │ ontime.csv^X��^E^R7 7 │ "^R ��^@^]?�ҹh?�2dD<�IY�~d�d��G�qJ^R^Nontime.csv.bz2^X�5^R6 8 │ "^R �5�����/0X�l��Sc -�C^Bڋh�tr��O^R^Montime.csv.gz^X�A^R7 9 │ "^R �!��Ec��Qj�J7�^R�^Q^M�1]��S^XA^G�o�l�^R^Nontime.csv.zst^X�C^RI 10 │ "^R �T�xj^^ :^Z+c�$���^?!�ș�bs^Fh,͗��c^R!special_dir !@#$%^&*()_+-=;'><,?^X<^RL 11 │ "^R ߀^Q���^Xe�h��@{�+<)!^P��N�Ie L�c�^B^R"special_file !@#$%^&*()_+-=;'><,?^X��^P 12 │ ^B^H^A ───────┴─────────────────────────────────────────────────────────────────────────────── ```
vasco-santos commented 2 years ago

@Xuanwo w3link does not support yet format raw and format car queries. We will look into supporting this soon

Xuanwo commented 2 years ago

I'm not familiar with js. Maybe we can add &format=raw in query or accept header if input request contains accept: application/vnd.ipld.raw header while redirecting? I'm willing to contribute to this patch.

Xuanwo commented 2 years ago

The problem I know is Cloudflare's IPFS gateway doesn't support format raw.

vasco-santos commented 2 years ago

Hey @Xuanwo

Thanks for your interest. We probably want to whitelist the gateways that support this and change over time once they support.

If you would like to submit a patch to support this, the repo for it would be the reads pipeline: