search

ipfs / helia-verified-fetch

A fetch-like API for obtaining verified & trustless IPFS CIDs on the web
https://npmjs.com/package/@helia/verified-fetch
Other
19 stars 4 forks source link

bug: gateway-conformance support for query parameter #35

Closed SgtPooki closed 7 months ago

SgtPooki commented 7 months ago

i'm doing some tests on https://github.com/ipfs/helia-http-gateway/pull/67/ and noticed one oddity. we're not converting dag-json properly? the content-type is being set, incorrectly, to text/plain; charset=utf-8

We should be falling into the block similar to https://github.com/ipfs/helia-verified-fetch/blob/39bb14e0a7a8441ff0c323db15264435e0121b23/packages/verified-fetch/src/verified-fetch.ts#L243-L249 but it doesn't look like we have a path from raw to dag-json ?

requesting http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json results in the following output from verified-fetch:

[helia] 2024-03-30T00:01:23.669Z helia:verified-fetch fetch http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json
[helia] 2024-03-30T00:01:23.676Z helia:verified-fetch incoming accept header "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"
[helia] 2024-03-30T00:01:23.676Z helia:verified-fetch incoming query format "dag-json", mapped to application/vnd.ipld.dag-json
[helia] 2024-03-30T00:01:23.676Z helia:verified-fetch output type application/octet-stream
[helia] 2024-03-30T00:01:23.676Z helia:verified-fetch:trace finding handler for cid code "85" and output type "application/octet-stream"
[helia] 2024-03-30T00:01:23.676Z helia:verified-fetch:trace calling handler "handleRaw"
[helia] 2024-03-30T00:01:23.676Z helia:verified-fetch:byte-range-context:trace no range request detected
[helia] 2024-03-30T00:01:23.677Z helia:trustless-gateway-block-broker getting block for bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm from http://127.0.0.1:8081/
[helia] 2024-03-30T00:01:23.720Z helia:trustless-gateway-block-broker:trace got block for bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm from http://127.0.0.1:8081/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=raw
[helia] 2024-03-30T00:01:23.721Z helia:verified-fetch:byte-range-context:trace set _fileSize to 34
[helia] 2024-03-30T00:01:23.721Z helia:verified-fetch:byte-range-context:trace requestRangeStart and requestRangeEnd are null
[helia] 2024-03-30T00:01:23.721Z helia:verified-fetch:byte-range-context:trace set request body with fileSize 34
[helia] 2024-03-30T00:01:23.721Z helia:verified-fetch:byte-range-context:trace returning body unmodified for non-range, or invalid range, request
[helia] 2024-03-30T00:01:23.724Z helia:verified-fetch:trace setting content type to "text/html; charset=utf-8"

gateway conformance output is:

[gateway-conformance] --- FAIL: TestGatewayJsonCbor/GET_UnixFS_file_with_JSON_bytes_is_returned_with_application%2Fjson_Content-Type_-_with_headers (0.02s)
[gateway-conformance] --- FAIL: TestGatewayJsonCbor (0.04s)
[gateway-conformance] --- FAIL: TestDagPbConversion/GET_UnixFS_file_as_DAG-JSON_with_format=dag-json_converts_to_the_expected_Content-Type/Header_Content-Type (0.00s)
[gateway-conformance]     run.go:53: 
[gateway-conformance]         Name: GET UnixFS file as DAG-JSON with format=dag-json converts to the expected Content-Type
[gateway-conformance]         Hint: 
[gateway-conformance]         
[gateway-conformance]         Error: Header 'Content-Type' expected 'application/vnd.ipld.dag-json', got 'text/plain; charset=utf-8'
[gateway-conformance]         
[gateway-conformance]         Expected Request:
[gateway-conformance]         {
[gateway-conformance]           "method": "GET",
[gateway-conformance]           "path": "/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm",
[gateway-conformance]           "query": {
[gateway-conformance]             "format": [
[gateway-conformance]               "dag-json"
[gateway-conformance]             ]
[gateway-conformance]           }
[gateway-conformance]         }
[gateway-conformance]         
[gateway-conformance]         Actual Request:
[gateway-conformance]         GET /ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json HTTP/1.1
[gateway-conformance]         Host: host.docker.internal:8080
[gateway-conformance]         User-Agent: Go-http-client/1.1
[gateway-conformance]         Accept-Encoding: gzip
[gateway-conformance]         
[gateway-conformance]         
[gateway-conformance]         
[gateway-conformance]         Expected Response:
[gateway-conformance]         {
[gateway-conformance]           "statusCode": 200,
[gateway-conformance]           "headers": [
[gateway-conformance]             {
[gateway-conformance]               "key": "Content-Type",
[gateway-conformance]               "check": {}
[gateway-conformance]             },
[gateway-conformance]             {
[gateway-conformance]               "key": "Content-Disposition",
[gateway-conformance]               "check": {}
[gateway-conformance]             },
[gateway-conformance]             {
[gateway-conformance]               "key": "Content-Type",
[gateway-conformance]               "check": {},
[gateway-conformance]               "not": true
[gateway-conformance]             }
[gateway-conformance]           ],
[gateway-conformance]           "body": "eyIvIjp7ImJ5dGVzIjoiU1NCaGJTQmhJSFI0ZENCbWFXeGxJRzl1SUhCaGRHZ2dkMmwwYUNCMWRHWTRDZyJ9fQ=="
[gateway-conformance]         }
[gateway-conformance]         
[gateway-conformance]         Actual Response:
[gateway-conformance]         HTTP/1.1 200 OK
[gateway-conformance]         Content-Length: 31
[gateway-conformance]         Access-Control-Allow-Headers: Content-Type, Range, User-Agent, X-Requested-With
[gateway-conformance]         Access-Control-Allow-Methods: GET, HEAD, OPTIONS
[gateway-conformance]         Access-Control-Allow-Origin: *
[gateway-conformance]         Access-Control-Expose-Headers: Content-Range, Content-Length, X-Ipfs-Path, X-Ipfs-Roots, X-Chunked-Output, X-Stream-Output
[gateway-conformance]         Connection: keep-alive
[gateway-conformance]         Content-Type: text/plain; charset=utf-8
[gateway-conformance]         Date: Fri, 29 Mar 2024 23:56:49 GMT
[gateway-conformance]         Keep-Alive: timeout=72
[gateway-conformance]         
[gateway-conformance]         
[gateway-conformance]
SgtPooki commented 7 months ago

it looks like the incoming accept header is overriding things:

const incomingAcceptHeader = requestHeaders.get('accept');
        if (incomingAcceptHeader != null) {
            this.log('incoming accept header "%s"', incomingAcceptHeader);
        }
        const queryFormatMapping = queryFormatToAcceptHeader(query.format);
        if (query.format != null) {
            this.log('incoming query format "%s", mapped to %s', query.format, queryFormatMapping);
        }
        const acceptHeader = incomingAcceptHeader ?? queryFormatMapping;

From a browser request, this is automatically being set for me:

helia:verified-fetch incoming accept header "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"

for a fetch request from the browser, this is also being set for me:

var resp = await fetch("http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json")
var resp = await fetch("http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json", {headers: {}})
// both result in: helia:verified-fetch incoming accept header "*/*"

curl also results in accept: */*

SgtPooki commented 7 months ago

if I try var resp = await fetch("http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json", {headers: { accept: undefined }})

I get

[helia] 2024-03-30T00:28:30.967Z helia-http-gateway:server fetching url "http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json" with @helia/verified-fetch
[helia] 2024-03-30T00:28:30.967Z helia:verified-fetch fetch http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json
[helia] 2024-03-30T00:28:30.967Z helia:verified-fetch incoming accept header "undefined"
[helia] 2024-03-30T00:28:30.967Z helia:verified-fetch incoming query format "dag-json", mapped to application/vnd.ipld.dag-json
[helia] 2024-03-30T00:28:30.967Z helia:verified-fetch output type undefined
[helia] 2024-03-30T00:28:30.968Z helia-http-gateway:server verified-fetch response not ok:  406
[helia] 2024-03-30T00:28:30.969Z helia-http-gateway:server:trace request destroyed for url "http://helia-http-gateway.localhost/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json"