HTTP Gateway fails to load sharded website

lidel commented 5 years ago

Part of an effort to run embedded js-ipfs in Brave :lion: https://github.com/ipfs-shipyard/ipfs-companion/issues/716

Version: js-ipfs version: 0.35.0-rc.3
Platform: Linux, Node.js v10.15.0
Subsystem: HTTP Gateway

Type: Feature/Bug?

Severity: High?

Description:

Tried to open wikipedia mirror from js-ipfs's Gateway and got sad. Not sure if a bug or just not implemented?

Steps to reproduce the error:

$ curl http://127.0.0.1:9090/ipfs/bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/
{"statusCode":404,"error":"Not Found","message":"no link named \"wiki\" under bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq"}

alanshaw commented 5 years ago

@achingbrain can you help with this one?

achingbrain commented 5 years ago

I think this error comes from this line in js-ipfs-http-response which is called by the gateway.

It looks like it does a bunch of DAG manipulations assuming it's dealing with a straight dag-pb node. That is I'm not sure this has ever worked with sharded directories.

It should probably use the unixfs-exporter instead as that can handle more types of node.

achingbrain commented 5 years ago

Actually I've just put a fix together for this, I'll submit a PR. We should be able to remove js-ipfs-http-response as a dep too so hooray for ever so slightly smaller bundle sizes.

lidel commented 5 years ago

@achingbrain Thank you for looking into this.

I think js-ipfs-http-response continues to be responsible for generating HTML of directory listings, and it is also a dep of service-worker-gateway – is it possible to fix it upstream, or is refactor too big to be worth it?

That is I'm not sure this has ever worked with sharded directories.

Ok, I've found this TODO in libdweb branch of Companion, so it seems it never worked.

ipfs / js-ipfs