Enumerating HAMT directory retrieves too many blocks?

lidel commented 9 months ago

I tried to open /ipfs/bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze/ and it seems to fetch too many blocks to find /index.html and then /wiki/index.html.

When I open that in Kubo with empty repo for the first time, big chunk of HAMT is fetched, and the +delta produced by ipfs repo stat --size-only is around 25MiB.

However if I open the same via service worker gateway at https://bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze.ipfs.sw.sgtpooki.com/wiki/ it takes a long time to fetch ~50MiB (twice as much?) – see (2) below.

Being unable to find index.html (1) could be related, or not, so I filled separate https://github.com/ipfs-shipyard/helia-service-worker-gateway/issues/19 for that.

2024-02-13_17-15

lidel commented 8 months ago

For testing,

opening /ipfs/bafybeib66nyby767evpqrnpen4u7jeflavtmrhcdkazptyrxaxhlt4qyaa/ should not fetch root blocks of child nodes, only the minimal number of blocks to Enumerate all directory contents.
opening /ipfs/bafybeib66nyby767evpqrnpen4u7jeflavtmrhcdkazptyrxaxhlt4qyaa/9288.png should only include subset of HAMT to get from root to 9288.png

SgtPooki commented 8 months ago

I think this is fixed. See https://github.com/ipfs-shipyard/helia-service-worker-gateway/issues/19#issuecomment-1962195897 && below:

ipfs / service-worker-gateway

Enumerating HAMT directory retrieves too many blocks? #18