feat: add entity matcher w/o preload, add matcher fn for consuming bytes

ipfs / go-unixfsnode

An ADL IPLD prime node that wraps go-codec-dagpb's implementation of protobuf to enable pathing

Other

11 stars 5 forks source link

feat: add entity matcher w/o preload, add matcher fn for consuming bytes #52

Closed rvagg closed 1 year ago

rvagg commented 1 year ago

~~Draft, WIP, for experimenting with~~ https://github.com/filecoin-project/lassie/pull/306

Added MatchUnixFSEntitySelector as a replacement for MatchUnixFSPreloadSelector that doesn't use unixfs-preload. Added a possible deprecation notice to MatchUnixFSPreloadSelector.
Added BytesConsumingMatcher as a matcher function for traversal.WalkMatching that will slurp bytes for you upon match, designed for use with the Match* selectors in here.

rvagg commented 1 year ago

FYI, this .. works, replacing unixfs-preload

    return ssb.ExploreInterpretAs("unixfs-preload", ssb.Matcher())

with

    return ssb.ExploreInterpretAs("unixfs", ssb.ExploreUnion(ssb.Matcher(),
        ssb.ExploreRecursive(
            selector.RecursionLimitDepth(1),
            ssb.ExploreAll(ssb.ExploreRecursiveEdge()),
        ),
    ))

and consuming bytes in a matcher.

But I think it fails for the non-unixfs case where we want to match "entity". I wonder was that why this approach wasn't taken with the original request that sparked unixfs-preload (filecoin data explorer IIRC)? Or was there something else I'm missing?

willscott commented 1 year ago

My explanation here is that preload when implemented was focused almost entirely on the directory case rather than on the file case.

hannahhoward commented 1 year ago

@rvagg doesn't this still traverse the dir links? It's not the same no? Therefore it traverses more than the required bytes?

rvagg commented 1 year ago

Ready for review now, updated OP with this:

Added MatchUnixFSEntitySelector as a replacement for MatchUnixFSPreloadSelector that doesn't use unixfs-preload. Added a possible deprecation notice to MatchUnixFSPreloadSelector.
Added BytesConsumingMatcher as a matcher function for traversal.WalkMatching that will slurp bytes for you upon match, designed for use with the Match* selectors in here.