Support seeking in large directories

[Stebalien]

We should probably support seeking in large, sharded directories. That is, allow listing the Nth-Mth file. This would be especially useful for pagination of massive directories.

[warpfork]

Is there any issues or specs for pagination in IPLD in general we can refer to here?

Seeking is a desirable feature, and yet if implemented must be done in such a way that any object splitting is transparent to the hash of parent objects or it would be backsliding on hash convergence if objects are uploaded while using different splitting parameters. (It seems it might be roughly analogous to the chunkfinding problems solved in file chunking by a rabin filter, also? e.g. if one path is added in the beginning or middle of a large dir, it would be preferable not to republish all subsequent dir shards.)

Issue #4, about inlining small files, encounters somewhat similar concerns about transparency to parent hashes. So also similarly, since this is a "can we vary data availability while maintaining hash transparency", this kind of seems like something to bump to IPLD-in-general discussion?

[kevina]

If we include a count of the number of entries for each directory and sharding is implemented in the way I think it should, then it should allow seeking, see #9.

[rvagg]

closing for archival