Chain export inefficiencies

[austinabell]

Describe the bug A clear and concise description of what the bug is.

When walking a snapshot, ipld nodes are loaded 2x more than they need to, as well as redundant data structures (2 hashsets and 1 array to do what it seems 1 hashset could)

For extra blockstore reads: Whenever the callback happens, the Cid is loaded from the blockstore (https://github.com/filecoin-project/lotus/blob/a63ef1dcd549d800b93bd504703d5b25f187606a/chain/store/store.go#L1360), but also in addition to this, it's also read when it's used during walking (https://github.com/filecoin-project/lotus/blob/a63ef1dcd549d800b93bd504703d5b25f187606a/chain/store/store.go#L1393, https://github.com/filecoin-project/lotus/blob/a63ef1dcd549d800b93bd504703d5b25f187606a/chain/store/store.go#L1318, ...)

Extra data structures:

• seen and walked hashsets (https://github.com/filecoin-project/lotus/blob/a63ef1dcd549d800b93bd504703d5b25f187606a/chain/store/store.go#L1378) They are updated in lockstep, but why would you need two hashsets for this, either it's been seen and it should be ignored, or it hasn't and it should be recursed.

• cids and out slices (https://github.com/filecoin-project/lotus/blob/a63ef1dcd549d800b93bd504703d5b25f187606a/chain/store/store.go#L1410, https://github.com/filecoin-project/lotus/blob/a63ef1dcd549d800b93bd504703d5b25f187606a/chain/store/store.go#L1430) My memory is failing as to if this would reallocate a new slice or it's just assigning the pointer in go, but neither of these should be needed. They are being added to this vector, then checked against the seen hashset, when this could be done before it's added

Let me know if I'm missing something here, I've implemented without these (using the callback to pass back the loaded bytes and only one hashset) and it seems to work the same, but I could be missing an edge case. This also doesn't matter that much to optimize, so probably a low priority ticket.

To Reproduce Steps to reproduce the behavior:

1. Run '...'
2. See error

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Version (run lotus version):

Additional context Add any other context about the problem here.

[raulk]

@austinabell ditto. I've ran into these inefficiencies when trying to produce a full snapshot. The current logic tries to remove duplicates, but it could be done differently.

[jennijuju]

@vyzo mentioned he may be interested in this😉

[Kubuxu]

Another thing of note is that lotus doesn't seem to stream the export CARs.