ipfs / go-cid

Content ID v1 implemented in go
MIT License
156 stars 47 forks source link

Possible to calculate cid using Writer interface instead of reading full file into memory? #120

Closed shoce closed 3 years ago

shoce commented 3 years ago

As Prefix.Sum() accepts []byte i need to read the full file into memory and then feed to .Sum(). How can i make use of streaming with io.Copy() instead and reduce memory requirements?

welcome[bot] commented 3 years ago

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review. In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment. Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

Finally, remember to use https://discuss.ipfs.io if you just need general support.

shoce commented 3 years ago

After some investigation, I think it is more a question for github.com/multiformats/go-multihash codebase.

warpfork commented 3 years ago

We did recently add features to go-multihash to make it usable in a streaming fashion :)

Unfortunately, you're right, go-cid still doesn't have exposed APIs for streaming usage today. More work is needed on this.