search

richardschneider / net-ipfs-engine

IPFS Core API implementation in .Net
MIT License
106 stars 48 forks source link

FileSystem.AddFileAsync generates unexpected hashes for larger files #125

Closed Jonax closed 5 years ago

Jonax commented 5 years ago

With a default configuration, FileSystem.AddFileAsync() appears to return a different hash than expected when the file in question is above a specific size threshold.

Based on local trial & error, this threshold appears to be exactly 43.5 MB (45613056 bytes) - Any file exactly this size or smaller should return a hash identical to an equivalent add -n operation in go-ipfs. Files larger than this will result in a different hash than go-ipfs provides.

This behaviour doesn't seem to trigger test AddFile_Large, since star_trails.mp4 is below this threshold (37.8MB). Replacing the file with a file larger than 44MB should cause the test to error on the hash assert in the first half of the test.

richardschneider commented 5 years ago

Thanks, that is a very good bug report. I'll investigate.

richardschneider commented 5 years ago

@Jonax What version of go-ipfs are you testing against?

Jonax commented 5 years ago

0.4.21, Windows x64 (go-ipfs_v0.4.21_windows-amd64.zip)

richardschneider commented 5 years ago

The issue is with generating a balanced tree. Need to change net-ipfs-engine to only generate up to 174 blocks per unixfs node.

richardschneider commented 5 years ago

@Jonax Sorry it took so long to get this working. I've added the test AddFile_Larger. I'll make a release in a few hours.

richardschneider commented 5 years ago

Fixed in release v0.11.0

Jonax commented 5 years ago

Tested with local live files of increasing 2^n file sizes up to 1GB. All passed with expected hashes.

I'd call this bug well & truly fixed. Nice job.