search

szwacz / fs-jetpack

Better file system API for Node.js
MIT License
777 stars 41 forks source link

inspectTree with checksum is not working on large files #101

Open GuillaumeHemmen opened 3 years ago

GuillaumeHemmen commented 3 years ago

Hello,

I try to use the inspectTree function (inspectTree(pathToFolder, { checksum: 'md5' })) with the checksum set in the option. The folder I'm trying to inspect contains a few large files (up to ±4GB).

When doing so, I get the following error:

RangeError [ERR_FS_FILE_TOO_LARGE]: File size (4008774437) is greater than 2 GB
    at tryCreateBuffer (fs.js:352:13)
    at Object.readFileSync (fs.js:388:14)
    at fileChecksum (<PATHTOPROJECT>/node_modules/fs-jetpack/lib/inspect.js:87:19)

I would expect the function to work without having this error.

szwacz commented 3 years ago

The problem here might be that synchronous operations need to load everything into memory, because need to do everything in one go. Does using inspectTreeAsync lead to the same error?

GuillaumeHemmen commented 3 years ago

Hello @szwacz,

Using inspectTreeAsync solved the issue.

Thanks a lot!

szwacz commented 3 years ago

Good... but bad at the same time :) I'll leave this issue open to investigate if this can be somehow fixed for sync operations.