git-lfs / git-lfs

Git extension for versioning large files
https://git-lfs.com
Other
13.05k stars 2.05k forks source link

git clone fails with "fatal: Out of memory, realloc failed", but can "git lfs pull" with "- missing --add option", and then "git add ." to get HEAD (maybe) #5340

Closed Infro closed 1 year ago

Infro commented 1 year ago

Overview: git clone fails with "fatal: Out of memory, realloc failed", but can "git lfs pull" with "- missing --add option", and then "git add ." to get HEAD (as far as I can tell.) Considering I am seeing things state STDOUT, my first guess is that this is somehow because my shell for this is: cmd->powershell->mingw'sgit which has caused different types of problems, like adding prefix characters to streams or not: 3.powershell.stupidity.txt

0.PSVersionTable.txt 1.git.clone_restore_and_config_packthings.txt 2.git.lfs.pull_and_git.add...txt

I don't know, but the SHA256 hash of pytorch_model.bin is what it's suppose to be, so I don't have any issues that need addressed, and hopefully I've provided you with the info that might be helpful, unless I missed some FAQ or something stating not to use powershell or something. Hope this helps and hope you are having a great day :) !

bk2204 commented 1 year ago

Hey,

This is not because of using PowerShell, but because Git will by default allocate sufficient memory to smudge the entire file into memory, and your system doesn't have enough memory. That's entirely a Git bug and Git LFS can't change that in any way.

To work around it, you can use git lfs install --skip-smudge and use git lfs pull to check out the files you want. The message with cannot add to the index is because Git failed to check out the files and so it didn't write an index, so you'll need to either re-clone the repository after having run git lfs install --skip-smudge or try to re-create it with git add . first. Again, this isn't an issue with Git LFS, since it can't control whether Git creates an index or not.