search

happybeing / safenetwork-fuse

SAFE Drive: access SAFE storage using the file system of Windows, Mac OS and Linux
GNU General Public License v3.0
12 stars 5 forks source link

Push to headless git repo fails: premature end of pack file, 65 bytes missing #20

Closed happybeing closed 5 years ago

happybeing commented 5 years ago

The bug here is probably in SafenetworkJS, but SAFE Drive is needed to investigate and fix it, so filing here.

To reproduce you need to be set up for development on SAFE Network using SAFE Browser.

OS Ubuntu 18.10 SAFE Browser 11.1 git 2.19.1 safenetwork-fuse master branch safenetworkjs master branch

To reproduce the error:

1) Make a copy of the safenetworkjs git repository in sjs.git just for testing:

cp -r safenetworkjs sjs.git

2) Start SAFE Browser (v11.1 or later) dev build and create / sign into an account:

safe-browser --debug

3) Start SAFE Drive with NODE_ENV=test to select mock SAFE Network, SAFENETWORKJS_TESTS=testing to create a test container, and DEBUG= to generate console debug output, and 'Accept' any auth requests in the browser. For example:

NODE_ENV=test SAFENETWORKJS_TESTS=testing DEBUG=safe-fuse:ops*,safe-fuse:stub*,safenetworkjs:file* node --inspect-brk bin.js

Note: --inspect-brk (or --inspect) enables you to connect with the Chromium/Chrome debugger by visiting chrome://inspect/#devices and clicking 'Open dedicated DevTools for Node'.

For details of how to setup your debug environment, see Development in the README.

4) Check your SAFE Drive is ready:

$ ls ~/SAFE
_public _publicNames

5) Initialise a bare headless git repo in the test container, exactly as follows: 

cd ~/SAFE/_public/tests/data1
git init --bare blah

6) Inside sjs.git execute the following commands in order:

cd <your path to sjs.git>
git remote remove origin
git remote add origin ~/SAFE/_public/tests/data1/blah
git push origin master

The git push output should be similar to this:

Enumerating objects: 373, done.
Counting objects: 100% (373/373), done.
Delta compression using up to 8 threads
Compressing objects: 100% (371/371), done.
Writing objects: 100% (373/373), 192.45 KiB | 26.00 KiB/s, done.
Total 373 (delta 255), reused 0 (delta 0)
remote: 
warning: die() called many times. Recursion error or racy threaded death!
remote: fatal: premature end of pack file, 65 bytes missing
remote: warning: die() called many times. Recursion error or racy threaded death!
error: remote unpack failed: index-pack abnormal exit
To /home/mrh/SAFE/_public/tests/data1/blah
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to '/home/mrh/SAFE/_public/tests/data1/blah'

The message remote: fatal: premature end of pack file, 65 bytes missing occurs because inside git (built from source code), while git is parsing the 'pack' file xpread() in git/index-pack.c / unpack_data(), prematurely returns zero bytes. Examining the SAFE Drive console output appears to show no errors, and that the last readFile() returned data as expected. So it isn't clear if the cause of problem is in SafenetworkJs, fuse-bindings or git.

Note: the above error can be avoided by setting a breakpoint in safenetworkjs/src/nfs-files.js openFile() in the else block ("// Handle failure to open file", about here). Just hitting the breakpoint and restarting execution avoids this error, although a different error occurs soon afterwards. The second failure is a different error, so the effect of pausing at the breakpoint for a few seconds may be a clue to the first error above.

Here's what the output looks like if you set the breakpoint, and restart immediately:

$ git push origin master
Enumerating objects: 373, done.
Counting objects: 100% (373/373), done.
Delta compression using up to 8 threads
Compressing objects: 100% (371/371), done.
Writing objects: 100% (373/373), 188.27 KiB | 56.00 KiB/s, done.
Total 373 (delta 255), reused 0 (delta 0)
remote: Resolving deltas: 100% (255/255), done.
remote: error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
remote: error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
remote: error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
remote: error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
remote: fatal: bad object 7190c86957ae10cb437fc1ad05a34932326e4dff
error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
error: packfile /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-2SiG0E/pack/pack-993ba7c0c3854f7cdbbc976ad9f6db3e9ce9b138.pack does not match index
fatal: bad object 7190c86957ae10cb437fc1ad05a34932326e4dff
To /home/mrh/SAFE/_public/tests/data1/blah
 ! [remote rejected] master -> master (missing necessary objects)
error: failed to push some refs to '/home/mrh/SAFE/_public/tests/data1/blah'
davidpbrown commented 5 years ago

Monkey see; monkey do.. not understanding all that but following the steps, getting a different output that looks alive.

$ git push origin master
Counting objects: 571, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (161/161), done.
Writing objects: 100% (571/571), 296.07 KiB | 0 bytes/s, done.
Total 571 (delta 408), reused 571 (delta 408)
To /home/safe/SAFE/_public/tests/data1/blah
 * [new branch]      master -> master

Referee? "No foul" ?no bug

happybeing commented 5 years ago

Very interesting, thanks this could be useful.

What OS / version and git version are you using. The messages ('Counting objects') and the numbers in your output are quite different, so maybe a different build of git.

Did you follow the instructions exactly - commands, environment variables etc? Just wondering about any explanations for the discrepancies in the output you have.

I really appreciate you diving in here David, thanks 😄

happybeing commented 5 years ago

This bug is still live - David's report was without SAFE Drive and created a repo on his local drive.

Closing in favour of issue #10 in new SAFE Drive repo as this repo is no longer in use.