Issue with git push-ing to Gerrit

[GoogleCodeExporter]

************************************************************
***** NOTE: THIS BUG TRACKER IS FOR GERRIT CODE REVIEW *****
***** DO NOT SUBMIT BUGS FOR CHROME, ANDROID, INTERNAL *****
***** ISSUES WITH YOUR COMPANY'S GERRIT SETUP, ETC.    *****
***** THOSE ISSUE BELONG IN DIFFERENT ISSUE TRACKERS!  *****
************************************************************

Affected Version: 2.4.2

What steps will reproduce the problem?
1. git push
2.
3.

What is the expected output? What do you see instead?

We expect successfull push, instead we get:
o$ git push
Counting objects: 5389, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (1801/1801), done.
Writing objects: 100% (5190/5190), 2.27 MiB | 15 KiB/s, done.
Total 5190 (delta 3401), reused 5006 (delta 3275)
remote: Resolving deltas: 100% (3401/3401)
error: unpack failed: error Missing tree 
bfe8d24e0232d8e8969fdadddea7b81aca863664
fatal: Unpack error, check server log
To ssh://**********:29418/********.git
 ! [remote rejected] HEAD -> refs/for/cmedia_3.0.9 (n/a (unpacker error))
error: failed to push some refs to 'ssh://************8:29418/ticketco.git'

Please provide any additional information below.

Original issue reported on code.google.com by YuriyPad...@gmail.com on 26 Sep 2012 at 3:21

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi guys, we have the same issue with 2.5.1.

Direct push or push to review or draft fails sometimes for some developers with 
error:

$ git review master
Counting objects: 152, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (66/66), done.
Writing objects: 100% (90/90), 15.94 KiB, done.
Total 90 (delta 33), reused 30 (delta 13)
remote: Resolving deltas: 100% (33/33)
error: unpack failed: error Missing tree 
42d769150decc8ee96bd9eb7afacc01fe14ffc55
fatal: Unpack error, check server log
To ssh://....
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://....'

In logs/error_log

Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree 
42d769150decc8ee96bd9eb7afacc01fe14ffc55
        at org.eclipse.jgit.transport.BaseReceivePack.checkConnectivity(BaseReceivePack.java:996)
        at org.eclipse.jgit.transport.BaseReceivePack.receivePackAndCheckConnectivity(BaseReceivePack.java:756)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:167)
        ... 15 more

Missing tree sha was different, but every time git ls-tree showed not-corrupted 
existed tree.
git fsck showed danglings only.

After investigation I found if I set Read right on /ref/* for Registered Users, 
issue would disappear.

What is the link between Read right on /ref/* for Registered Users on 
time-to-time missing tree exception? 

Original comment by aleksey....@gmail.com on 20 Mar 2013 at 12:10

[GoogleCodeExporter]

Hello guys,
we are exiriencing the same issues with 2.5.1 and 2.5.2.

Aleksey's workaround hasn't helped.

Any suggestions on cause of this issue and possible workarounds?

Original comment by presich....@gmail.com on 17 Apr 2013 at 11:52

[GoogleCodeExporter]

Hello again,
we started to receive such errors very often.

Push fails with such errors (even push force!! through the Gerrit):
Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing tree
or Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing blob

Any thoughts what could we check?

Those missing trees and blobs are really exist in both local and gerrit 
repository and i can look in them by: git show

I mentioned that one of missing trees was in Packed objects, but can't say for 
all of them.

Also I can suggest that such errors come up with commits, that was not uploaded 
to the Gerrit review previously.

The only way we were able to push changes, is to push them through the pure Git 
on port 22.

Original comment by presich....@gmail.com on 18 Apr 2013 at 3:11

[GoogleCodeExporter]

Good day. I've tried to configure local Gerrit for evaluation purposes and got 
the same problem. Seems that there is some problem in git protocol 
implementation.
I've checked on git 1.8.3.2 and everything worked fine. When I've updated to 
1.8.4.3 problems on commit changes were similar to the one in this topic:

k.stolyarov@DGIS-184 ~/test $ git clone 
ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
Cloning into 'v2'...
The authenticity of host '[dgis-184.dvlp.2gis.local]:29418 
([10.54.200.106]:29418)' can't be established.
RSA key fingerprint is c7:4d:d5:23:d1:b3:02:24:c6:21:2c:1d:3e:4e:de:8b.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 
'[dgis-184.dvlp.2gis.local]:29418,[10.54.200.106]:29418' (RSA) to the list of 
known hosts.
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
remote: Counting objects: 2, done
remote: Finding sources: 100% (2/2)
remote: Total 2 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (2/2), done.
skjdhjksdf
Checking connectivity... done
k.stolyarov@DGIS-184 ~/test $ cd v2
k.stolyarov@DGIS-184 ~/test/v2 $ vim 1 
k.stolyarov@DGIS-184 ~/test/v2 $ scp -p -P 29418 
a.baskanov@dgis-184.dvlp.2gis.local:hooks/commit-msg .git/hooks/
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
commit-msg                                                                      
                                                                           100% 
4367     4.3KB/s   4.3KB/s   00:00    
k.stolyarov@DGIS-184 ~/test/v2 $ git add 1
k.stolyarov@DGIS-184 ~/test/v2 $ git commit
[master dd26813] sdfhsdf
 1 file changed, 1 insertion(+)
 create mode 100644 1
k.stolyarov@DGIS-184 ~/test/v2 $ git push origin HEAD:refs/for/master
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
Counting objects: 4, done.
Writing objects: 100% (3/3), 269 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
remote: Processing changes: new: 1, refs: 1, done    
remote: 
remote: New Changes:
remote:   http://dgis-184.dvlp.2gis.local:8080/1
remote: 
To ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
 * [new branch]      HEAD -> refs/for/master
k.stolyarov@DGIS-184 ~/test/v2 $ vim  2
k.stolyarov@DGIS-184 ~/test/v2 $ git add 2
k.stolyarov@DGIS-184 ~/test/v2 $ git commit -a  --amend
[master 6026dbb] sdfhsdf
 2 files changed, 2 insertions(+)
 create mode 100644 1
 create mode 100644 2
k.stolyarov@DGIS-184 ~/test/v2 $ git push origin HEAD:refs/for/master
Enter passphrase for key '/home/k.stolyarov/.ssh/id_rsa': 
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 302 bytes | 0 bytes/s, done.
Total 3 (delta 0), reused 0 (delta 0)
error: unpack failed: error Missing blob 
e89ef0eb5feb4db1e4c757374b83ed1d8e6081ca
fatal: Unpack error, check server log
To ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 
'ssh://a.baskanov@dgis-184.dvlp.2gis.local:29418/v2'

I've checked this on both Gerrit 2.7 and 2.8-rc2 versions from two different 
linux machines (Gentoo and Ubunto builds). Latest git version bakes review 
updates.

Hope that this information will help you to fix the issue.

Original comment by KNibb...@gmail.com on 20 Nov 2013 at 10:20

[GoogleCodeExporter]

This is much more than a minor issue as it is leading us to have corrupted 
repos at this time. Can someone please take a deeper look into why this might 
be happening?

Original comment by joseph.r...@gmail.com on 21 Nov 2013 at 9:13

[GoogleCodeExporter]

Issue 2296, which seems to duplicate this issue, suggests to set 
receive.checkReferencedObjectsAreReachable = false (available since Gerrit 2.6) 
to work around this.

Original comment by sschuberth on 29 Jan 2014 at 8:54

[GoogleCodeExporter]

Why in the world would one turn that off - it sounds like the suggestion is to 
allow what looks like corruption in the reference tree...?

Original comment by joseph.r...@gmail.com on 29 Jan 2014 at 12:26

[GoogleCodeExporter]

Just consult the Gerrit 2.6 release notes [1] for why it would make sense to 
turn off that option:

"New config option receive.checkReferencedObjectsAreReachable

If set to true, Gerrit will validate that all referenced objects that are not 
included in the received pack are reachable by the user.

Carrying out this check on Git repositories with many refs and commits can be a 
very CPU-heavy operation. For non public Gerrit servers it may make sense to 
disable this check, which is now possible."

[1] 
http://gerrit-documentation.googlecode.com/svn-history/r63/ReleaseNotes/ReleaseN
otes-2.6.html

Original comment by sschuberth on 29 Jan 2014 at 12:48

[GoogleCodeExporter]

Also, I don't believe turning off that option allows for corruption in the 
reference tree. It's just that Gerrit's JGit implementation cannot handle [1] 
yet (see issue 2296), which is why you get this bogus error with recent Git 
clients.

[1] https://github.com/git/git/commit/fbd4a7036dfa71ec89e7c441cef1ac9aaa59a315

Original comment by sschuberth on 29 Jan 2014 at 12:52

[GoogleCodeExporter]

Is there any work around for this from the client side? If say I can't upgrade 
the version of gerrit being used, or change server settings? Maybe there's an 
option to git I could set to not do the optimization that sends fewer trees?

Original comment by onlyn...@gmail.com on 23 Apr 2014 at 8:07

[GoogleCodeExporter]

Per a coworker, sounds like specifying --no-thin on the git client command line 
avoids this optimization without having to set 
receive.checkReferencedObjectsAreReachable = false on Gerrit.

Original comment by mpe...@gmail.com on 10 Jun 2014 at 6:25

[GoogleCodeExporter]

JGit fix https://git.eclipse.org/r/#/c/31081/ was merged as 
c4797fe98655b3d52d0a90ba44fce6e053db3b8b

Original comment by matthias...@gmail.com on 28 Aug 2014 at 10:31

[GoogleCodeExporter]

Issue 2296 has been merged into this issue.

Original comment by jrn@google.com on 5 Sep 2014 at 4:06

[GoogleCodeExporter]

https://gerrit-review.googlesource.com/#/c/59935/

Original comment by david.pu...@sonymobile.com on 9 Sep 2014 at 3:52

• Changed state: ChangeUnderReview

[GoogleCodeExporter]

I'm using gerrit 2.9.1, which already has jgit update (change 59935), but I 
still can reproduce the missing tree error. It seems to happen only for pushes 
where *only commit message is changed* without any other commit modifications 
o.O

$ echo 'a' > a
$ git commit -a -m 'msg'
$ git push origin HEAD:refs/for/master
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: new: 1, refs: 1, done    
remote: New Changes:
remote:   https://gerrit/2202

$ echo 'b' > a
$ git commit -a --amend
/ possibly some changes in msg in editor/
$ git push origin HEAD:refs/for/master
remote: Resolving deltas: 100% (1/1)
remote: Processing changes: updated: 1, refs: 1, done    
remote: Updated Changes:
remote:   https://gerrit.sio2project.mimuw.edu.pl/2202

$ git commit -a --amend
/ changes in msg/
$ git push origin HEAD:refs/for/master 
error: unpack failed: error Missing tree 
a9596a3d9e2f4fa81912124d3dacd58113f11e85
fatal: Unpack error, check server log
To ssh://gerrit/project
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://gerrit/project'

This is not too common to change only commit msg, also it can be done from 
gerrit web ui, but still is annoying.

Can someone try to reproduce and confirm it?
Thanks!

Original comment by mdemp...@gmail.com on 8 Oct 2014 at 9:53

[GoogleCodeExporter]

How did you determine that 2.9.1 has 59935? 59935 was uploaded and merged to 
master. The only JGit update I see on stable-2.9 is from May: 
https://gerrit-review.googlesource.com/#/q/status:merged+project:gerrit+branch:s
table-2.9+message:%22JGit%22

Original comment by nas...@codeaurora.org on 8 Oct 2014 at 10:18

[GoogleCodeExporter]

Oops, right, I thought I've seen it in some changelog, but apparently it's not 
there.
Sorry!

Original comment by mdemp...@gmail.com on 8 Oct 2014 at 10:22

[GoogleCodeExporter]

Is there gonna be a new release soon? We're plagued by this bug in various 
teams.
Or is installing 2.10.rc0 a good alternative to waiting for the release?

Original comment by silvio.h...@gmail.com on 12 Nov 2014 at 11:46

[GoogleCodeExporter]

I've been hitting this issue for quite a while and it still exists in 2.9.1. A 
simple case to duplicate is to push a change, merge, and try to push the git 
reverted change. I've been able to reproduce this in more complicated cases 
where the commit tree ends up being the same as one that was pushed prior.

Original comment by jlst...@gmail.com on 14 Nov 2014 at 3:07

[GoogleCodeExporter]

If you are using git 1.8.4.2 or later, there is some incompatibility between 
jgit 2.3 shipped with Gerrit 2.7 and jgit 3.1 shipped with Gerrit 2.8 and this 
version of git due to this enhancement, i.e. 
https://github.com/git/git/commit/fbd4a7036dfa71ec89e7c441cef1ac9aaa59a315

With this enhancement, if git finds out that the tree sha1 already exists in 
the server, it wont send it again, but gerrit wants to search the tree sha1 
associated with the commit sha1 in the uploaded pack. There is a bug report on 
jgit library for this issue at: 
https://bugs.eclipse.org/bugs/show_bug.cgi?id=426044. This was reported to be 
fixed for jgit 3.4 which is shipped with Gerrit 2.9, but there is still the 
same behavior reported with Gerrit 2.9.1.

A proposed workaround for this issue is to go to the local branch you are 
trying to push, and pull from origin and then retry pushing, or use older 
version of git to push such branches. 

Original comment by bassem.rabil on 2 Dec 2014 at 8:30

[GoogleCodeExporter]

Fix submitted and should be part of 2.10-rc1.

Original comment by jrn@google.com on 3 Dec 2014 at 3:22

• Changed state: Submitted

[GoogleCodeExporter]

(The fix I'm referring to is https://gerrit-review.googlesource.com/59935.)

Original comment by jrn@google.com on 3 Dec 2014 at 3:22

[GoogleCodeExporter]

Can anyone verify this is fixed in 2.10?

Original comment by jlst...@gmail.com on 12 Feb 2015 at 4:26

[GoogleCodeExporter]

Got this error in 2.9.3. After update to 2.10 works.

Original comment by fraczwoj...@gmail.com on 12 Apr 2015 at 10:26