Missing blob error when pushing to another Gitblit instance via pushhook

Hi!

We have the following situation: There is a Gitblit 1.6.2 instance at one of our external developer companies network, connected via ours via site-to-site VPN. Whenever they push, a pushhook is triggered to send the data to our Gitblit 1.7.1 Instance (Hosted in Openshift/Docker Container), but sometimes the push fails with the following error:

2016-11-04 07:05:49 [INFO ] *************** START CUSTOM PUSH *******************
2016-11-04 07:05:52 [ERROR] push failed
org.eclipse.jgit.api.errors.TransportException: https://<service_user>@<destination_gitblit>/r/<developer>/frontend/client.git: error occurred during unpacking on the remote end: error Missing blob 04f6e6754cf05b06d04aa5fea5534e642120620a
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:160)
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:79)
        at java_util_concurrent_Callable$call.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
        at pushhook.run(pushhook.groovy:28)
        at groovy.util.GroovyScriptEngine.run(GroovyScriptEngine.java:565)
        at com.gitblit.git.GitblitReceivePack.runGroovy(GitblitReceivePack.java:593)
        at com.gitblit.git.GitblitReceivePack.onPostReceive(GitblitReceivePack.java:362)
        at com.gitblit.git.PatchsetReceivePack.onPostReceive(PatchsetReceivePack.java:243)
        at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:236)
        at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:160)
        at com.gitblit.transport.ssh.git.Receive.runImpl(Receive.java:34)
        at com.gitblit.transport.ssh.git.BaseGitCommand.service(BaseGitCommand.java:72)
        at com.gitblit.transport.ssh.git.BaseGitCommand.access$100(BaseGitCommand.java:33)
        at com.gitblit.transport.ssh.git.BaseGitCommand$1.run(BaseGitCommand.java:59)
        at com.gitblit.transport.ssh.commands.BaseCommand$TaskThunk.run(BaseCommand.java:331)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
        at com.gitblit.utils.WorkQueue$Task.run(WorkQueue.java:332)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.eclipse.jgit.errors.TransportException: https://<service_user>@<destination_gitblit>/r/<developer>/frontend/client.git: error occurred during unpacking on the remote end: error Missing blob 04f6e6754cf05b06d04aa5fea5534e642120620a
        at org.eclipse.jgit.transport.BasePackPushConnection.readStatusReport(BasePackPushConnection.java:319)
        at org.eclipse.jgit.transport.BasePackPushConnection.doPush(BasePackPushConnection.java:200)
        at org.eclipse.jgit.transport.TransportHttp$SmartHttpPushConnection.doPush(TransportHttp.java:811)
        at org.eclipse.jgit.transport.BasePackPushConnection.push(BasePackPushConnection.java:152)
        at org.eclipse.jgit.transport.PushProcess.execute(PushProcess.java:165)
        at org.eclipse.jgit.transport.Transport.push(Transport.java:1177)
        at org.eclipse.jgit.api.PushCommand.call(PushCommand.java:156)
        ... 25 more
2016-11-04 07:05:52 [INFO ] *************** DONE CUSTOM PUSH*******************

The following Groovy script is used to push:


try {
  File baseFolder = new File("/opt/gitblit/data/git");
  File repoFolder = new File(baseFolder, repository.name);
  Git git = Git.open(repoFolder, FS.detect())
  git.push().setCredentialsProvider(new NetRCCredentialsProvider()).setRemote("origin").setThin(false).setPushAll().setPushTags().call();
  logger.info("push succeeded");
} catch (Exception e) {
  e.printStackTrace();
  logger.error("push failed", e);
}

logger.info("*************** DONE CUSTOM PUSH*******************")

We have also experienced missing tree and missing blob errors. I have checked the underlying git repositories with git fsck and not found any errors. I can also find the missing blob with a combination of git log, and git ls-tree.

I have found the JGit bug mentioned in #989 , and seen that the underlying JGit version has reverted the fix in the version used by Gitblit 1.7.1. I have set git.checkReferencedObjectsAreReachable = false, and pushed the source repository content with --mirror, but the issue still coming up...

Any ideas? Do you think we would experience similar issues with a mirror repository setup?

gitblit-org / gitblit

Missing blob error when pushing to another Gitblit instance via pushhook #1151