chanseokoh
commented
4 years ago Hi @SniXosha,
I've looked into our code to see where this might potentially fail. I can think of one hypothetical scenario. However, I'm still not convinced of my scenario, because it is unrelated to the level of concurrency; in my scenario, your registry should occasionally fail regardless of whether Jib pushes concurrently or serially. Therefore, my scenario is in conflict with your account that disabling parallel execution seemingly made it more stable. (In any case, my scenario is still not about the fault of Jib but the server not conforming to the HTTP standard.) (UPDATE: turned out to be a bug in Nexus.)
If that is not the case, I am highly suspicious of the server malfunctioning. In the past, we often did see confirmed cases of server issues in some popular registries when there is a very high level of concurrency (e.g., https://github.com/GoogleContainerTools/jib/issues/1986#issuecomment-536905823 and https://github.com/GoogleContainerTools/jib/issues/2013). From our track record, there's a high chance that this is a server issue. (BTW, Docker CLI by default doesn't allow the same level of high concurrency that Jib outputs.)
Anyways, it will become obvious once I can get low-level HTTP request/response logs. Unfortunately, Jib Gradle has an issue generating HTTP logs, so I need your help to build a Jib Gradle plugin from a patched source. Don't be scared, it's very easy to build Jib from source: (UPDATE: patching no longer needed with newer Jib versions.)
$ git clone https://github.com/GoogleContainerTools/jib.git
$ cd jib
$ ./gradlew :jib-gradle-plugin:installThis will build Jib 2.1.1-SNAPSHOT and install it into your local Maven repo (~/.m2/repository). Then configure your build script to use this SNAPSHOT as explained in https://github.com/GoogleContainerTools/jib/issues/2270#issuecomment-584177904.
Then follow these instructions to capture detailed HTTP logs, except passing -Djib.serialize=true. Using -Djib.serialize=true disables parallel pushes, so we don't want that. If everything is set correctly, you'll see logs like
Mar 31, 2020 9:55:52 AM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
HTTP/1.1 202 Accepted
Content-Length: 0
Docker-Distribution-Api-Version: registry/2.0
Docker-Upload-Uuid: 6292f0d7-93cb-4a8e-8336-78a1bf7febd2
Location: https://registry-1.docker.io/v2/francium25/test/blobs/uploads/6292f0d7-93cb-4a8e-8336-78a1bf7febd2?_state=6lvUYgy-Xw0N3L5SVgciJGhhUO928fGfHS35zpGIiJx7Ik5hbWUiOiJmcmFuY2l1bTI1L3Rlc3QiLCJVVUlEIjoiNjI5MmYwZDctOTNjYi00YThlLTgzMzYtNzhhMWJmN2ZlYmQyIiwiT2Zmc2V0Ijo2NTcyOTMsIlN0YXJ0ZWRBdCI6IjIwMjAtMDMtMzFUMTM6NTU6NTBaIn0%3D
Range: 0-657292
Date: Tue, 31 Mar 2020 13:55:52 GMT
Strict-Transport-Security: max-age=31536000
Mar 31, 2020 9:55:52 AM com.google.api.client.http.HttpRequest execute
CONFIG: -------------- REQUEST --------------
PUT https://registry-1.docker.io/v2/francium25/test/blobs/uploads/6292f0d7-93cb-4a8e-8336-78a1bf7febd2?_state=6lvUYgy-Xw0N3L5SVgciJGhhUO928fGfHS35zpGIiJx7Ik5hbWUiOiJmcmFuY2l1bTI1L3Rlc3QiLCJVVUlEIjoiNjI5MmYwZDctOTNjYi00YThlLTgzMzYtNzhhMWJmN2ZlYmQyIiwiT2Zmc2V0Ijo2NTcyOTMsIlN0YXJ0ZWRBdCI6IjIwMjAtMDMtMzFUMTM6NTU6NTBaIn0%3D&digest=sha256:24f0c933cbef83faee52f82c7f889c727b1ece5123b92d036c52fa865480f037
Accept:
Accept-Encoding: gzip
Authorization: <Not Logged>
User-Agent: jib 2.1.1-SNAPSHOT jib-maven-plugin Google-HTTP-Java-Client/1.34.0 (gzip)Please let me know once you get the network logs for the error.
SniXosha
johnburbridge
DmitryTen
stephan-roolvink
MrPowerGamerBR
KRK1ST
Environment:
Description of the issue: Jib fails with BLOB_UPLOAD_UNKNOWN error. I have a gradle project with about 15 submodules and I want to push images in our private registry. When I run jib for all sumbodules in my gradle project, sometimes this error occurs. Jib doesn't fail immediately, it's often able to create and push image for about 12-13 submodules and only then it crashes. If I rerun jib after waiting a few seconds, all the remaining submodules are processed successfully (currently, this is my workaround). I asked people responsible for our registry and they said that this is probably jib's problem, because they have not encountered this error using docker cli. They also suggested to disable parallel execution in gradle, which seemingly made jib more stable, but didn't solve the problem completely.
Expected behavior: Jib runs successfully regardless of number of modules/execution time without BLOB_UPLOAD_UNKNOWN error.
Steps to reproduce:
jib-gradle-pluginConfiguration:Log output:
Additional Information:
I asked people responsible for our registry and they said that this is probably jib's problem, because they have not encountered this error using docker cli. They also suggested to disable parallel execution in gradle, which seemingly made jib more stable, but didn't solve the problem completely.