search

GoogleContainerTools / jib

🏗 Build container images for your Java applications.
Apache License 2.0
13.58k stars 1.43k forks source link

Duplicate put requests to registry trigger errors #4291

Open jtama opened 1 month ago

jtama commented 1 month ago

Environment:

Description of the issue: During our ci pipeline, we have an error occuring randomly. Jib triggers the same patch request which generates an error.

We know that the first patch takes a bit long to (1m12s) successfully process, but in the mean time Jib sends a second patch requests. This second request is the one that generates an error.

Expected behavior:

Should ends up fine.

Steps to reproduce:

Unable to reproduce surely the error.

Log output:

Mon, 01 Jul 2024 09:05:44 GMT Caused by: com.google.cloud.tools.jib.http.ResponseException: 404 Not Found
Mon, 01 Jul 2024 09:05:44 GMT PATCH https://.................../blobs/uploads/f7b334b6-be1f-4bfc-bd12-ca739c994a9b?_state=Z4RE2P18shUE5tW_Pk_XmXKYTOAtUlOoUA_G5isOd3l7Ik5hbWUiOiJwcm92b2x5L2RhdGEtdmlydCIsIlVVSUQiOiJmN2IzMzRiNi1iZTFmLTRiZmMtYmQxMi1jYTczOWM5OTRhOWIiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDctMDFUMDk6MDQ6MTUuMTQ1NTc4NzcxWiJ9
Mon, 01 Jul 2024 09:05:44 GMT {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"blob upload invalid"}]}
Mon, 01 Jul 2024 09:05:44 GMT
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:355)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:266)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:138)
Mon, 01 Jul 2024 09:05:44 GMT ... 48 more
Mon, 01 Jul 2024 09:05:44 GMT Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
Mon, 01 Jul 2024 09:05:44 GMT PATCH https://........................./blobs/uploads/f7b334b6-be1f-4bfc-bd12-ca739c994a9b?_state=Z4RE2P18shUE5tW_Pk_XmXKYTOAtUlOoUA_G5isOd3l7Ik5hbWUiOiJwcm92b2x5L2RhdGEtdmlydCIsIlVVSUQiOiJmN2IzMzRiNi1iZTFmLTRiZmMtYmQxMi1jYTczOWM5OTRhOWIiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDctMDFUMDk6MDQ6MTUuMTQ1NTc4NzcxWiJ9
Mon, 01 Jul 2024 09:05:44 GMT {"errors":[{"code":"BLOB_UPLOAD_INVALID","message":"blob upload invalid"}]}
Mon, 01 Jul 2024 09:05:44 GMT
Mon, 01 Jul 2024 09:05:44 GMT at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
Mon, 01 Jul 2024 09:05:44 GMT at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
Mon, 01 Jul 2024 09:05:44 GMT ... 50 more

Additional Information: The registry is a privately managed harbor

chanseokoh commented 1 month ago

I don't think they are duplicate PATCH requests. It's just that Jib does concurrent blob uploads. I think for whatever reason, the harbor registry sometimes fails to handle concurrent uploads correctly. I guess retrying Jib in a loop or even completely disabling concurrency in Jib (slow, of course, particularly bad for first-time build and uploads) would be a workaround.

ddittmar commented 1 month ago

I don't think they are duplicate PATCH requests. It's just that Jib does concurrent blob uploads. I think for whatever reason, the harbor registry sometimes fails to handle concurrent uploads correctly. I guess retrying Jib in a loop or even disabling concurrency in Jib (slow, of course, particularly for initial uploads) would be a workaround.

Hi @chanseokoh ... I think this looks like the problem we have too. I could not find howto "disable concurrency" in Jib.

chanseokoh commented 1 month ago

@ddittmar to disable Jib concurrency, set the system property jib.serialize to false, like -Djib.serialize=false on the command line. See this example. Note it's a system property only; no Maven or Gradle configuration for this Jib system property.