Unknown blob when pushing to Artifactory , with s3 as the storage

[RahmanBadru]

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Version of Helm and Kubernetes: 3.12.3 and V1.29

Which chart: Artifactory-107.90.8

Which product license (Enterprise/Pro/oss): Pro

JFrog support reference (if already raised with support team): N/A

What happened: Pushing images to our artifactory instance, throws the error unknown blob after pushing images

the error in the logs show

Deploy to 'docker-local:docker-local/mongo/_uploads/6d562ef3-cab2-4b8d-b60e-f4d5b6a05c2d' Content-Length: 0 (estimation) artificial: false

Failed while saving temp file /opt/jfrog/artifactory/var/data/artifactory/filestore/_pre/dbRecord870694586599854700-35d5a484b17b6f0a-docker-local.bin
org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: Failed to read stream: java.io.EOFException
Caused by: org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: java.io.EOFException
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.EOFException
Caused by: java.io.EOFException: null
2024-09-12T08:20:21.331Z [jfrt ] [ERROR] [35d5a484b17b6f0a] [c.e.m.GlobalExceptionMapper:51] [ttp-nio-8081-exec-97] - org.jfrog.storage.StorageException: Failure during saveToTempFileAndUpload
org.artifactory.storage.StorageException: org.jfrog.storage.StorageException: Failure during saveToTempFileAndUpload
        at org.artifactory.sh.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:140)
        at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:2649)
        at jdk.internal.reflect.GeneratedMethodAccessor359.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
        at jdk.proxy2/jdk.proxy2.$Proxy74.saveResource(Unknown Source)
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:795)
        at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:766)
        at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:629)
        at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:618)
        at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:597)
        at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:335)
        at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:290)
        at org.artifactory.engine.UploadServiceImpl.processUpload(UploadServiceImpl.java:228)
        at org.artifactory.api.request.GracefulShutdownAware.track(GracefulShutdownAware.java:36)
        at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:210)
        at jdk.internal.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)

What you expected to happen: the image pushes successfully

How to reproduce it (as minimally and precisely as possible): docker push artifactory-url/image-name

Anything else we need to know: No support, so we pretty much have to fix this on our own

[arikogan]

Hi @RahmanBadru. Can you please enable debug logging for org.jfrog.type.s3? This way we may be able to have better visibility of what may have caused this. Thanks.

<logger name="org.jfrog.type.s3" level="debug"/>

[RahmanBadru]

Here is what i can see

Failed while saving temp file /opt/jfrog/artifactory/var/data/artifactory/filestore/_pre/dbRecord1770488452772219306-40f476e121806379-docker-local.bin
2024-09-12T10:54:35.729Z [jfrt ] [DEBUG] [40f476e121806379] [.j.t.s.S3AwsBinaryProvider:283] [http-nio-8081-exec-3] -
org.jfrog.storage.binstore.ifc.ClientInputStreamException: Failed to read stream: Failed to read stream: java.io.EOFException
        at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:36)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1487)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1107)
        at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1460)
        at org.apache.commons.io.IOUtils.copy(IOUtils.java:1085)
        at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1026)
        at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:934)
        at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamToTempFile(FilePersistenceHelper.java:52)
        at org.jfrog.type.s3.S3AwsBinaryProvider.saveToTempFileAndUpload(S3AwsBinaryProvider.java:269)
        at org.jfrog.type.s3.S3AwsBinaryProvider.addStream(S3AwsBinaryProvider.java:236)
        at org.jfrog.storage.binstore.providers.cachefs.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:225)
        at org.jfrog.storage.binstore.providers.federated.FederatedBinaryProvider.addStream(FederatedBinaryProvider.java:360)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addStreamToBinaryProviderIfNeeded(BinaryServiceImpl.java:1342)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:435)
        at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:422)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
--
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:312)
        at org.apache.catalina.connector.InputBuffer.checkByteBufferEof(InputBuffer.java:615)
        at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:358)
        at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:132)
        at org.glassfish.jersey.message.internal.EntityInputStream.read(EntityInputStream.java:79)
        at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:205)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:343)
        at org.jfrog.storage.binstore.common.Sha1Sha2Md5ChecksumInputStream.read(Sha1Sha2Md5ChecksumInputStream.java:134)
        at org.jfrog.storage.binstore.common.Sha1Sha2Md5ChecksumInputStream.read(Sha1Sha2Md5ChecksumInputStream.java:118)
        at org.jfrog.storage.binstore.ifc.ClientStream.read(ClientStream.java:34)
        ... 145 common frames omitted
Caused by: java.io.EOFException: null
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.fillReadBuffer(NioEndpoint.java:1320)
        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.read(NioEndpoint.java:1231)
        at org.apache.coyote.http11.Http11InputBuffer.fill(Http11InputBuffer.java:789)
        at org.apache.coyote.http11.Http11InputBuffer.access$400(Http11InputBuffer.java:41)
        at org.apache.coyote.http11.Http11InputBuffer$SocketInputBuffer.doRead(Http11InputBuffer.java:1195)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.readBytes(ChunkedInputFilter.java:254)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.fill(ChunkedInputFilter.java:295)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.parseChunkHeader(ChunkedInputFilter.java:328)
        at org.apache.coyote.http11.filters.ChunkedInputFilter.doRead(ChunkedInputFilter.java:136)
        at org.apache.coyote.http11.Http11InputBuffer.doRead(Http11InputBuffer.java:244)
        at org.apache.coyote.Request.doRead(Request.java:626)
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:303)
        ... 156 common frames omitted

[arikogan]

Thank you for this. It's not related to the Helm Charts but seems to be something with your client interrupting the connection. Please contact our Support.

[RahmanBadru]

Hi, currently dont have access to support, so would appreciate help

[arikogan]

@RahmanBadru, I replied on your Stack Overflow question, let's correspond there.

[RahmanBadru]

Hi @ArielKogan, I have responded

On Sun, Sep 22, 2024 at 14:09 Ariel Kogan @.***> wrote:

@RahmanBadru https://github.com/RahmanBadru, I replied on your Stack Overflow question, let's correspond there.

— Reply to this email directly, view it on GitHub https://github.com/jfrog/charts/issues/1921#issuecomment-2366781587, or unsubscribe https://github.com/notifications/unsubscribe-auth/AI5RREGKATY6TJSZWBHVUTTZX26P3AVCNFSM6AAAAABOCUELDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRWG44DCNJYG4 . You are receiving this because you were mentioned.Message ID: @.***>