search

jfrog / charts

JFrog official Helm Charts
https://jfrog.com/integration/helm-repository/
Apache License 2.0
256 stars 447 forks source link

new splitServicesToContainers default configuration breaks azure storage blob #1797

Closed EvertonSA closed 1 year ago

EvertonSA commented 1 year ago

After upgrading our production instance to new jfrog helm chart, we notice that all services are deployed correctly, but jfrog stop working when uploading to azure storage account.

setting splitServicesToContainers to false make it work again.

I want to have splitServicesToContainers to true, because it's the default. Can you help me to fix? The open ticket to support is https://support.jfrog.com/s/tickets/5006900006gvRe5/migrating-from-75911-to-7637-breaks-azure-storage-account

The application logs indicate an error on the following:

2023-07-25T07:19:56.938Z [jfrt ] [INFO ] [517dd42dedac829f] [o.a.e.UploadServiceImpl:504   ] [http-nio-8081-exec-1] - Deploy to 'team-docker-local:bitnami/redis/_uploads/470a204b-0a5c-409d-916e-8374b4b83173' Content-Length: 1 (estimation) artificial: false
2023-07-25T07:19:56.938Z [jfrt ] [DEBUG] [517dd42dedac829f] [a.a.j.JFConnectServiceImpl:181] [http-nio-8081-exec-1] - Verifying if the entitlement "cold_storage" is available
2023-07-25T07:19:56.938Z [jfrt ] [DEBUG] [517dd42dedac829f] [o.a.a.AddonsManagerImpl:416   ] [http-nio-8081-exec-1] - The entitlement server return "false" for the entitlement "COLD_STORAGE"
2023-07-25T07:19:56.938Z [jfrt ] [DEBUG] [517dd42dedac829f] [lobStorageV2BinaryProvider:276] [http-nio-8081-exec-1] - Uploading to Azure by writing to temp file and upload it
2023-07-25T07:19:56.938Z [jfrt ] [DEBUG] [517dd42dedac829f] [b.c.SavedToFileInputStream:136] [http-nio-8081-exec-1] - Deleting temp file file /var/opt/jfrog/artifactory/data/cache/_pre/dbRecord8923238832558008007-517dd42dedac829f-team-docker-local.bin
2023-07-25T07:19:56.939Z [jfrt ] [DEBUG] [517dd42dedac829f] [s.d.s.DbTransactionManager:853] [http-nio-8081-exec-1] - Should roll back transaction but cannot - no transaction available
2023-07-25T07:19:56.939Z [jfrt ] [DEBUG] [517dd42dedac829f] [o.a.e.UploadServiceImpl:205   ] [http-nio-8081-exec-1] - Process 517dd42dedac829f tracking stopped
2023-07-25T07:19:56.939Z [jfrt ] [DEBUG] [517dd42dedac829f] [AggregatingEventServiceImpl:93] [http-nio-8081-exec-1] - An exception occurred within the invocation remove remote address 1REDACT6 and the events related to it.
2023-07-25T07:19:56.939Z [jfrt ] [ERROR] [517dd42dedac829f] [c.e.m.GlobalExceptionMapper:52] [http-nio-8081-exec-1] - org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
2023-07-25T07:19:56.939Z [jfrt ] [DEBUG] [517dd42dedac829f] [c.e.m.GlobalExceptionMapper:53] [http-nio-8081-exec-1] - org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
org.artifactory.storage.StorageException: org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
    at org.artifactory.storage.db.binstore.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:143)
    at org.artifactory.repo.service.RepositoryServiceImpl.saveResource(RepositoryServiceImpl.java:2578)
    at jdk.internal.reflect.GeneratedMethodAccessor489.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
    at jdk.proxy4/jdk.proxy4.$Proxy333.saveResource(Unknown Source)
    at org.artifactory.engine.UploadServiceImpl.uploadItemWithContent(UploadServiceImpl.java:702)
    at org.artifactory.engine.UploadServiceImpl.uploadItemWithProvidedContent(UploadServiceImpl.java:674)
    at org.artifactory.engine.UploadServiceImpl.uploadItem(UploadServiceImpl.java:537)
    at org.artifactory.engine.UploadServiceImpl.uploadFile(UploadServiceImpl.java:527)
    at org.artifactory.engine.UploadServiceImpl.uploadArtifact(UploadServiceImpl.java:506)
    at org.artifactory.engine.UploadServiceImpl.adjustResponseAndUpload(UploadServiceImpl.java:318)
    at org.artifactory.engine.UploadServiceImpl.validateRequestAndUpload(UploadServiceImpl.java:278)
    at org.artifactory.engine.UploadServiceImpl.processUpload(UploadServiceImpl.java:220)
    at org.artifactory.api.request.GracefulShutdownAware.track(GracefulShutdownAware.java:36)
    at org.artifactory.engine.UploadServiceImpl.upload(UploadServiceImpl.java:205)
    at jdk.internal.reflect.GeneratedMethodAccessor483.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.artifactory.request.aop.RequestAdvice.invoke(RequestAdvice.java:67)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
    at jdk.proxy4/jdk.proxy4.$Proxy399.upload(Unknown Source)
    at org.artifactory.addon.common.repomd.ArtifactoryService.upload(ArtifactoryService.java:548)
    at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.uploadBlobToUuidPath(DockerV2LocalRepoHandler.java:155)
    at org.jfrog.repomd.docker.v2.rest.handler.DockerV2LocalRepoHandler.patchBlobUpload(DockerV2LocalRepoHandler.java:141)
    at org.jfrog.repomd.docker.v2.rest.DockerV2Resource.patchUpload(DockerV2Resource.java:123)
    at jdk.internal.reflect.GeneratedMethodAccessor482.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.artifactory.webapp.servlet.RepoFilter.execute(RepoFilter.java:204)
    at org.artifactory.webapp.servlet.RepoFilter.doFilter(RepoFilter.java:106)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.artifactory.webapp.servlet.AccessFilter.useAuthentication(AccessFilter.java:627)
    at org.artifactory.webapp.servlet.AccessFilter.authenticateAndExecute(AccessFilter.java:457)
    at org.artifactory.webapp.servlet.AccessFilter.doFilterInternal(AccessFilter.java:308)
    at org.artifactory.webapp.servlet.AccessFilter.doFilter(AccessFilter.java:226)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.artifactory.webapp.servlet.RequestFilter.doFilter(RequestFilter.java:88)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.artifactory.webapp.servlet.ArtifactoryCsrfFilter.doFilter(ArtifactoryCsrfFilter.java:96)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:164)
    at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:80)
    at org.artifactory.webapp.servlet.SessionFilter.doFilter(SessionFilter.java:67)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.artifactory.webapp.servlet.ArtifactoryTracingFilter.doFilter(ArtifactoryTracingFilter.java:38)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:126)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:661)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:425)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:357)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:294)
    at org.artifactory.util.DockerInternalRewrite.redirect(DockerInternalRewrite.java:66)
    at org.artifactory.webapp.servlet.ArtifactoryFilter.redirectIfNeeded(ArtifactoryFilter.java:156)
    at org.artifactory.webapp.servlet.ArtifactoryFilter.doFilter(ArtifactoryFilter.java:116)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
    at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:289)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.valves.rewrite.RewriteValve.invoke(RewriteValve.java:542)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:960)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
    at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.jfrog.storage.StorageException: Failed to upload blob to Azure blob storage
    at org.jfrog.type.azureblobstorage.v2.AzureBlobStorageV2BinaryProvider.saveToTempFileAndUpload(AzureBlobStorageV2BinaryProvider.java:560)
    at org.jfrog.type.azureblobstorage.v2.AzureBlobStorageV2BinaryProvider.addStream(AzureBlobStorageV2BinaryProvider.java:277)
    at org.jfrog.storage.binstore.providers.cachefs.FileCacheBinaryProviderImpl.addStream(FileCacheBinaryProviderImpl.java:220)
    at org.jfrog.storage.binstore.providers.federated.FederatedBinaryProvider.addStream(FederatedBinaryProvider.java:364)
    at org.jfrog.storage.binstore.BinaryServiceImpl.addBinary(BinaryServiceImpl.java:390)
    at jdk.internal.reflect.GeneratedMethodAccessor490.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:220)
    at jdk.proxy4/jdk.proxy4.$Proxy376.addBinary(Unknown Source)
    at org.artifactory.storage.db.binstore.service.ArtifactoryBinaryServiceImpl.addBinary(ArtifactoryBinaryServiceImpl.java:141)
    ... 115 common frames omitted
Caused by: java.io.IOException: Directory 'data/filestore/temp' could not be created
    at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:2436)
    at org.apache.commons.io.FileUtils.openOutputStream(FileUtils.java:2397)
    at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1085)
    at org.apache.commons.io.FileUtils.copyInputStreamToFile(FileUtils.java:992)
    at org.jfrog.storage.binstore.providers.tools.FilePersistenceHelper.saveStreamToTempFile(FilePersistenceHelper.java:87)
    at org.jfrog.type.azureblobstorage.v2.AzureBlobStorageV2BinaryProvider.saveToTempFileAndUpload(AzureBlobStorageV2BinaryProvider.java:547)
    ... 132 common frames omitted
Logeshwarsn commented 1 year ago

@EvertonSA Thanks for reporting this issue. We will look into this issue internally and get back to you.

gitta-jfrog commented 1 year ago

Hi @EvertonSA

To fix this issue, please remove the following parameters from your custom binarystore.xml:

<binariesDir>data/filestore</binariesDir>
<tempDir>temp</tempDir>

These parameters (which are part of the templates published in JFrog docs) are redundant and create a conflict with the filesystem structure of Artifactory with splitServicesToContainers enabled.

Following the removal of these parameters from your binarystore.xml, Artifactory will use the code default location (/opt/jfrog/artifactory/var/data/artifactory/filestore), and upload of binaries should work as expected.

I'll appreciate your response. Yarden JFrog Support.

EvertonSA commented 1 year ago

@gitta-jfrog right,

image

if I remove this, does it mean that we will not be able to use Direct (Eventual-less) uploads anymore?

gitta-jfrog commented 1 year ago

Hi @EvertonSA
Removing these values should not impact the chain template you are using.

When using a Direct Chain Template, Artifactory needs the binariesDir/temp location only during the upload operation. Following the removal of binariesDir and tempDir from the binarystore.xml (and restart Artifactory nodes), Artifactory will use the default location for temp files during uploads ($JFROG_HOME/artifactory/var/data/artifactory/filestore)

Hope this clarifies.

EvertonSA commented 1 year ago

I hereby confirm the solution works.