search

halo-dev / plugin-s3

为 Halo 提供 S3 对象存储协议的存储策略
https://halo.run/store/apps/app-Qxhpp
GNU General Public License v3.0
34 stars 21 forks source link

S3存储策略上传附件到minio云存储，部分成功，部分报错 #114

Closed jiayan586 closed 10 months ago

jiayan586 commented 10 months ago

是什么版本出现了此问题？ 2.11.2

使用的什么数据库？ mariadb

使用的哪种方式运行？ Docker Compose

在线站点地址

发生了什么？ halo: 2.11.2 plugin-s3: 1.7.0

minio 通过nginx代理的api地址 通过halo后台配置S3存储，上传jpg png gif等文件报错，但是上传csv, json, txt, excel等格式文件正常。 同样大小文件png,修改文件后缀为.pn 也能上传成功

image

上传成功： image

2024-01-16T14:42:15.609+08:00  INFO 7 --- [boundedElastic-170] run.halo.s3os.S3OsAttachmentHandler      : operation: createMultipartUpload, result: CreateMultipartUploadResponse(Bucket=icepig, Key=files/2024/1/FI 操作命令笔记.md, UploadId=ZTRkMzE0ZjMtNmY4Ny00MTg3LWFjMjMtYzE0MmVjMDFlNTYxLjg0Y2YwYzk3LWFhYmMtNDlkMS1iNjYxLTIxN2UyOGM1MGU5OA)
2024-01-16T14:42:15.615+08:00  INFO 7 --- [boundedElastic-170] run.halo.s3os.S3OsAttachmentHandler      : operation: uploadPart, result: UploadPartResponse(ETag="a5906223f1a7c604b511bce551e2b442")
2024-01-16T14:42:15.619+08:00  INFO 7 --- [boundedElastic-170] run.halo.s3os.S3OsAttachmentHandler      : operation: completeUpload, result: CompleteMultipartUploadResponse(Location=https://minio.XXX.XXX.com:1314/icepig/files/2024/1/FI%20%E6%93%8D%E4%BD%9C%E5%91%BD%E4%BB%A4%E7%AC%94%E8%AE%B0.md, Bucket=icepig, Key=files/2024/1/FI 操作命令笔记.md, ETag="2d21428d5e68b0b0c5a830a936e9ca61-1")
2024-01-16T14:42:15.623+08:00  INFO 7 --- [boundedElastic-170] run.halo.s3os.S3OsAttachmentHandler      : operation: getMetadata, result: HeadObjectResponse(AcceptRanges=bytes, LastModified=2024-01-16T06:42:15Z, ContentLength=10380, ETag="2d21428d5e68b0b0c5a830a936e9ca61-1", ContentType=application/octet-stream, Metadata={})
2024-01-16T14:42:15.623+08:00  INFO 7 --- [boundedElastic-170] run.halo.s3os.S3OsAttachmentHandler      : Uploaded object files/2024/1/FI 操作命令笔记.md to bucket icepig successfully
2024-01-16T14:42:15.624+08:00  INFO 7 --- [boundedElastic-170] run.halo.s3os.S3OsAttachmentHandler      : Built attachment files/2024/1/FI 操作命令笔记.md successfully

上传失败报错：

2024-01-16T14:43:55.863+08:00 ERROR 7 --- [boundedElastic-169] a.w.r.e.AbstractErrorWebExceptionHandler : [9ccb5b2d-2716]  500 Server Error for HTTP POST "/apis/api.console.halo.run/v1alpha1/attachments/upload"

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, Request ID: 17AAC0D65EF2C9F4, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
    at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[na:na]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ run.halo.app.security.InitializeRedirectionWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.authentication.login.UsernamePasswordLogoutHandler [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.authentication.login.DelegatingLogoutPageGeneratingWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.authentication.login.UsernamePasswordAuthenticator [DefaultWebFilterChain]
    *__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ AnonymousAuthenticationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ AuthenticationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ AuthenticationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ CsrfWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ CorsWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ OrderedWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ OrderedWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ OrderedWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
    *__checkpoint ⇢ HTTP POST "/apis/api.console.halo.run/v1alpha1/attachments/upload" [ExceptionHandlingWebHandler]
Original Stack Trace:
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156) ~[na:na]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108) ~[na:na]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85) ~[na:na]
        at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43) ~[na:na]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:95) ~[na:na]
        at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:266) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:78) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:40) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:81) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[na:na]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) ~[na:na]
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) ~[na:na]
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[na:na]
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) ~[na:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[na:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:171) ~[na:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) ~[na:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[na:na]
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[na:na]
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[na:na]
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[na:na]
        at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:5438) ~[na:na]
        at run.halo.s3os.S3OsAttachmentHandler.lambda$checkFileExistsAndRename$29(S3OsAttachmentHandler.java:402) ~[na:na]
        at reactor.core.publisher.MonoSupplier$MonoSupplierSubscription.request(MonoSupplier.java:126) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.MonoSupplier.subscribe(MonoSupplier.java:48) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.FluxRetryWhen.subscribe(FluxRetryWhen.java:80) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.MonoRetryWhen.subscribeOrReturn(MonoRetryWhen.java:46) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:258) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:78) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.MonoUsing.subscribe(MonoUsing.java:109) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.Mono.subscribe(Mono.java:4512) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) ~[reactor-core-3.6.0.jar:3.6.0]
        at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) ~[reactor-core-3.6.0.jar:3.6.0]
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]
longjuan commented 10 months ago

参考 https://github.com/halo-dev/plugin-s3/issues/86 那位用户是反向代理（caddy）中配置了自己的规则，导致上传图片后缀会返回403/404状态码。

JohnNiang commented 10 months ago

/triage needs-information

JohnNiang commented 10 months ago

/triage duplicate /remove-triage needs-information

Close in favor of https://github.com/halo-dev/plugin-s3/issues/86.

/close

f2c-ci-robot[bot] commented 10 months ago

@JohnNiang: Closing this issue.

In response to [this](https://github.com/halo-dev/plugin-s3/issues/114#issuecomment-1894814570): >/triage duplicate >/remove-triage needs-information > >Close in favor of . > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
jiayan586 commented 10 months ago

参考 #86 那位用户是反向代理（caddy）中配置了自己的规则，导致上传图片后缀会返回403/404状态码。

我去检查了一遍nginx，没有做多余的规则，仅仅只是转发 而且我的lsky pro图床也用了相同的minio对象存储，上传图片是正常的。相同的域名，理论上应该走同一个反向代理规则。 太奇怪了..... image

longjuan commented 10 months ago

你可以试试不经过反向代理，endpoint直接指向minio，应该是能正常上传的。

jiayan586 commented 10 months ago

破案了，回来记录一下，NPM 这个cache打开 上传图片就会报错，一定要关掉 image 感谢 @longjuan 大佬的帮助！