guicaiyue / plugin-githuboss

为 Halo 提供 GitHub 存储策略
GNU General Public License v3.0
7 stars 3 forks source link

多文件上传可能造成并发问题 #2

Open MartyAlien opened 1 year ago

MartyAlien commented 1 year ago

image 图上我上传了4个图片,其中会有一个出现其中一个请求失败:

2023-05-23T13:59:46.937+08:00 ERROR 1940 --- [ctor-http-nio-6] run.halo.oss.GitHubAttachmentHandler     : Caught an ClientException, which means the client encountered a serious internal
problem while trying to communicate with OSS, such as not being able to access
the network.

2023-05-23T13:59:46.938+08:00 DEBUG 1940 --- [ctor-http-nio-6] a.w.r.e.AbstractErrorWebExceptionHandler : [a9df9d37-670] Resolved [RuntimeException: Failed to upload file] for HTTP POST /apis/api.console.halo.run/v1alpha1/attachments/upload
2023-05-23T13:59:46.938+08:00 ERROR 1940 --- [ctor-http-nio-6] a.w.r.e.AbstractErrorWebExceptionHandler : [a9df9d37-670]  500 Server Error for HTTP POST "/apis/api.console.halo.run/v1alpha1/attachments/upload"

java.lang.RuntimeException: Failed to upload file
    at run.halo.oss.GitHubAttachmentHandler.lambda$upload$14(GitHubAttachmentHandler.java:191) ~[na:na]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    *__checkpoint ⇢ run.halo.app.console.ConsoleProxyFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.InitializeRedirectionWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.authentication.login.UsernamePasswordLogoutHandler [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.authentication.login.DelegatingLogoutPageGeneratingWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ run.halo.app.security.authentication.login.UsernamePasswordAuthenticator [DefaultWebFilterChain]
    *__checkpoint ⇢ AuthorizationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ExceptionTranslationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ LogoutWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ServerRequestCacheWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ AnonymousAuthenticationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ AuthenticationWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ReactorContextWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ CsrfWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ CorsWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HttpHeaderWriterWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ OrderedWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ OrderedWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ OrderedWebFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
    *__checkpoint ⇢ org.springframework.web.filter.reactive.ServerHttpObservationFilter [DefaultWebFilterChain]
    *__checkpoint ⇢ HTTP POST "/apis/api.console.halo.run/v1alpha1/attachments/upload" [ExceptionHandlingWebHandler]
Original Stack Trace:
        at run.halo.oss.GitHubAttachmentHandler.lambda$upload$14(GitHubAttachmentHandler.java:191) ~[na:na]
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2071) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144) ~[reactor-core-3.5.5.jar:3.5.5]
        at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:413) ~[reactor-netty-core-1.1.6.jar:1.1.6]
        at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:431) ~[reactor-netty-core-1.1.6.jar:1.1.6]
        at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:485) ~[reactor-netty-core-1.1.6.jar:1.1.6]
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:712) ~[reactor-netty-http-1.1.6.jar:1.1.6]
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113) ~[reactor-netty-core-1.1.6.jar:1.1.6]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1383) ~[netty-handler-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[netty-handler-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[netty-handler-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.92.Final.jar:4.1.92.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.92.Final.jar:4.1.92.Final]
        at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

这是我在开发插件时捞出来的日志记录,从代码上看上传文件前会对文件进行重命名,由于并发过快,两个不同的图片可能会被命名成同一个名字,但这不是引起上述报错的主要问题

guicaiyue commented 1 year ago

好的,非常感谢你能发现这个问题,我会抽时间处理的

guicaiyue commented 1 year ago

我已经更新1.2版本了,我自己试没问题了,你可以再试试,有问题随时沟通

MartyAlien commented 1 year ago

你好,问题依然能复现: 成功设置最大并发数5个: image

同时上传4个图片容易触发异常,但是也存在偶然不会触发问题;同时上传4个以下数量的图片就基本上是能保证不会报错的 image

不过问题应该不大,是否考虑是GIthub的限流或者是并发限制

guicaiyue commented 1 year ago

github对于用户访问速率是5000每小时,对于单接口有10秒的超时限制。 所以我在想你这个报错是不是超时导致的,可以开启 debug 日志看下,如果github调用结果为”Server Error“则触发的10秒超时,出现10秒问题可能网络问题,也有可能是带宽上行不够传的慢。 我忘记说了,那个并行配置变更后,需要停用再启用插件才能生效。