search

AirboZH / plugin-uposs

为 Halo 2.0 提供又拍云 OSS 的存储策略
GNU General Public License v3.0
6 stars 2 forks source link

⚠️大文件上传栈溢出 #1

Closed AirboZH closed 1 year ago

AirboZH commented 1 year ago

log:

2023-01-11T01:12:19.466+08:00  WARN 91724 --- [undedElastic-13] reactor.core.Exceptions                  : throwIfFatal detected a jvm fatal exception, which is thrown and logged below:

java.lang.StackOverflowError: null
    at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198) ~[na:na]
    at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198) ~[na:na]
    at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198) ~[na:na]
    at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198) ~[na:na]
    at java.base/java.io.SequenceInputStream.read(SequenceInputStream.java:198) ~[na:na]
        ………………
JohnNiang commented 1 year ago

https://github.com/AirboZH/plugin-uposs/blob/8af6b5672e7166bb9c41f29fd7616fb002f74646/src/main/java/run/halo/uposs/UPOssAttachmentHandler.java#L117

建议不要在这里转换成 SequenceInputStream，这里会把文件内容完整拷贝到 byte 数组中。请参考：

https://github.com/halo-sigs/plugin-alioss/blob/9d4225f9c4f7920e9b9f5ab6086fbea24307b50e/src/main/java/run/halo/alioss/AliOssAttachmentHandler.java#L155

longjuan commented 1 year ago

image 这又是需要统计长度的，可能要通过分段上传方式才不会oom

AirboZH commented 1 year ago

谢谢@JohnNiang @longjuan 两位的帮助。经排查，SequenceInputStream 在读取流的时候会递归调用read(byte[], int, int)方法，所以在大量递归调用后就会造成SOE。

参考@JohnNiang AliOss后解决了这个问题。

测试后发现100M文件可以上传，但是JVM内存占用也会特别大，将流全部读到内存还是有很大问题，再大的文件会存在 @longjuan 说的OOM问题，计划采用分片上传的方案。

再次感谢两位~