FileUpload OutOfDirectMemoryError

markbanyang commented 3 years ago

当我在上传大文件的时候会发生OOM 下面是文件上传代码: `

@RequestMapping(value = "/stream-upload-test")

  public ResponseEntity<String> upload(HttpServletRequest request, HttpServletResponse response)
        throws IOException, FileUploadException {
    boolean isMultipart = ServletFileUpload.isMultipartContent(request);
    if (isMultipart) {
        ServletFileUpload upload = new ServletFileUpload();

        Map<String, String> params = new HashMap<>();
        InputStream is = null;
        FileItemIterator iter = upload.getItemIterator(request);
        while (iter.hasNext()) {
            FileItemStream item = iter.next();
            if (!item.isFormField()) {
                is = item.openStream();
                try {
                    int i = 0;
                    byte bb [] = new byte[4096];
                    while((i = is.read(bb)) != -1){
                        System.out.println(Arrays.toString(bb));
                    }
                } catch (Exception e) {
                    e.printStackTrace();
                } finally {
                    if (null != is) {
                        is.close();
                    }
                }
            } else {

                String fieldName = item.getFieldName();
                String value = Streams.asString(item.openStream());
                System.out.println(fieldName + " : " + value);
            }
        }
    }
    return new ResponseEntity<String>(HttpStatus.OK);
}

`

错误信息截图如下：

wangzihaogithub commented 3 years ago

我看下 , 估计明天上午解决

wangzihaogithub commented 3 years ago

解决了. 在这次提交, https://github.com/wangzihaogithub/spring-boot-protocol/commit/d65fe249948bb39ea437831850b1f4b27b081a1d 已经合并到master分支了, 也已经提交到maven中央仓库了. 过两天就同步到中央仓库了.

原因是: 错误的将netty的HttpPostMultipartRequestDecoder 的 discardThreshold参数设置成了Integer.MAX_VALUE导致的.

public int getDiscardThreshold(){ int discardThreshold = 0; if(multipartConfigElement != null) { discardThreshold = (int)multipartConfigElement.getMaxFileSize(); } if(discardThreshold <= 0){ discardThreshold = Integer.MAX_VALUE; //就是这里, 这个参数是控制复制文件的缓冲区, 我现在改成了不设置, 即netty默认的最大10M. } return discardThreshold; }

markbanyang commented 3 years ago

我验证了下，上传文件OOM的问题已经解决了。

markbanyang commented 3 years ago

另外我在测试下载大文件的时候，耗时比tomcat长很多，从抓取的数据包发现，下载文件的时候发送一部分包后，会等待一段时间才会继续发送下一个包：

下面是我的测试代码： `@RequestMapping("/file-download-test")

public ResponseEntity<String> downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception {
    String fileName = "CentOS-7-x86_64-DVD-2003.iso";
    String filePath = "./" + fileName;
    handleDownloadStream(fileName, filePath, request, response);
    return new ResponseEntity<>(HttpStatus.OK);
}

public void handleDownloadStream(String fileName, String filePath, HttpServletRequest request, HttpServletResponse res) throws IOException { byte[] buffer = new byte[4 * 1024]; OutputStream os = null; FileInputStream fStream = null; try { os = new BufferedOutputStream(res.getOutputStream()); res.reset(); String agent = request.getHeader("User-Agent"); if (agent == null) { return; } agent = agent.toUpperCase();

        //ie浏览器,火狐,Edge浏览器
        if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
            fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
        } else {
            fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
        }
        //safari RFC 5987标准
        if (agent.indexOf("SAFARI") > -1) {
            res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
        } else {
            res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
        }
        File file = new File(filePath);
        res.setContentType("application/octet-stream");
        res.setCharacterEncoding("UTF-8");
        res.setContentLength((int) file.length());
        fStream = new FileInputStream(file);
        int length = 0;
        while ((length = fStream.read(buffer)) != -1) {
            os.write(buffer, 0, length);
        }
        os.flush();

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(fStream);
    }
}`

wangzihaogithub commented 3 years ago

这是由这个参数控制的，你可以改成1KB。默认达到是1M后，或者主动调用flush方法，再给客户端发数据。

server.netty.http-servlet.response-max-buffer-size=1024

wangzihaogithub commented 3 years ago

我下个版本把默认值改小点，另外，你可以这样发文件，性能会好一些，因为用的是netty的零拷贝

NettyOutputStream nettyOutputStream = ((NettyOutputStream)response.getOutputStream())； nettyOutputStream.write(new File("/home/temp.json"));

markbanyang commented 3 years ago

好的我尝试下

markbanyang commented 3 years ago

文件下载是否存在内存不释放的问题，配置启动内存-Xmx512M -XX:MaxDirectMemorySize=512M，下载一个不到2G文件会卡住，并且会将分配的内存耗尽。

wangzihaogithub commented 3 years ago

可以看下代码吗？我用的你第一次的代码复现不出来。

master分支的这个类 com.github.netty.http.example.HttpController，是我用的例子，不知道一样不

`

@RequestMapping("/downloadFile") public ResponseEntity downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception { String fileName = "CentOS-7-x86_64-DVD-2003.iso";

    byte[] file = new byte[1024 * 1024 * 7];
    for (int i = 0; i < file.length; i++) {
        file[i] = (byte) i;
    }
    handleDownloadStream(fileName, new ByteArrayInputStream(file), request, response);
    return new ResponseEntity<>(HttpStatus.OK);
}

public void handleDownloadStream(String fileName, InputStream inputStream, HttpServletRequest request, HttpServletResponse res) throws IOException {
    byte[] buffer = new byte[4 * 1024];
    OutputStream os = null;
    try {
        os = new BufferedOutputStream(res.getOutputStream());
        res.reset();
        String agent = request.getHeader("User-Agent");
        if (agent == null) {
            return;
        }
        agent = agent.toUpperCase();

        //ie浏览器,火狐,Edge浏览器
        if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
            fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
        } else {
            fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
        }
        //safari RFC 5987标准
        if (agent.contains("SAFARI")) {
            res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
        } else {
            res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
        }
        res.setContentType("application/octet-stream");
        res.setCharacterEncoding("UTF-8");
        res.setContentLength(inputStream.available());
        int length = 0;
        while ((length = inputStream.read(buffer)) != -1) {
            os.write(buffer, 0, length);
        }
        os.flush();

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(inputStream);
    }
}

`

wangzihaogithub commented 3 years ago

辛苦发一下吧感谢

markbanyang commented 3 years ago

好的，稍等下。

markbanyang commented 3 years ago

`@RequestMapping("/file-download-test") public ResponseEntity downloadFile(HttpServletRequest request, HttpServletResponse response) throws Exception { String fileName = "CentOS-7-x86_64-DVD-2003.iso"; String filePath = "./" + fileName; handleDownloadStream(fileName, filePath, request, response); return new ResponseEntity<>(HttpStatus.OK); }

public void handleDownloadStream(String fileName, String filePath, HttpServletRequest request, HttpServletResponse res) throws IOException {
    byte[] buffer = new byte[4 * 1024];
    OutputStream os = null;
    FileInputStream fStream = null;
    try {
        os = new BufferedOutputStream(res.getOutputStream());
        res.reset();
        String agent = request.getHeader("User-Agent");
        if (agent == null) {
            return;
        }
        agent = agent.toUpperCase();

        //ie浏览器,火狐,Edge浏览器
        if (agent.indexOf("MSIE") > 0 || agent.indexOf("RV:11.0") > 0 || agent.indexOf("EDGE") > 0 || agent.indexOf("SAFARI") > -1) {
            fileName = URLEncoder.encode(fileName, "utf8").replaceAll("\\+", "%20");
        } else {
            fileName = new String(fileName.getBytes(StandardCharsets.UTF_8), "ISO8859_1");
        }
        //safari RFC 5987标准
        if (agent.indexOf("SAFARI") > -1) {
            res.addHeader("content-disposition", "attachment;filename*=UTF-8''" + fileName);
        } else {
            res.addHeader("Content-disposition", "attachment; filename=\"" + fileName + '"');
        }
        File file = new File(filePath);
        res.setContentType("application/octet-stream");
        res.setCharacterEncoding("UTF-8");
        fStream = new FileInputStream(file);
        int length = 0;
        while ((length = fStream.read(buffer)) != -1) {
            os.write(buffer, 0, length);
            os.flush();
        }
        os.flush();

    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        IOUtils.closeQuietly(fStream);
    }
}`

被下载的文件大小大概4个G jvm启动参数：

wangzihaogithub commented 3 years ago

发的文件好像坏了，看不到

wangzihaogithub commented 3 years ago

要不贴代码吧，就需要启动参数和代码片段就行

markbanyang commented 3 years ago

我发你邮箱了

markbanyang commented 3 years ago

下载的文件信息：

jvm启动参数：

------------------ 原始邮件 ------------------ 发件人: "wangzihaogithub/spring-boot-protocol" <notifications@github.com>; 发送时间: 2020年12月5日(星期六) 下午4:19 收件人: "wangzihaogithub/spring-boot-protocol"<spring-boot-protocol@noreply.github.com>; 抄送: "tntym"<2236350912@qq.com>;"Author"<author@noreply.github.com>; 主题: Re: [wangzihaogithub/spring-boot-protocol] FileUpload OutOfDirectMemoryError (#12)

辛苦发一下吧感谢

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

wangzihaogithub commented 3 years ago

复现了，我改下。

markbanyang commented 3 years ago

好的

wangzihaogithub commented 3 years ago

解决了，这是netty的bug，已经修复了在master分支了。后续pull-request 给netty。

修复netty的写失败后，没有再重试的bug。会导致下次无法再写入数据。

https://github.com/wangzihaogithub/spring-boot-protocol/commit/af43b8977adbb5dec65571369e9bd213ab8cc301

https://github.com/netty/netty/issues/10353

wangzihaogithub commented 3 years ago

netty before

`

                int attemptedBytes = buffer.remaining();
                final int localWrittenBytes = ch.write(buffer);
                if (localWrittenBytes <= 0) {
                    incompleteWrite(true);
                    return;
                }
                adjustMaxBytesPerGatheringWrite(attemptedBytes, localWrittenBytes, maxBytesPerGatheringWrite);
                in.removeBytes(localWrittenBytes);
                --writeSpinCount;

`

netty after

`

                int attemptedBytes = buffer.remaining();
                final int localWrittenBytes = ch.write(buffer);
                if (localWrittenBytes <= 0) {
                    incompleteWrite(true);
                }else {
                    adjustMaxBytesPerGatheringWrite(attemptedBytes, localWrittenBytes, maxBytesPerGatheringWrite);
                    in.removeBytes(localWrittenBytes);
                }
                --writeSpinCount;

`

markbanyang commented 3 years ago

使用commons-fileupload上传文件，以流的方式读取上传的数据，这个过程也是需要等上传的数据发送完成暂存在本地，才会到controller层处理么。直接读取流数据的方式上传文件，可不可以做到不在本地暂存一份。

wangzihaogithub commented 3 years ago

可以，我改下。不过这种模式和servlet的模式冲突(因为servlet要求解析)，我打算加个配置参数，让你可以选择开启或关闭。

markbanyang commented 3 years ago

这个配置可以通过重写spring提供CommonsMultipartResolver的来处理么： `public class CommonsMultipartResolverForProgress extends CommonsMultipartResolver {

@Override
public boolean isMultipart(HttpServletRequest request) {

    if (request.getRequestURI().startsWith("/stream-upload-test")) {
        return false;
    }
    return super.isMultipart(request);
}

} `

wangzihaogithub commented 3 years ago

你改这一处不够， spring还会调用 getParameter，导致触发解析body。

wangzihaogithub commented 3 years ago

改好了，在master分支，你可以重写spring提供CommonsMultipartResolver的来处理了。这是这次改动， https://github.com/wangzihaogithub/spring-boot-protocol/commit/603ce3a4159095f6d63b26480145c01cd001fd11

如果你需要在maven中央仓库中，需要我发新版本，可以直接告诉我，我就发新版了。

markbanyang commented 3 years ago

还需要什么额外的特殊配置么，用你的测试代码是生效了，在我的环境里面依赖的master的代码，没有生效。测试的代码是使用的你提交的测试代码。

wangzihaogithub commented 3 years ago

需要你从我这的master分支, 同步一下代码. 或者我上传到中央仓库,

wangzihaogithub commented 3 years ago

idea的git 有个 remote的选项, 把我这个项目加进去, 然后merge into到你的分支就行

wangzihaogithub commented 3 years ago

是的

markbanyang commented 3 years ago

嗯我是拉下来最新的代码做的测试，测试代码跟你应该是一致的。

wangzihaogithub commented 3 years ago

要不麻烦下, 你断点到 ServletInputStreamWrapper#awaitDataIfNeed 方法上, 看是哪个方法触发了阻塞解析.

wangzihaogithub commented 3 years ago

你是不是这个类 CommonsMultipartResolver, 没有正确执行啊.

`

@Override public boolean isMultipart(HttpServletRequest request) { if (request.getRequestURI().startsWith("/test/uploadForApache")) { return false; } return super.isMultipart(request); }

`

markbanyang commented 3 years ago

我断点了下，主要在com.github.netty.protocol.servlet.NettyMessageToServletRunnable#onMessage这个方法上代码会有半分钟时间，还没有执行到CommonsMultipartResolver跟ServletInputStreamWrapper#awaitDataIfNeed 方法上。下面这个地方

wangzihaogithub commented 3 years ago

哦哦发现了, 这是个bug, 刚提交了, 你再拉下代码

markbanyang commented 3 years ago

好的, 拉下来验证可以了。

markbanyang commented 3 years ago

目前的数据处理是否缺少tcp的负反馈机制。从最外层的表现看：以下载文件举例，当请求端不读取数据时，ServletOutputStream的write方法不会阻塞。

netty启动时的高水位设置为Integer.MAX_VALUE，导致com.yunanbao.webserver.core.util.ChunkedWriteHandler#doFlush0 在判断channel.isWritable()永远为true，而导致队列一直增加。

将netty的高水位设置为1M时，下载一个大文件，但请求端不读取时会发生pointer being freed was not allocated错误，导致系统崩溃

markbanyang commented 3 years ago

是不是应该当阻塞的处理放到com.yunanbao.webserver.protocol.servlet.ServletOutputStream#writeHttpBody这个方法中，简单修改代码测试了下：

wangzihaogithub commented 3 years ago

可以先这样解决, 只要不触发netty的取消OP_WRITE, 就不会有问题.

不过发文件, 建议你用NettyOutputStream#write(file), 一直read and write, 现在的bug比较多. 正在找办法从根源解决.

wangzihaogithub commented 3 years ago

一旦把网卡写繁忙一次, 后续这个tcp链接就会卡住, 停止继续刷写数据, 直到你停止write, 并且释放当前线程, 才会恢复. 这是因为在IO线程中处理,

但是在非IO线程处理, 高并发的时候, 会触发HttpEncoder 的state报错.

我最近几天正在看咋弄..

markbanyang commented 3 years ago

好的

markbanyang commented 3 years ago

上面那个方式也解决不了，网卡写繁忙一次就在也写不了数据了

markbanyang commented 3 years ago

这个问题后面会解决么

wangzihaogithub commented 3 years ago

会，不过可能要下周开始解决了，这周工作有些忙

markbanyang commented 3 years ago

好的

wangzihaogithub commented 3 years ago

已经在master分支解决了, 同时发布了2.1.0版本.

markbanyang commented 3 years ago

好的我验证下

wangzihaogithub / spring-boot-protocol

FileUpload OutOfDirectMemoryError #12