mrniko / netty-socketio

Socket.IO server implemented on Java. Realtime java framework
Apache License 2.0
6.82k stars 1.65k forks source link

1.7.3 leak memory && Unexpected end-of-input in VALUE_STRING #195

Closed tselishev-semen closed 9 years ago

tselishev-semen commented 9 years ago

Hi. I use a framework(1.7.3 version) already some time and I have accumulated a number of issues:).

  1. Memory leak when loading Blobs: ServerSIde
public class FileDTO {
    private byte[] data;
    public FileDTO(int id, byte[] data, String error, String key, String name, String sql) {
        super();
       ...
        this.data = data;
       ...
    }
    ...
}

public class Launcher {
  public static void main(String[] args) throws InterruptedException {
...
  Configuration config = new Configuration();
        config.setHostname("192.168.0.34");
        config.setPort(3000);
        config.setMaxFramePayloadLength(10000000);
        config.setMaxHttpContentLength(10000000);
        final SocketIOServer server = new SocketIOServer(config);

....
 }
....
  server.addEventListener("fileUpload", FileDTO.class, new DataListener<FileDTO>() {
            Connection conn = new WebService();
            @Override
            public void onData(SocketIOClient client, FileDTO data, AckRequest ackRequest) {
               //code for upload file
            }
        });
}

Client Side

socket.emit('fileUpload', {
  data: base64Data
..
})

Problem: when I upload a large file( ~150+kb ), the server generates an error

type: class com.corundumstudio.socketio.protocol.Event
com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input in VALUE_STRING
 at [Source: io.netty.buffer.ByteBufInputStream@25f5445b; line: 1, column: 129893]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:508)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:445)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:441)
        at com.fasterxml.jackson.core.base.ParserBase.loadMoreGuaranteed(ParserBase.java:408)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2184)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString(UTF8StreamJsonParser.java:2165)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:279)
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:224)
        at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:262)
        at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:64)
        at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:14)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3023)
        at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1705)
        at com.corundumstudio.socketio.protocol.JacksonJsonSupport$EventDeserializer.deserialize(JacksonJsonSupport.java:140)
        at com.corundumstudio.socketio.protocol.JacksonJsonSupport$EventDeserializer.deserialize(JacksonJsonSupport.java:126)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3051)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2192)
        at com.corundumstudio.socketio.protocol.JacksonJsonSupport.readValue(JacksonJsonSupport.java:223)
        at com.corundumstudio.socketio.JsonSupportWrapper.readValue(JsonSupportWrapper.java:55)
        at com.corundumstudio.socketio.protocol.PacketDecoder.decode(PacketDecoder.java:180)
        at com.corundumstudio.socketio.protocol.PacketDecoder.decodePackets(PacketDecoder.java:118)
        at com.corundumstudio.socketio.handler.InPacketHandler.channelRead0(InPacketHandler.java:65)
        at com.corundumstudio.socketio.handler.InPacketHandler.channelRead0(InPacketHandler.java:36)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at com.corundumstudio.socketio.transport.WebSocketTransport.channelRead(WebSocketTransport.java:92)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at com.corundumstudio.socketio.transport.PollingTransport.channelRead(PollingTransport.java:109)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at com.corundumstudio.socketio.handler.AuthorizeHandler.channelRead(AuthorizeHandler.java:115)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)
[nioEventLoopGroup-3-8] ERROR com.corundumstudio.socketio.handler.InPacketHandler - Error during data processing. Client sessionId: 3b89da4b-d899-4399-9a32-2d93236b1ad3, data:
java.lang.NullPointerException
        at com.corundumstudio.socketio.protocol.PacketDecoder.decode(PacketDecoder.java:181)
        at com.corundumstudio.socketio.protocol.PacketDecoder.decodePackets(PacketDecoder.java:118)
        at com.corundumstudio.socketio.handler.InPacketHandler.channelRead0(InPacketHandler.java:65)
        at com.corundumstudio.socketio.handler.InPacketHandler.channelRead0(InPacketHandler.java:36)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:168)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at com.corundumstudio.socketio.transport.WebSocketTransport.channelRead(WebSocketTransport.java:92)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at com.corundumstudio.socketio.transport.PollingTransport.channelRead(PollingTransport.java:109)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at com.corundumstudio.socketio.handler.AuthorizeHandler.channelRead(AuthorizeHandler.java:115)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:108)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
        at java.lang.Thread.run(Thread.java:745)

a) How can I fix this error? b) If several times to upload a large file (2mb) in different pages, server crashes:

[nioEventLoopGroup-3-8] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() 
was not called before it's garbage-collected. 
Enable advanced leak reporting to find out where the leak occurred. 
To enable advanced leak reporting, specify the JVM option 
'-Dio.netty.leakDetectionLevel=advanced' or call ResourceLeakDetector.setLevel()

How can I avoid it?

mrniko commented 9 years ago

Hi! Thanks for using :)

Unexpected end-of-input in VALUE_STRING

I think the data source has been trimmed by some kind of limit, please check it. Anyway netty-socketio receives a half of string, i think.

was not called before it's garbage-collected.

try to allocate more memory to your server

tselishev-semen commented 9 years ago

Thank you for the quick response. But that's not all :) 2 How to run the server as a daemon I'm used https://github.com/mrniko/netty-socketio-demo and I found a solution:

mvn exec:java
ctrl+z
disown - h
bg 1

Maybe there is a better solution?

3 I tried to upgrade to version 1.7.5. But my code is not working correctly. Where you can read what has changed?) and sorry for my english)

mrniko commented 9 years ago

2 How to run the server as a daemon

mvn exec:java .... & try to add & to run it in background

3 I tried to upgrade to version 1.7.5

change list is always available in readme.md file

tselishev-semen commented 9 years ago

try to add & to run it in background

It worked, thanks) How i can reboot server?

I think the data source has been trimmed by some kind of limit, please check it. Anyway netty-socketio receives a half of string, i think.

Before I added:

        config.setMaxFramePayloadLength(10000000);
        config.setMaxHttpContentLength(10000000);

was an error:

Max frame length of 65536 has been exceeded.

I think maybe data truncated, when the nginx passes the data to the websoket server

//nginx.conf
    location /socket
     {
        proxy_pass http://192.168.0.34:3000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;

    }

I'll check it out

try to allocate more memory to your server

4GB of memory on the server. This is not enough?

mrniko commented 9 years ago

How i can reboot server?

Usually you need to wrap your server in some kind of application and run it via service or with some app-container. All functionality (i mean .start and .stop methods) are provided by my lib.

4GB of memory on the server. This is not enough?

May be it's your case? http://stackoverflow.com/questions/22563167/netty-bytebuf-dont-get-the-full-message-of-a-large-input-when-decoding

mrniko commented 9 years ago

I think maybe data truncated, when the nginx passes the data to the websoket server

Try to reproduce this problem without nginx first

mrniko commented 9 years ago

Also try to set Config.tcpReceiveBufferSize setting

tselishev-semen commented 9 years ago

try to add & to run it in background

When I close the terminal, the process is terminated too.This is not what I wanted :)

Usually you need to wrap your server in some kind of application and run it via service or with some app-container. All functionality (i mean .start and .stop methods) are provided by my lib.

Looks, like I need more knowledge about linux:). Thanks.

Try to reproduce this problem without nginx first

I will raise test server, where the websocket server will run without nginx. Then write about the results:)

pablojr commented 9 years ago

| 2 How to run the server as a daemon Have you tried Procrun (for Windows)? It wraps a Java application into a Windows service that you can control (start/stop/etc) from the Windows Services control app. We use it with vey good results.

tselishev-semen commented 9 years ago

Have you tried Procrun (for Windows)? It wraps a Java application into a Windows service that you can control (start/stop/etc) from the Windows Services control app. We use it with vey good results.

I only have a linux:) Any solution for linux(centos)? I am not yet a java developer and I will be glad of any help:)

tselishev-semen commented 9 years ago

mrniko, Вы же с Москвы) по русски мб лучше?

tselishev-semen commented 9 years ago

Try to reproduce this problem without nginx first

Without a nginx problem still occurs.

mrniko commented 9 years ago

mrniko, Вы же с Москвы) по русски мб лучше?

it's better on english, because other users can join too in this case

mrniko commented 9 years ago

Also try to set Config.tcpReceiveBufferSize setting. did you tried to change this setting?

tselishev-semen commented 9 years ago

Also try to set Config.tcpReceiveBufferSize setting. did you tried to change this setting?

I added:

 Configuration config = new Configuration();
   SocketConfig socketConfig = new SocketConfig();
        socketConfig.setTcpReceiveBufferSize(20);
        config.setSocketConfig(socketConfig);
//another settings ...
 final SocketIOServer server = new SocketIOServer(config);

result: 1) error does not appear 2) But the file(Blob) is not uploaded

What value of the attribute(tcpReceiveBufferSize ) to use? For example, to upload 1 mb.

mrniko commented 9 years ago

try to set setTcpReceiveBufferSize=10000000

tselishev-semen commented 9 years ago

try to set setTcpReceiveBufferSize=10000000

error Unexpected end-of-input in VALUE_STRING appeared again :)

mrniko commented 9 years ago

at [Source: io.netty.buffer.ByteBufInputStream@25f5445b; line: 1, column: 129893]

and which value in column now in error?

tselishev-semen commented 9 years ago

at [Source: io.netty.buffer.ByteBufInputStream@25f5445; line: 1, column: 129893] and which value in column now in error?

I do not know how to write this type of errors to a text file. In the terminal does not fit the full log :(

//other base64 data
cHVzaCgnXHJcbicpOwogICAgICAgIH0KCiAgICAgICAgcmV0dXJuIHJlc3VsdC5qb2luKCcnKTsKICAgIH0KCiAgICBmdW5jdGlvbiBwYXVzZShuKSB7CiAgICAgICAgdG9kYXkgPSBuZXcgRGF0ZSgpCiAgICAgICAgdG9kYXkyID0gdG9kYXkKICAgICAgICB3aGlsZSAoKHRvZGF5MiAtIHRvZGF5KSA8PSBuKSB7CiAgICAgICAgICAgIHRvZGF5MiA9IG5ldyBEYXRlKCkKICAgICAgICB9CiAgICB9CgogICAgdGhpcywgImV2ZW50cyIpIHx8IHt9KVthLnR5cGVdIHx8IFtdLCBrID0gbi5ldmVudC5zcGVjaWFsW2EudHlwZV0gfHwge307CiAgICBpZiAoaVswXSA9IGEsIGEuZGVsZWdhdGVUYXJnZXQgPSB0aGlzLCAhay5wcmVEaXNwYXRjaCB8fCBrLnByZURpc3BhdGNoLmNhbGwodGhpcywgYSkgIT09ICExKSB7CiAgICAgICAgaCA9IG4uZXZlbnQuaGFuZGxlcnMuY2FsbCh0aGlzLCBhLCBqKSwgYiA9IDA7CiAgICAgICAgd2hpbGUgKChmID0gaFtiKytdKSAmJiAhYS5pc1Byb3BhZ2F0aW9uU3RvcHBlZCgpKSB7CiAgICAgICAgICAgIGEuY3VycmVudFRhcmdldCA9IGYuZWxlbSwgYyA9IDA7CiAgICAgICAgICAgIHdoaWxlICgoZ for type: class com.corundumstudio.socketio.protocol.Event
com.fasterxml.jackson.core.JsonParseException: Unexpected end-of-input in VALUE_STRING
 at [Source: io.netty.buffer.ByteBufInputStream@42709226; line: 1, column: 129981]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1419)
....
tselishev-semen commented 9 years ago

I think, I found a bug. It looks like a bug in Google Chrome. In another browser (ff) everything(upload blob) works fine. I can not believe it :)

tselishev-semen commented 9 years ago

probably a bug in Chromium. Opera and Chrome do not work FF и IE work well

mrniko commented 9 years ago

so setTcpReceiveBufferSize=10000000 helps to upload in FF and IE?

tselishev-semen commented 9 years ago

so setTcpReceiveBufferSize=10000000 helps to upload in FF and IE?

no. FF and IE work without setTcpReceiveBufferSize

tselishev-semen commented 9 years ago

https://code.google.com/p/chromium/issues/detail?id=420578 Probably, the Chrome sends the fragmented message(when message > 100kb?) and Websocket server does not know to handle them. What do you think? Problem may be in this?

mrniko commented 9 years ago

as wrote in comments it's 128Kb limit. In your case the string has end at 129981 or 129981 byte, so i think it's really this issue.

tselishev-semen commented 9 years ago

Any idea how to fix it?

mrniko commented 9 years ago

as i know netty supports websocket frame fragmentation by ContinuationWebSocketFrame. Don't know which side is broken - chromium or netty.

tselishev-semen commented 9 years ago

as i know netty supports websocket frame fragmentation by ContinuationWebSocketFrame. Don't know which side is broken - chromium or netty.

If I replace the *WebSocketFrame classes in netty-socketio to ContinuationWebSocketFrame class, it can help?

1) I will try to use a different socket server (nodejs or php) 2) and try to write a working example for https://github.com/mrniko/netty-socketio-demo, which will appear this error

tselishev-semen commented 9 years ago

2) and try to write a working example for https://github.com/mrniko/netty-socketio-demo, which will appear this error

change fileChatLauncher

public class ChatLauncher {

    public static void main(String[] args) throws InterruptedException {

        Configuration config = new Configuration();
        config.setHostname("localhost");
        config.setPort(9092);
        config.setMaxFramePayloadLength(10000000);
        config.setMaxHttpContentLength(10000000);

        final SocketIOServer server = new SocketIOServer(config);
        server.addEventListener("chatevent", ChatObject.class, new DataListener<ChatObject>() {
            @Override
            public void onData(SocketIOClient client, ChatObject data, AckRequest ackRequest) {
                // broadcast messages to all clients
                server.getBroadcastOperations().sendEvent("chatevent", data);
            }
        });

        server.start();

        Thread.sleep(Integer.MAX_VALUE);

        server.stop();
    }

}

To reproduce the error: 0) mvn exec:java 1) Open chrome 2) ctrl+shifr+i (open console) 3) execute code:

var str = 'sadsadsa1231231dfds;fdsfsdfsdfdsfdsf1121fadsfdsfsfssadsadasdsadasdsadas';
var i = 0;
var result = [];
while( i <10000){
    i= i + 1 ;
    result.push(str);
}
var message = btoa(result.join('dfadfdf'));
socket.emit('chatevent', {userName: 'sem', message: message });

4) Then you will see an my error in the terminal I tested it in version 1.7.3.

tselishev-semen commented 9 years ago

1) I will try to use a different socket server (nodejs or php)

I tried to run the same code on nodejs(http://socket.io/get-started/chat/). It worked perfectly.

var str = 'sadsadsa1231231dfds;fdsfsdfsdfdsfdsf1121fadsfdsfsfssadsadasdsadasdsadas';
var i = 0;
var result = [];
while( i <10000){
    i= i + 1 ;
    result.push(str);
}
var message = btoa(result.join('dfadfdf'));
  socket.emit('chat message', message);

Result: there are no errors and message send

mrniko commented 9 years ago

I found the cause of this issue. Will fix during a day

mrniko commented 9 years ago

fixed