socket is not closed after the client request ended

[ghost]

the socket fd is not closed after the client request ended. reproduce procedure:

use the benchmark/woo.lisp $sbcl --script benchmark/woo.lisp Running at http://localhost:5000/

in another terminal, $ ps -A|grep sbcl 12374 pts/3 00:00:00 sbcl $ ls -l /proc/12374/fd total 0 lrwx------ 1 64 May 20 19:11 0 -> /dev/pts/3 lrwx------ 1 64 May 20 19:11 1 -> /dev/pts/3 lrwx------ 1 64 May 20 19:10 2 -> /dev/pts/3 lr-x------ 1 64 May 20 19:11 3 -> /usr/lib/sbcl/sbcl.core lr-x------ 1 64 May 20 19:11 4 -> /usr/lib/sbcl/sbcl.core lrwx------ 1 64 May 20 19:11 5 -> /dev/tty lr-x------ 1 64 May 20 19:11 6 -> /woo/benchmark/woo.lisp lrwx------ 1 64 May 20 19:11 7 -> anon_inode:[eventpoll] lrwx------ 1 64 May 20 19:11 8 -> socket:[4047374]

$ http http://localhost:5000 HTTP/1.1 200 OK Connection: keep-alive Date: Wed, 20 May 2015 23:12:02 GMT Transfer-Encoding: chunked

Hello, World

$ ls -l /proc/12374/fd total 0 lrwx------ 1 64 May 20 19:11 0 -> /dev/pts/3 lrwx------ 1 64 May 20 19:11 1 -> /dev/pts/3 lrwx------ 1 64 May 20 19:10 2 -> /dev/pts/3 lr-x------ 1 64 May 20 19:11 3 -> /usr/lib/sbcl/sbcl.core lr-x------ 1 64 May 20 19:11 4 -> /usr/lib/sbcl/sbcl.core lrwx------ 1 64 May 20 19:11 5 -> /dev/tty lr-x------ 1 64 May 20 19:11 6 -> /woo/benchmark/woo.lisp lrwx------ 1 64 May 20 19:11 7 -> anon_inode:[eventpoll] lrwx------ 1 64 May 20 19:11 8 -> socket:[4047374] lrwx------ 1 64 May 20 19:12 9 -> socket:[4047376] ------> note this one added

the newly added socket fd will be kept forever.

this causes process error: too many files opened.

[fukamachi]

It's because you sent a keep-alive HTTP request that establishes a persistent connection.

$ http http://localhost:5000
HTTP/1.1 200 OK
Connection: keep-alive
Date: Wed, 20 May 2015 23:12:02 GMT
Transfer-Encoding: chunked

Hello, World

It will be timeout in 15 min after the last event.

[ghost]

But the http client has dropped the underneath tcp connection explicitly, in this case, the 'keep-alive' doesn't make sense to keep the connection at server side.

'keep-alive' is to ask client and server to keep the tcp connection, but if one of them closes the tcp by fin or rst, how is it possible to keep the socket open and reuse it?

[fukamachi]

But the http client has dropped the underneath tcp connection explicitly

Are you sure? Woo closes a socket when it receives EOF even when the connection is 'keep-alive'. The above situation should happen only when the client disappears unexpectedly.

Can you tell us some information about the 'http' client?

[ghost]

Sorry for the late.

I found this problem when I use wrk to test the example code in the repo, the file handle limits reached very quickly, even I expand the it to 16384, that sounds very wield. then I use httpie tool to test one request at a time. the command line just shown as above.

Since the httpie tool just run one time and exit, the tcp connection definitely should be closed. but I still can see the fd in its proc fd directory.

the steps to reproduce it:

$ cd woo/benchmark $ shly -I $PWD/.. -L woo run-repl

> load "woo.lisp"
To load "woo":
  Load 1 ASDF system:
    woo
; Loading "woo"
Running at http://localhost:5000/

in another terminal, $ http localhost:5000 $ ps -A|grep sbcl 10617 pts/11 00:00:00 sbcl $ ls -l /proc/10617/fd total 0 lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 0 -> /dev/pts/11 lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 1 -> /dev/pts/11 lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 2 -> /dev/pts/11 lr-x------ 1 jzhu jzhu 64 Jun 8 13:28 3 -> /home/jzhu/.shelly/dumped-cores/sbcl.core lr-x------ 1 jzhu jzhu 64 Jun 8 13:28 4 -> /home/jzhu/.shelly/dumped-cores/sbcl.core lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 5 -> /dev/tty lr-x------ 1 jzhu jzhu 64 Jun 8 13:28 6 -> /home/jzhu/code/woo/benchmark/woo.lisp lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 7 -> anon_inode:[eventpoll] lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 8 -> socket:[13130212] $ http localhost:5000 # run again $ ls -l /proc/10617/fd total 0 lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 0 -> /dev/pts/11 lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 1 -> /dev/pts/11 lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 2 -> /dev/pts/11 lr-x------ 1 jzhu jzhu 64 Jun 8 13:28 3 -> /home/jzhu/.shelly/dumped-cores/sbcl.core lr-x------ 1 jzhu jzhu 64 Jun 8 13:28 4 -> /home/jzhu/.shelly/dumped-cores/sbcl.core lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 5 -> /dev/tty lr-x------ 1 jzhu jzhu 64 Jun 8 13:28 6 -> /home/jzhu/code/woo/benchmark/woo.lisp lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 7 -> anon_inode:[eventpoll] lrwx------ 1 jzhu jzhu 64 Jun 8 13:28 8 -> socket:[13130212] lrwx------ 1 jzhu jzhu 64 Jun 8 13:29 9 -> socket:[13130236]

we can see the last socket fd is still there.

BTW, using curl to do the test, we get the same result.

$ curl http://localhost:5000

Best regards, Eric

[kayhman]

Hi,

I've had a similar issue, but with a client written in go. You can do a

lsof -p mySbclId | wc -l

To see that the number of open (zombi) socket connections increases with each request.

I've provided a merge request to fix this issue : https://github.com/fukamachi/woo/pull/27

In the current code, shutdown() is used to end the socket, but close() must be applied too.

Regards,

Guillaume

[kayhman]

More info on shutdown() and close() for socket : http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html#closedown

[fukamachi]

Fixed at https://github.com/fukamachi/woo/commit/e6b0166db17d184f9862ef168904e6f86f505d3f.