search

ginuerzh / gost

GO Simple Tunnel - a simple tunnel written in golang
MIT License
16.08k stars 2.49k forks source link

goroutine 泄漏问题 #432

Open violarulan opened 5 years ago

violarulan commented 5 years ago

Gost 版本：gost 2.8.1 (go1.12.10 linux/amd64)

Client <--TCP--> GostServerA <--Websocket--> GostServerB <--> Internet

GostServerA 运行期间内存持续上升，使用 pprof 定位到有越来越多的 goroutine 阻塞在 err := <-errc (https://github.com/ginuerzh/gost/blob/master/server.go#L162)

运行接近 10 小时，占用内存接近 600M，查询 TIME_WAIT 发现较多，不知是否和这个有关

netstat -anpt | grep TIME_WAIT | wc -l
1113
File: gost
Type: goroutine
Time: Oct 12, 2019 at 1:26pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 4917, 99.94% of 4920 total
Dropped 73 nodes (cum <= 24)
Showing top 10 nodes out of 37
      flat  flat%   sum%        cum   cum%
      4917 99.94% 99.94%       4917 99.94%  runtime.gopark
         0     0% 99.94%        814 16.54%  bufio.(*Reader).Peek
         0     0% 99.94%        815 16.57%  bufio.(*Reader).fill
         0     0% 99.94%       1631 33.15%  bytes.(*Buffer).ReadFrom
         0     0% 99.94%       1631 33.15%  crypto/tls.(*Conn).Read
         0     0% 99.94%       1631 33.15%  crypto/tls.(*Conn).readFromUntil
         0     0% 99.94%       1631 33.15%  crypto/tls.(*Conn).readRecord
         0     0% 99.94%       1631 33.15%  crypto/tls.(*Conn).readRecordOrCCS
         0     0% 99.94%       1631 33.15%  crypto/tls.(*atLeastReader).Read
         0     0% 99.94%       1631 33.15%  github.com/ginuerzh/gosocks5.(*Conn).Read

附 pprof debug=1 结果

goroutine profile: total 4820
797 @ 0x430f6f 0x40798a 0x407960 0x40764b 0x9abfaf 0x97aedd 0x45e141
#   0x9abfae    github.com/ginuerzh/gost.transport+0xfe                 /root/gost/server.go:162
#   0x97aedc    github.com/ginuerzh/gost.(*tcpDirectForwardHandler).Handle+0x72c    /root/gost/forward.go:149

797 @ 0x430f6f 0x40798a 0x407960 0x40764b 0x9abfaf 0x9b55eb 0x9b3f28 0x45e141
#   0x9abfae    github.com/ginuerzh/gost.transport+0xfe             /root/gost/server.go:162
#   0x9b55ea    github.com/ginuerzh/gost.(*socks5Handler).handleConnect+0x126a  /root/gost/socks.go:923
#   0x9b3f27    github.com/ginuerzh/gost.(*socks5Handler).Handle+0x367      /root/gost/socks.go:812

797 @ 0x430f6f 0x42c67a 0x42bc66 0x4d1efb 0x4d2d0b 0x4d2cec 0x58a30f 0x59e809 0x468b82 0x9d9c55 0x9d9bfb 0x45e141
#   0x42bc65    internal/poll.runtime_pollWait+0x55     /snap/go/4520/src/runtime/netpoll.go:182
#   0x4d1efa    internal/poll.(*pollDesc).wait+0x9a     /snap/go/4520/src/internal/poll/fd_poll_runtime.go:87
#   0x4d2d0a    internal/poll.(*pollDesc).waitRead+0x19a    /snap/go/4520/src/internal/poll/fd_poll_runtime.go:92
#   0x4d2ceb    internal/poll.(*FD).Read+0x17b          /snap/go/4520/src/internal/poll/fd_unix.go:169
#   0x58a30e    net.(*netFD).Read+0x4e              /snap/go/4520/src/net/fd_unix.go:202
#   0x59e808    net.(*conn).Read+0x68               /snap/go/4520/src/net/net.go:177
#   0x468b81    io.copyBuffer+0x121             /snap/go/4520/src/io/io.go:402
#   0x9d9c54    io.CopyBuffer+0x194             /snap/go/4520/src/io/io.go:375
#   0x9d9bfa    github.com/ginuerzh/gost.transport.func2+0x13a  /root/gost/server.go:158

797 @ 0x430f6f 0x42c67a 0x42bc66 0x4d1efb 0x4d2d0b 0x4d2cec 0x58a30f 0x59e809 0x4767df 0x47693f 0x967f20 0x96a44c 0x96b400 0x96bb9f 0x9d7ac8 0x468b82 0x9d9a05 0x9d99ab 0x45e141
#   0x42bc65    internal/poll.runtime_pollWait+0x55             /snap/go/4520/src/runtime/netpoll.go:182
#   0x4d1efa    internal/poll.(*pollDesc).wait+0x9a             /snap/go/4520/src/internal/poll/fd_poll_runtime.go:87
#   0x4d2d0a    internal/poll.(*pollDesc).waitRead+0x19a            /snap/go/4520/src/internal/poll/fd_poll_runtime.go:92
#   0x4d2ceb    internal/poll.(*FD).Read+0x17b                  /snap/go/4520/src/internal/poll/fd_unix.go:169
#   0x58a30e    net.(*netFD).Read+0x4e                      /snap/go/4520/src/net/fd_unix.go:202
#   0x59e808    net.(*conn).Read+0x68                       /snap/go/4520/src/net/net.go:177
#   0x4767de    bufio.(*Reader).fill+0x10e                  /snap/go/4520/src/bufio/bufio.go:100
#   0x47693e    bufio.(*Reader).Peek+0x4e                   /snap/go/4520/src/bufio/bufio.go:138
#   0x967f1f    gopkg.in/gorilla/websocket%2ev1.(*Conn).read+0x3f       /root/go/pkg/mod/gopkg.in/gorilla/websocket.v1@v1.4.0/conn.go:357
#   0x96a44b    gopkg.in/gorilla/websocket%2ev1.(*Conn).advanceFrame+0x5b   /root/go/pkg/mod/gopkg.in/gorilla/websocket.v1@v1.4.0/conn.go:787
#   0x96b3ff    gopkg.in/gorilla/websocket%2ev1.(*Conn).NextReader+0x9f     /root/go/pkg/mod/gopkg.in/gorilla/websocket.v1@v1.4.0/conn.go:947
#   0x96bb9e    gopkg.in/gorilla/websocket%2ev1.(*Conn).ReadMessage+0x2e    /root/go/pkg/mod/gopkg.in/gorilla/websocket.v1@v1.4.0/conn.go:1028
#   0x9d7ac7    github.com/ginuerzh/gost.(*websocketConn).Read+0x117        /root/gost/ws.go:769
#   0x468b81    io.copyBuffer+0x121                     /snap/go/4520/src/io/io.go:402
#   0x9d9a04    io.CopyBuffer+0x194                     /snap/go/4520/src/io/io.go:375
#   0x9d99aa    github.com/ginuerzh/gost.transport.func1+0x13a          /root/gost/server.go:150

797 @ 0x430f6f 0x42c67a 0x42bc66 0x4d1efb 0x4d2d0b 0x4d2cec 0x58a30f 0x59e809 0x6029d0 0x4726ed 0x602c28 0x601115 0x605317 0x605309 0x67152f 0x468b82 0x9d9c55 0x9d9bfb 0x45e141
#   0x42bc65    internal/poll.runtime_pollWait+0x55     /snap/go/4520/src/runtime/netpoll.go:182
#   0x4d1efa    internal/poll.(*pollDesc).wait+0x9a     /snap/go/4520/src/internal/poll/fd_poll_runtime.go:87
#   0x4d2d0a    internal/poll.(*pollDesc).waitRead+0x19a    /snap/go/4520/src/internal/poll/fd_poll_runtime.go:92
#   0x4d2ceb    internal/poll.(*FD).Read+0x17b          /snap/go/4520/src/internal/poll/fd_unix.go:169
#   0x58a30e    net.(*netFD).Read+0x4e              /snap/go/4520/src/net/fd_unix.go:202
#   0x59e808    net.(*conn).Read+0x68               /snap/go/4520/src/net/net.go:177
#   0x6029cf    crypto/tls.(*atLeastReader).Read+0x5f       /snap/go/4520/src/crypto/tls/conn.go:761
#   0x4726ec    bytes.(*Buffer).ReadFrom+0xbc           /snap/go/4520/src/bytes/buffer.go:207
#   0x602c27    crypto/tls.(*Conn).readFromUntil+0xf7       /snap/go/4520/src/crypto/tls/conn.go:783
#   0x601114    crypto/tls.(*Conn).readRecordOrCCS+0x124    /snap/go/4520/src/crypto/tls/conn.go:590
#   0x605316    crypto/tls.(*Conn).readRecord+0x136     /snap/go/4520/src/crypto/tls/conn.go:558
#   0x605308    crypto/tls.(*Conn).Read+0x128           /snap/go/4520/src/crypto/tls/conn.go:1236
#   0x67152e    github.com/ginuerzh/gosocks5.(*Conn).Read+0x8e  /root/go/pkg/mod/github.com/ginuerzh/gosocks5@v0.2.0/conn.go:139
#   0x468b81    io.copyBuffer+0x121             /snap/go/4520/src/io/io.go:402
#   0x9d9c54    io.CopyBuffer+0x194             /snap/go/4520/src/io/io.go:375
#   0x9d9bfa    github.com/ginuerzh/gost.transport.func2+0x13a  /root/gost/server.go:158

796 @ 0x430f6f 0x42c67a 0x42bc66 0x4d1efb 0x4d2d0b 0x4d2cec 0x58a30f 0x59e809 0x6029d0 0x4726ed 0x602c28 0x601115 0x605317 0x605309 0x67152f 0x468b82 0x5aa91d 0x5aa8e7 0x5aa8a4 0x5a85be 0x468d5c 0x9d9a05 0x9d99ab 0x45e141
#   0x42bc65    internal/poll.runtime_pollWait+0x55     /snap/go/4520/src/runtime/netpoll.go:182
#   0x4d1efa    internal/poll.(*pollDesc).wait+0x9a     /snap/go/4520/src/internal/poll/fd_poll_runtime.go:87
#   0x4d2d0a    internal/poll.(*pollDesc).waitRead+0x19a    /snap/go/4520/src/internal/poll/fd_poll_runtime.go:92
#   0x4d2ceb    internal/poll.(*FD).Read+0x17b          /snap/go/4520/src/internal/poll/fd_unix.go:169
#   0x58a30e    net.(*netFD).Read+0x4e              /snap/go/4520/src/net/fd_unix.go:202
#   0x59e808    net.(*conn).Read+0x68               /snap/go/4520/src/net/net.go:177
#   0x6029cf    crypto/tls.(*atLeastReader).Read+0x5f       /snap/go/4520/src/crypto/tls/conn.go:761
#   0x4726ec    bytes.(*Buffer).ReadFrom+0xbc           /snap/go/4520/src/bytes/buffer.go:207
#   0x602c27    crypto/tls.(*Conn).readFromUntil+0xf7       /snap/go/4520/src/crypto/tls/conn.go:783
#   0x601114    crypto/tls.(*Conn).readRecordOrCCS+0x124    /snap/go/4520/src/crypto/tls/conn.go:590
#   0x605316    crypto/tls.(*Conn).readRecord+0x136     /snap/go/4520/src/crypto/tls/conn.go:558
#   0x605308    crypto/tls.(*Conn).Read+0x128           /snap/go/4520/src/crypto/tls/conn.go:1236
#   0x67152e    github.com/ginuerzh/gosocks5.(*Conn).Read+0x8e  /root/go/pkg/mod/github.com/ginuerzh/gosocks5@v0.2.0/conn.go:139
#   0x468b81    io.copyBuffer+0x121             /snap/go/4520/src/io/io.go:402
#   0x5aa91c    io.Copy+0x13c                   /snap/go/4520/src/io/io.go:364
#   0x5aa8e6    net.genericReadFrom+0x106           /snap/go/4520/src/net/net.go:614
#   0x5aa8a3    net.(*TCPConn).readFrom+0xc3            /snap/go/4520/src/net/tcpsock_posix.go:54
#   0x5a85bd    net.(*TCPConn).ReadFrom+0x4d            /snap/go/4520/src/net/tcpsock.go:103
#   0x468d5b    io.copyBuffer+0x2fb             /snap/go/4520/src/io/io.go:388
#   0x9d9a04    io.CopyBuffer+0x194             /snap/go/4520/src/io/io.go:375
#   0x9d99aa    github.com/ginuerzh/gost.transport.func1+0x13a  /root/gost/server.go:150

16 @ 0x430f6f 0x42c67a 0x42bc66 0x4d1efb 0x4d450a 0x4d44eb 0x58ac22 0x5ab012 0x5a9428 0x9abe1f 0x9ab93e 0xa4540a 0x45e141
#   0x42bc65    internal/poll.runtime_pollWait+0x55             /snap/go/4520/src/runtime/netpoll.go:182
#   0x4d1efa    internal/poll.(*pollDesc).wait+0x9a             /snap/go/4520/src/internal/poll/fd_poll_runtime.go:87
#   0x4d4509    internal/poll.(*pollDesc).waitRead+0x1b9            /snap/go/4520/src/internal/poll/fd_poll_runtime.go:92
#   0x4d44ea    internal/poll.(*FD).Accept+0x19a                /snap/go/4520/src/internal/poll/fd_unix.go:384
#   0x58ac21    net.(*netFD).accept+0x41                    /snap/go/4520/src/net/fd_unix.go:238
#   0x5ab011    net.(*TCPListener).accept+0x31                  /snap/go/4520/src/net/tcpsock_posix.go:139
#   0x5a9427    net.(*TCPListener).AcceptTCP+0x47               /snap/go/4520/src/net/tcpsock.go:247
#   0x9abe1e    github.com/ginuerzh/gost.tcpKeepAliveListener.Accept+0x2e   /root/gost/server.go:135
#   0x9ab93d    github.com/ginuerzh/gost.(*Server).Serve+0xcd           /root/gost/server.go:65
#   0xa45409    main.(*router).Serve+0x1f9                  /root/gost/cmd/gost/route.go:480

7 @ 0x430f6f 0x42c67a 0x42bc66 0x4d1efb 0x4d2d0b 0x4d2cec 0x58a30f 0x59e809 0x6029d0 0x4726ed 0x602c28 0x601115 0x605317 0x605309 0x67152f 0x4687d8 0x673fe4 0x9b28be 0x977e7b 0x977e12 0x977ae1 0x97aae6 0x45e141

..

重启程序后 list

(pprof) traces
File: gost
Type: goroutine
Time: Oct 12, 2019 at 1:26pm (CST)
-----------+-------------------------------------------------------
       815   runtime.gopark
             runtime.goparkunlock
             runtime.chanrecv
             runtime.chanrecv1
             github.com/ginuerzh/gost.transport
             github.com/ginuerzh/gost.(*tcpDirectForwardHandler).Handle
-----------+-------------------------------------------------------
       815   runtime.gopark
             runtime.netpollblock
             internal/poll.runtime_pollWait
             internal/poll.(*pollDesc).wait
             internal/poll.(*pollDesc).waitRead
             internal/poll.(*FD).Read
             net.(*netFD).Read
             net.(*conn).Read
             io.copyBuffer
             io.CopyBuffer
             github.com/ginuerzh/gost.transport.func2

......

(pprof) list github.com/ginuerzh/gost.transport
Total: 4920
ROUTINE ======================== github.com/ginuerzh/gost.transport in /root/gost/server.go
         0       1629 (flat, cum) 33.11% of Total
         .          .    157:
         .          .    158:           _, err := io.CopyBuffer(rw2, rw1, buf)
         .          .    159:           errc <- err
         .          .    160:   }()
         .          .    161:
         .       1629    162:   err := <-errc
         .          .    163:   if err != nil && err == io.EOF {
         .          .    164:           err = nil
         .          .    165:   }
         .          .    166:   return err
         .          .    167:}
ROUTINE ======================== github.com/ginuerzh/gost.transport.func1 in /root/gost/server.go
         0       1629 (flat, cum) 33.11% of Total
         .          .    145:   errc := make(chan error, 1)
         .          .    146:   go func() {
         .          .    147:           buf := lPool.Get().([]byte)
         .          .    148:           defer lPool.Put(buf)
         .          .    149:
         .       1629    150:           _, err := io.CopyBuffer(rw1, rw2, buf)
         .          .    151:           errc <- err
         .          .    152:   }()
         .          .    153:
         .          .    154:   go func() {
         .          .    155:           buf := lPool.Get().([]byte)
ROUTINE ======================== github.com/ginuerzh/gost.transport.func2 in /root/gost/server.go
         0       1629 (flat, cum) 33.11% of Total
         .          .    153:
         .          .    154:   go func() {
         .          .    155:           buf := lPool.Get().([]byte)
         .          .    156:           defer lPool.Put(buf)
         .          .    157:
         .       1629    158:           _, err := io.CopyBuffer(rw2, rw1, buf)
         .          .    159:           errc <- err
         .          .    160:   }()
         .          .    161:
         .          .    162:   err := <-errc
         .          .    163:   if err != nil && err == io.EOF {
(
f4nff commented 5 years ago

同样遇到, image 一个不足20个连接数,几乎无流量,占用内存就这么大, golang1.13.4

ginuerzh commented 5 years ago

具体的用法是怎样的？客户端和服务端的命令行参数?

f4nff commented 5 years ago

443