openresty / openresty

High Performance Web Platform Based on Nginx and LuaJIT
https://openresty.org
Other
12.46k stars 1.53k forks source link

Tcp reverse proxy by openresty+lua can not transfter data after link established? #724

Open zhangwenjian opened 3 years ago

zhangwenjian commented 3 years ago

My TCP-based protocol needs to use openresty+lua as a gateway. After tcp client establishes a link with the gateway, the first 1536 bytes packet will tell the destination server address. The lua script parses and obtains the server address in preread_by_lua_block, and sets it to the ngx.ctx.hsr_server_ip variable. Then I set the server address as upstream in balancer_by_lua_block through balancer.set_current_peer. In actual running, the step of set_current_peer is executed, the final tcp server behind the gateway also received the connect request, the tcp link was successfully established. But the subsequent tcp data sent by the client was never transferred to the server. If I sleep 1 seconds after first packet sent to gateway in client code, then server can receive data later.

`events { worker_connections 1024; }

stream { lua_code_cache on;

init_by_lua_block {
    require('ngx.balancer')
}

upstream lua_dispatcher {
    # just an invalid address as a placeholder
    server 0.0.0.1:1234;

    balancer_by_lua_block {
        local balancer = require('ngx.balancer')
        local host = ngx.ctx.hsr_server_ip
        local port = ngx.ctx.hsr_server_port

        -- set peer
        local ok, err = balancer.set_current_peer(host, port)
        if ok then
          ngx.log(ngx.NOTICE, "Forward a client to server: ", host)
        else
          ngx.log(ngx.ERR, err)
          ngx.exit(ngx.ERROR)
        end
    }
}

# proxy
server {
    listen 5701;
    proxy_pass lua_dispatcher;

    # Parse HSR Server IP from first packet
    preread_by_lua_block {
        local sock = ngx.req.socket()
        local data, err = sock:receive(1536)
        if not data then
        ngx.log(ngx.ERR, err)
        ngx.exit(ngx.ERROR)
        end

        local datastr = tostring(data)

        -- get x.x.x.x from first tcp packet with length of 1536
        local ipStrTable = {}
        local tempStr = '\0'
        for i=1,#datastr do
          tempStr = string.sub(datastr,i,i)
          if tonumber(tempStr) or tempStr == "." then
              table.insert(ipStrTable, tempStr)
          end
        end

        -- save upstream server ip to ngx.ctx
        local ipstr = table.concat(ipStrTable)
        ngx.ctx.hsr_server_ip = ipstr
        ngx.ctx.hsr_server_port = 5701
        ngx.log(ngx.NOTICE, "Connect to hsr server: " ,ngx.ctx.hsr_server_ip)
    }
}

}`

zhuizhuhaomeng commented 3 years ago

this is a bug in the stream module. for now, you can wait for a reply in the client side and send reply in preread_by_lua_block.