starwing / lua-protobuf

A Lua module to work with Google protobuf
MIT License
1.73k stars 387 forks source link

the memory keeps rising when the library is used in openresty #163

Closed wedvefv closed 2 years ago

wedvefv commented 2 years ago

i was used it like this:

` -- start assert(pb.loadfile("/usr/local/openresty/lualib/pb_file/my_test.pb")) .... res = assert(pb.encode("com.meitu.openrtb.BidResponse", BidResponse)) -- end

`

in the product environment, after 4hours,the memrory will be 70% ; when i reload the nginx, the memory is Exhausted and the server can‘t login with ssh; but before when i use old library protoc-gen-lua then memory will keep in 10% all the time。

wedvefv commented 2 years ago

As we all known when we open nginx's lua code cache #lua_code_cache off;,
it will load the module (c module or lua module) only once when we reload the nginx;
the loaded infomation will store the into memory and every request will shared the cached module.(every work process has one luaVM to deal request, so the cache is only shared on this nginx work process) so if you modify the module's variable or memory malloc in the module then next request will be affected; for lua-protobuf i think pb.loadfile function malloc memory for every request so the server's momory keeps rising...

Two ways to deal with :

use the openresty's one feature: lua module variable will be cached and shared between requests。

  1. wrote in lua module like this:
    
    --  /usr/local/openresty/lualib/pb_module.lua
    local pb = require "pb"
    local _M = {}

_M.pb = nil function _M.new(self, type) if _M.pb == nil then pb.loadfile("/usr/local/openresty/lualib/pb_file/test1.pb") pb.loadfile("/usr/local/openresty/lualib/pb_file/test2.pb") -- pb support more then one pb file _M.pb = pb end

return _M.pb

end

return _M

the use the module in *xxxx_by_lua_file*
```lua
-- /usr/local/openresty/nginx/test_pb.lua

local pb = require("pb_module"):new() 
...
-- pb_message_type1 is message type
ngx.req.read_body()
local postdata = ngx.req.get_body_data()
local postinfo = assert(pb.decode("pb_message_type1", postdata))
  1. if you don't want to lua wite module, you can do it like :
-- /usr/local/openresty/nginx/test_pb.lua

...
-- `pb_message_type1` is message type

--  
local pb = require("pb")
local type_request = pb.type("pb_message_type1")
-- print(type_request)
if not type_request then
    --print("------------init lua--------------")
    pb.loadfile("/usr/local/openresty/lualib/pb_file/test1.pb")
end

ngx.req.read_body()
local postdata = ngx.req.get_body_data()
local postinfo = assert(pb.decode("pb_message_type1", postdata))

the require("pb") also will be cached , you can check the message type exists or not. if the pb file loaded, you can find it by pb.type() function.