Envoy resty framework like OpenResty

[zengyuxing007]

Envoy extension is a threshold for developers who do not have C++ foundation.

Although the community currently has a wasm extension development method, but it currently supports relatively few languages (C++, Rust, assembly scripts),using these languages to extend development is still not that simple.

Reverse proxy software with functions similar to Envoy, we know that there is Nginx. Based on Nginx, there is a very famous framework OpenResty. Of course, there are many other well-known implementations based on Openresty, such as Kong, apisix, etc.

OpenResty can use Lua to extend the development to support multiple plugins. Although the community Envoy version currently supports Lua extension, but its extension functions and methods are relatively weak, for example, it does not support multiple Lua plugins, and does different levels of configuration like native c++ filter, envoy.lua is similar to serverless dynamically executing a script.

Can we refer to Openresty to implement a similar Envoy Resty? @mattklein123 @lizan

You can view the complete proposal document

[mattklein123]

Cool stuff! I think my main question is whether this needs to live in the main Envoy repo? Could enough functionality be added to the existing Lua filter that then the rest of the framework could be hosted in a different repo?

[zengyuxing007]

Cool stuff! I think my main question is whether this needs to live in the main Envoy repo? Could enough functionality be added to the existing Lua filter that then the rest of the framework could be hosted in a different repo?

envoy.resty design goal is similar to openresty, features include lua plugin configuration and lua plugin chain management , itself somewhat similar to the http_connection_manager, of course, there are some other similar lua-ffi, cosocket(coroutine + socket) basic capabilities such as

I think that envoy.lua is well positioned (serverless function), which is different from the envoy.resty oriented scenario, so I suggest designing envoy.resty as a separate filter. The rest lua pdk framwork part, as you said, needs to be hosted in a single repo.

[mattklein123]

I’m not sure I follow why we can’t add whatever hooks you need to the lua filter. These hooks would likely help anyone else that uses the lua filter. Can you outline what the lua filter would need to do differently? cc @dio.

[zengyuxing007]

I’m not sure I follow why we can’t add whatever hooks you need to the lua filter. These hooks would likely help anyone else that uses the lua filter. Can you outline what the lua filter would need to do differently? cc @dio.

@mattklein123

Extending the features described in the proposal to envoy.lua is of course possible， but it will import some dependencies, which is what I said above about envoy.lua's positioning and corresponding solution scenarios.

When constructing an envoy.resty filter, the lua vm based on the aggra framework is constructed, i.e., it relies on aggra

aggra lua framework will be commonly used http envoy api through luajit ffi do a layer of wrapping , using ffi mainly for better performance.

Of course, aggra lua framework will also do secondary wrappers for some commonly used functions, such as get_downstream_local_address, get_user_agent and so on. Because of these, users of the envoy.lua filter will need to be aware of the existence of aggra framework, and understand the relevant interface of its wrappers. What I mean is that if envoy.lua is positioned as a serverless function, does it need to be relatively simple and not import aggra dependencies.

[mattklein123]

When constructing an envoy.resty filter, the lua vm based on the aggra framework is constructed, i.e., it relies on aggra

What is the aggra framework? Is it Lua code or C++ code?

I feel pretty strongly that we make the existing Lua filter do what you need to do on the C++ side to support the more expressive Lua framework on top. Can we list out what changes would be needed and we can see what that would look like? It could potentially be an option on the existing Lua filter but it's not clear to me that is required.

[dio]

Sorry if this sidetracks the discussion. Just throwing some ideas and perspectives.

To make it to have the same vein with OpenResty, we need something similar to https://github.com/openresty/lua-nginx-module (and https://github.com/openresty/stream-lua-nginx-module if we want to deal with TCP), which to some extent what current Envoy Lua HTTP filter trying to do (also it has a similar approach to Kong's https://docs.konghq.com/2.1.x/plugin-development/custom-logic/#available-contexts).

Looking at the proposed config, I think we can extend this with the capability to inspect a dir and load *.lua plugins inside that dir and initialize each of them with some "context".

FWIW, If we want to have more "extensibility points" to Envoy via Lua, I think we can leverage the proxy-wasm interface here: https://github.com/proxy-wasm/spec/tree/master/abi-versions/vNEXT to extend the current Lua support on extending Envoy. Hence it will be consistent with the other efforts for Envoy extensibility.

[zengyuxing007]

When constructing an envoy.resty filter, the lua vm based on the aggra framework is constructed, i.e., it relies on aggra

What is the aggra framework? Is it Lua code or C++ code?

I feel pretty strongly that we make the existing Lua filter do what you need to do on the C++ side to support the more expressive Lua framework on top. Can we list out what changes would be needed and we can see what that would look like? It could potentially be an option on the existing Lua filter but it's not clear to me that is required.

Aggra is a lua framework similar to kong, which wraps the common related scaffolding functions.

[caitong93]

Can we list out what changes would be needed and we can see what that would look like? It could potentially be an option on the existing Lua filter but it's not clear to me that is required.

@mattklein123 Thank for your response and suggestions ! @zengyuxing007 and I have some conversation , and we agreed that it would be a good start to extend the existing Lua filter, will update the doc with more details.

[caitong93]

@dio

To make it to have the same vein with OpenResty, we need something similar to https://github.com/openresty/lua-nginx-module (and https://github.com/openresty/stream-lua-nginx-module if we want to deal with TCP), which to some extent what current Envoy Lua HTTP filter trying to do (also it has a similar approach to Kong's https://docs.konghq.com/2.1.x/plugin-development/custom-logic/#available-contexts).

Yes, actually there is an internally implementation(we name it EnvoyResty) following the OpenResty design, and we have run it on production for more than half year. Now we hope to contribute this feature to upstream.

FWIW, If we want to have more "extensibility points" to Envoy via Lua, I think we can leverage the proxy-wasm interface here: https://github.com/proxy-wasm/spec/tree/master/abi-versions/vNEXT to extend the current Lua support on extending Envoy. Hence it will be consistent with the other efforts for Envoy extensibility.

wasm is definitely the future, but from our experience, it's still a while from production availability. I think the two can go together. Many users want to migrate to envoy-based cloud native api gateways, but they have many Lua plugins based on OpenResty. At NetEase, we use EnvoyResty to migrate Lua plugins. At the same time we track the development of wasm filter, hopefully we can contribute to it someday.

[dio]

@caitong93 sounds good. Re: leveraging the proxy-wasm interface, I meant to say about adopting the extension points defined in that vNEXT doc.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or other activity occurs. Thank you for your contributions.

[caitong93]

leveraging the proxy-wasm interface, I meant to say about adopting the extension points defined in that vNEXT doc.

@dio sorry for late response, I think it is very reasonable.

Based on current Lua filter, I list some key points here:

Load Lua scripts(name it envoy.core here) during the Lua VM initialization and initialize the global variables

This process initializes envoy global module and its submodules. The user's Lua code can call Lua C API, or C++ code wrapped in C functions using the FFI library via the envoy module.

envoy.core also defines two global entry points: envoy_plugin_on_request(plugin_name, request_handle, config) and envoy_plugin_on_response(plugin_name, request_handle, config) to invoke specific plugin.

Following is an example of handler.lua of a plugin that shows how to use envoy.* APIs

local envoy = envoy

-- the file.log api takes advantage of envoy's ability to write file logs asynchronously
local filelog = envoy.file.log

local _M = {}

function _M:on_request(request_handle, config)
   -- envoy.ctx stores variables that can be used in the request lifetime
   envoy.ctx.start_time = envoy.now_ms()
end

function _M:on_response(response_handle, config)
   local elapsed = envoy.now_ms() - envoy.ctx.start_time
   filelog(config.filename, "elapsed time(in ms): "..elapsed)
end

return _M

Utilize LuaJIT FFI

In Envoy Lua filter, add a C++ class/struct FilterContext to expose needed functionality, set pointer of this object as a global lightweight userdata(lightweight userdata is broken in arm64, need a solution) in Lua. envoy.* apis will get the pointer and pass it to C functions with other arguments through FFI. C functions receiving void * as the first argument, converts it to FilterContext*, and call its methods.

see https://github.com/openresty/lua-resty-core/blob/v0.1.19/lib/resty/core/request.lua#L89 , an an equivalent implementation for envoy is

ffi.cdef[[
    int envoy_ffi_req_get_headers(FilterContext *r, envoy_lua_ffi_table_elt_t *out, int count);
]]

function envoy.req.get_headers(max_headers)
   -- get_request may simply return _G.__filter_context, __filter_context is set to FilterContext*  by lua filter
   local r = get_request()

   C.envoy_ffi_req_get_headers(r, out, max_headers)
end

It's very ease to add new API via FFI, and the performance is great.

Plugin Configuration

Some consideration points are listed as following:

• define a schema and validation rule
• cache the validation result for performance
• each plugin contains a handler.lua and schema.lua

compatible with current APIs

The plugin feature is optional at runtime.

Draft config:

name: envoy.filters.http.lua
typed_config:
  "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
  scripts_search_path: ["/opt/envoyresty"]
  # core libraries which initialize the `envoy` module
  preload_modules: ["envoy.core"]
  # an option to enable the plugin feature
  enable_plugins: true
  plugins:
  # plugins are searched from `scripts_search_path`
  - name: ip-restriction
    config: {}

per_filter_config:
  envoy.filters.http.lua:
    plugins:
    - name: ip-restriction
      config: {}

cc @mattklein123 @zengyuxing007 @wbpcode

[mattklein123]

@caitong93 can you further clarify the changes to the Lua filter that need to live in the Envoy repo? As much as possible of this should live outside the Envoy repo, so I would like to understand specifically what changes will need to be made (please describe them independently of resty if you can).

[caitong93]

Yes, I will try to pull out some generic logic here:

Load Lua scripts during the Lua VM initialization

New options in Lua filter config:

repeated string scripts_search_path : path to search Lua scripts/modules

repeated string preload_modules: absolute/relative path to Lua scripts/modules

Expose envoy's internal interfaces via C functions so that Lua code can call them via FFI

For example, pseudocode for getting request header using FFI:

add a class FilterContext(initialized by Lua filter) and a C wrapper function

typedef struct {
  int len;
  const char *data;
} str_slice;

class FilterContext {
  int getRequestHeader(const char* key, size_t key_len,  str_slice* value)
}

int envoy_lua_get_request_header(FilterContext* context, const char* key, size_t key_len,  str_slice* value) {
  return context->(key, key_len, value);
}

inject pointer of FilterContext as a global lightweight userdata before start Lua coroutine

static void setFilterContext(lua_State *L, FilterContext *r) {
  lua_pushlightuserdata(L, static_cast<void *>(r));
  lua_setglobal(L, "__filter_context");
}

in Lua code, invoke envoy_lua_get_request_header to get header

-- declare C function to use it
ffi.cdef[[
    int envoy_lua_get_request_header(FilterContext *r, const char* key, size_t key_len,  str_slice* value);
]]

-- get pointer of FilterContext
local function get_context()
  return getfenv(0).__filter_context
end

function get_header(key)
  local context = get_context()
  local value = ffi_new("str_slice[1]")

  C.envoy_lua_get_request_header(context, key, #key, value)

  -- convert to a Lua string
  return ffi_str(value[0].data, value[0].len)
end

[mattklein123]

Can you explain why we need a new C interface via FFI? Don't the existing bindings/APIs work or we can add new bindings as needed?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[bochuxt]

Envoy extension is a threshold for developers who do not have C++ foundation.

Although the community currently has a wasm extension development method, but it currently supports relatively few languages (C++, Rust, assembly scripts),using these languages to extend development is still not that simple.

Reverse proxy software with functions similar to Envoy, we know that there is Nginx. Based on Nginx, there is a very famous framework OpenResty. Of course, there are many other well-known implementations based on Openresty, such as Kong, apisix, etc.

OpenResty can use Lua to extend the development to support multiple plugins. Although the community Envoy version currently supports Lua extension, but its extension functions and methods are relatively weak, for example, it does not support multiple Lua plugins, and does different levels of configuration like native c++ filter, envoy.lua is similar to serverless dynamically executing a script.

Can we refer to Openresty to implement a similar Envoy Resty? @mattklein123 @lizan

You can view the complete proposal document

About A Lua framework that support Apache APISIX plugins run directly in Envoy Lua filter without modify Envoy.

origin https://github.com/api7/envoy-apisix.git (fetch)

[caitong93]

Can you explain why we need a new C interface via FFI? Don't the existing bindings/APIs work or we can add new bindings as needed?

There several advantages to use FFI binding:

• Lua call C function without C API overhead, arguments and return values using C data, much more efficient (according to benchmark, a simple Lua plugin that get and set some headers can achieve 80% performance of a C++ plugin with some logic)
• less work to add new APIs

To conclude this issue, I think these features can be added to Envoy as needed and be tracked in new issues. We(at NetEase) are planning to open source the internal code as a separate project.

Thanks! I think it's clear now.