WASM: OOM when using regexp.ReplaceAllString

[life-learner1]

Describe the bug / error

The memory keeps on growing , until OOM. We find many 401 before pods restart .after deleting the wasmplugin, it recovers

I tried build wasm with following commands, but they didn't work. it seems like a GC issue, after I remove the code about 'regexp.ReplaceAllString' , it works fine, no OOM

1. tinygo build -o ratelimit.wasm -scheduler=none -target=wasi ./main.go
2. tinygo build -o ratelimit.wasm -scheduler=none -target=wasi -gc=conservative -print-allocs=. ./main.go

2022-11-11T09:25:59.177343Z error   envoy lua   script log: 401 Debugging Main #1
2022-11-11T09:25:59.177423Z error   envoy lua   script log: [string "----- lib begin ---..."]:162: attempt to concatenate local 's' (a nil value)
2022-11-11T09:25:59.177447Z error   envoy lua   script log: 401 Debugging Main #8
2022-11-11T09:25:59.177447Z error   envoy lua   script log: [string "----- lib begin ---..."]:162: attempt to concatenate local 's' (a nil value)
2022-11-11T09:25:59.177468Z error   envoy lua   script log: 401 Debugging Main #8
2022-11-11T09:25:59.177481Z warning envoy lua   script log: logtxt in http response:8e9eed1e-b310-9e9e-aad0-b064c21c829f
2022-11-11T09:25:59.177615Z warning envoy lua   script log: logtxt in http response:a8e8d3ba-e5d9-9e41-b963-b45673ccf63a
2022-11-11T09:25:59.191452Z critical    envoy backtrace #23: event_base_loop [0x55738b2acc31]
2022-11-11T09:25:59.206848Z critical    envoy backtrace #24: Envoy::Server::InstanceImpl::run() [0x55738aafe19b]
2022-11-11T09:25:59.220814Z critical    envoy backtrace #25: Envoy::MainCommonBase::run() [0x557389719264]
2022-11-11T09:25:59.233984Z critical    envoy backtrace #26: Envoy::MainCommon::main() [0x5573897199c6]
2022-11-11T09:25:59.247644Z critical    envoy backtrace #27: main [0x557389715c3c]
2022-11-11T09:25:59.247885Z critical    envoy backtrace #28: __libc_start_main [0x7fe847493083]
AsyncClient 0x55739311cf00, stream_id_: 8650725868373510219
&stream_info_: 
  StreamInfoImpl 0x55739311d0d0, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: 1, health_check_request_: 0, route_name_:   upstream_info_: 
    UpstreamInfoImpl 0x55738daee498, upstream_connection_id_: 1042778
Http2::ConnectionImpl 0x557392b6d870, max_headers_kb_: 60, max_headers_count_: 100, per_stream_buffer_limit_: 268435456, allow_metadata_: 0, stream_error_on_invalid_http_messaging_: 0, is_outbound_flood_monitored_control_frame_: 0, dispatching_: 1, raised_goaway_: 0, pending_deferred_reset_streams_.size(): 0
&protocol_constraints_: 
  ProtocolConstraints 0x557392b6d8f0, outbound_frames_: 0, max_outbound_frames_: 10000, outbound_control_frames_: 0, max_outbound_control_frames_: 1000, consecutive_inbound_frames_with_empty_payload_: 0, max_consecutive_inbound_frames_with_empty_payload_: 1, opened_streams_: 1, inbound_priority_frames_: 0, max_inbound_priority_frames_per_stream_: 100, inbound_window_update_frames_: 19, outbound_data_frames_: 18, max_inbound_window_update_frames_per_data_frame_sent_: 10
Number of active streams: 1, current_stream_id_: 1 Dumping current stream:
stream: 
  ConnectionImpl::StreamImpl 0x557390899400, stream_id_: 1, unconsumed_bytes_: 0, read_disable_count_: 0, local_end_stream_: 0, local_end_stream_sent_: 0, remote_end_stream_: 0, data_deferred_: 1, received_noninformational_headers_: 1, pending_receive_buffer_high_watermark_called_: 0, pending_send_buffer_high_watermark_called_: 0, reset_due_to_messaging_error_: 0, cookies_:   pending_trailers_to_encode_:   null
  absl::get<ResponseHeaderMapPtr>(headers_or_trailers_):   null
Dumping corresponding downstream request for upstream stream 1:
  UpstreamRequest 0x5573928f1400
2022-11-11T09:25:59.342409Z info    ads ADS: "@" istio-ingressgateway-ccc6bc794-ws5wh.istio-system-2 terminated rpc error: code = Canceled desc = context canceled
2022-11-11T09:25:59.342407Z info    ads ADS: "@" istio-ingressgateway-ccc6bc794-ws5wh.istio-system-1 terminated rpc error: code = Canceled desc = context canceled
2022-11-11T09:25:59.344102Z error   Epoch 0 exited with error: signal: segmentation fault
2022-11-11T09:25:59.344152Z info    No more active epochs, terminating

What is your Envoy/Istio version?

docker.io/istio/proxyv2:1.13.5

What is the SDK version?

b2ee162e7b5bb26ed20dadfcbc98170527273ffb

What is your TinyGo version?

tinygo:0.26.0

URL or snippet of your code including Envoy configuration

package main

import (
    "fmt"
    "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm"
    "github.com/tetratelabs/proxy-wasm-go-sdk/proxywasm/types"
    "github.com/tidwall/gjson"
    "regexp"
    "strings"
)

func main() {
    // SetVMContext is the entrypoint for setting up this entire Wasm VM.
    // Please make sure that this entrypoint be called during "main()" function, otherwise
    // this VM would fail.
    proxywasm.SetVMContext(&vmContext{})
}

// vmContext implements types.VMContext interface of proxy-wasm-go SDK.
type vmContext struct {
    // Embed the default VM context here,
    // so that we don't need to reimplement all the methods.
    types.DefaultVMContext
}

// Override types.DefaultVMContext.
func (*vmContext) NewPluginContext(contextID uint32) types.PluginContext {
    return &pluginContext{}
}

// pluginContext implements types.PluginContext interface of proxy-wasm-go SDK.
type pluginContext struct {
    // Embed the default plugin context here,
    // so that we don't need to reimplement all the methods.
    types.DefaultPluginContext
    configuration pluginConfiguration
}

// pluginConfiguration is a type to represent an example configuration for this wasm plugin.
type pluginConfiguration struct {
    // Example configuration field.
    // The plugin will validate if those fields exist in the json payload.
    //projectLevelMap map[string]string
    uuidRe *regexp.Regexp
    uidRe  *regexp.Regexp
}

// Override types.DefaultPluginContext.
func (ctx *pluginContext) OnPluginStart(pluginConfigurationSize int) types.OnPluginStartStatus {
    data, err := proxywasm.GetPluginConfiguration()
    if err != nil && err != types.ErrorStatusNotFound {
        proxywasm.LogCriticalf("error reading plugin configuration: %v", err)
        return types.OnPluginStartStatusFailed
    }
    config, err := parsePluginConfiguration(data)
    if err != nil {
        proxywasm.LogCriticalf("error parsing plugin configuration: %v", err)
        return types.OnPluginStartStatusFailed
    }
    ctx.configuration = config
    return types.OnPluginStartStatusOK
}

// parsePluginConfiguration parses the json plugin confiuration data and returns pluginConfiguration.
// Note that this parses the json data by gjson, since TinyGo doesn't support encoding/json.
// You can also try https://github.com/mailru/easyjson, which supports decoding to a struct.
func parsePluginConfiguration(data []byte) (pluginConfiguration, error) {
    if len(data) == 0 {
        return pluginConfiguration{}, nil
    }

    config := &pluginConfiguration{}
    if !gjson.ValidBytes(data) {
        return pluginConfiguration{}, fmt.Errorf("the plugin configuration is not a valid json: %q", string(data))
    }

    //jsonData := gjson.ParseBytes(data)
    //requiredKeys := jsonData.Get("requiredKeys").Array()
    //for _, requiredKey := range requiredKeys {
    //  config.requiredKeys = append(config.requiredKeys, requiredKey.Str)
    //}
    //config.projectLevelMap = projectLevelMap
    config.uuidRe = regexp.MustCompile(`\w{8}(-\w{4}){3}-\w{12}`)
    config.uidRe = regexp.MustCompile(`[\da-z]{32}`)

    return *config, nil
}

// Override types.DefaultPluginContext.
func (ctx *pluginContext) NewHttpContext(contextID uint32) types.HttpContext {
    return &payloadValidationContext{
        //projectLevelMap: ctx.configuration.projectLevelMap,
        uuidRe: ctx.configuration.uuidRe,
        uidRe:  ctx.configuration.uidRe,
    }
}

// payloadValidationContext implements types.HttpContext interface of proxy-wasm-go SDK.
type payloadValidationContext struct {
    // Embed the default root http context here,
    // so that we don't need to reimplement all the methods.
    types.DefaultHttpContext
    totalRequestBodySize int
    uuidRe               *regexp.Regexp
    uidRe                *regexp.Regexp
}

var _ types.HttpContext = (*payloadValidationContext)(nil)

// Override types.DefaultHttpContext.
func (ctx *payloadValidationContext) OnHttpRequestHeaders(numHeaders int, _ bool) types.Action {

    // ActionContinue lets the host continue the processing the body.
    _ = proxywasm.AddHttpRequestHeader("ratelimit-source-ip", "remoteIp")
    path, _ := proxywasm.GetHttpRequestHeader(":path")
    pathArr := strings.Split(path, "?")
    realPath := pathArr[0]
    realPath = ctx.uidRe.ReplaceAllString(realPath, "{uid}")
    //yeah := ctx.uidRe.MatchString(realPath)
    realPath = ctx.uuidRe.ReplaceAllString(realPath, "{uuid}")
    proxywasm.LogInfof("ratelimit-path: %s")
    _ = proxywasm.AddHttpRequestHeader("ratelimit-path", realPath)
    return types.ActionContinue
}

// Override types.DefaultHttpContext.
func (ctx *payloadValidationContext) OnHttpRequestBody(bodySize int, endOfStream bool) types.Action {
    ctx.totalRequestBodySize += bodySize
    if !endOfStream {
        // OnHttpRequestBody may be called each time a part of the body is received.
        // Wait until we see the entire body to replace.
        return types.ActionPause
    }
    return types.ActionContinue
}

Additional context (Optional)

[life-learner1]

related to this issue: https://github.com/tetratelabs/proxy-wasm-go-sdk/issues/340

[dgryski]

Can you try to reproduce this on wasmtime or another wasm interpreter without all the envoy SDK extras, just using the regexp package?