wasm_fail_stream error && RuntimeError

[withlin]

Version Infos

• istio/proxyv2 - 1.13.7 • envoy - v1.21.5

Envoy Error

envoy return wasm_fail_stream 503 code.

Envoy Log

error envoy wasm Function: proxy_on_context_create failed: Uncaught RuntimeError: unreachable error envoy wasm Function: proxy_on_log failed: Uncaught RuntimeError: unreachable error failed scraping envoy metrics: error reading http://localhost:15090/stats/prometheus: unexpected EO

[daixiang0]

proxy_on_context_create failed: Uncaught RuntimeError: unreachable

I think something wrong with your WASM code, you can learn from https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/wasm-cc.

Please share more details like WASM runtime, WASM binary/code and Envoy config if possible.

[withlin]

@daixiang0 we not use third part WASM code, envoy its.

[phlax]

@withlin its not really clear from your report what the issue is here

as a side note envoy v1.21.x is not a supported Envoy or Istio/Envoy version (https://istio.io/latest/docs/releases/supported-releases/#supported-envoy-versions)

[withlin]

@phlax This is a small mistake on my part, I wrote it wrong in this version. the istio version is 1.13.7

[withlin]

@phlax @daixiang0 hello, i only enable envoy access_loggers : expression: response.code >= 500 and istio.stats, they are officialWASM code(fliter) .

- accessLog:
  - filter:
      andFilter:
        filters:
        - responseFlagFilter:
            flags:
            - NR
        - extensionFilter:
            name: envoy.access_loggers.extension_filters.cel
            typedConfig:
              '@type': type.googleapis.com/envoy.extensions.access_loggers.filters.cel.v3.ExpressionFilter
              expression: response.code >= 500
    name: envoy.access_loggers.file
    typedConfig:
      '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      logFormat:
        textFormatSource:
          inlineString: |
            [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%
      path: /dev/stdout
  address:
    socketAddress:
      address: 0.0.0.0
      portValue: 8080
  filterChains:
  - filters:
    - name: envoy.filters.network.http_connection_manager
      typedConfig:
        '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
        accessLog:
        - filter:
            extensionFilter:
              name: envoy.access_loggers.extension_filters.cel
              typedConfig:
                '@type': type.googleapis.com/envoy.extensions.access_loggers.filters.cel.v3.ExpressionFilter
                expression: response.code >= 500
          name: envoy.access_loggers.file
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            logFormat:
              textFormatSource:
                inlineString: |
                  [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS% "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%
            path: /dev/stdout
        forwardClientCertDetails: SANITIZE_SET
        generateRequestId: false
        httpFilters:
        - name: istio.metadata_exchange
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            config:
              configuration:
                '@type': type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange
              vmConfig:
                code:
                  local:
                    inlineString: envoy.wasm.metadata_exchange
                runtime: envoy.wasm.runtime.null
        - name: envoy.filters.http.fault
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault
        - name: envoy.filters.http.cors
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors
        - name: istio.stats
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            config:
              configuration:
                '@type': type.googleapis.com/google.protobuf.StringValue
                value: '{"disable_host_header_fallback":true,"metrics":[{"dimensions":{"request_host":"request.host"},"name":"request_messages_total"},{"dimensions":{"request_host":"request.host"},"name":"response_messages_total"},{"dimensions":{"request_host":"request.host"},"name":"requests_total"},{"dimensions":{"request_host":"request.host"},"name":"request_duration_milliseconds"},{"dimensions":{"request_host":"request.host"},"name":"request_bytes"},{"dimensions":{"request_host":"request.host"},"name":"response_bytes"},{"dimensions":{"request_host":"request.host"},"name":"tcp_connections_closed_total"},{"dimensions":{"request_host":"request.host"},"name":"tcp_connections_opened_total"},{"dimensions":{"request_host":"request.host"},"name":"tcp_received_bytes_total"},{"dimensions":{"request_host":"request.host"},"name":"tcp_sent_bytes_total"}]}'
              rootId: stats_outbound
              vmConfig:
                code:
                  local:
                    inlineString: envoy.wasm.stats
                runtime: envoy.wasm.runtime.null
                vmId: stats_outbound
        - name: istio.stats
          typedConfig:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            typeUrl: type.googleapis.com/env

[withlin]

Grafana Monitor

Strange phenomenon: as qps gets higher and higher(200k QPS), so does its memory. pod memory limit is 6GB.

[withlin]

any update? :(

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[withlin]

help?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.