search

corazawaf / coraza-proxy-wasm

proxy-wasm filter based on Coraza WAF
Apache License 2.0
115 stars 24 forks source link

[segmentation fault] response_payload_too_large #280

Open YvesZelros opened 5 months ago

YvesZelros commented 5 months ago

I am facing to a segmentation fault on a simple GET request to an static PNG image of 3Mb (3731858 bytes)

The issue is reproductible on each request to this image.

As I import recommended-conf, I don't expect that png response are scaned, only text/plain text/html text/xml should be scan with the recommended conf.

Then the size of the response must not be an issue here, not more generated a segmentation fault.

Envoy configuration =>

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
spec:
  imagePullPolicy: IfNotPresent
  match:
  - mode: SERVER
    ports:
    - number: 8080
  phase: AUTHN
  pluginConfig:
    default_directives: default
    directives_map:
      default:
      - Include @recommended-conf
      - SecDebugLogLevel 3
      - SecRuleEngine DetectionOnly
      - SecRequestBodyAccess On
      - SecResponseBodyLimitAction ProcessPartial
      - Include @crs-setup-conf
      - SecAction "id:900230,phase:1,pass,t:none,nolog,setvar:'tx.allowed_http_versions=HTTP/1.1
        HTTP/2 HTTP/2.0'"
      - SecAction "id:900220,phase:1,pass,t:none,nolog,setvar:'tx.allowed_request_content_type=|text/plain|
        |application/x-www-form-urlencoded| |multipart/form-data| |multipart/related|
        |application/json|'"
      - SecRule REQUEST_URI "@beginsWith /auth/metrics" "id:1000,phase:1,pass,nolog,ctl:ruleRemoveById=920350"
      - SecRule REQUEST_URI "@beginsWith /auth/admin/realms/" "id:1010,phase:1,pass,nolog,ctl:ruleRemoveById=920180"
      - Include @owasp_crs/*.conf
      - SecRuleUpdateTargetById 931100 "!ARGS_POST:/json\.redirectUris\..*/|!ARGS_POST:/json\.webOrigins\..*/"
      - SecRuleUpdateTargetById 934110 "!ARGS_POST:/json\..*/"
      - SecRuleUpdateTargetById 930120 "!REQUEST_COOKIES:KEYCLOAK_IDENTITY"
      - SecRuleUpdateTargetById 930120 "!ARGS_POST_NAMES:/json\..*/"
      - SecRuleUpdateTargetById 930120 "!ARGS_POST_NAMES:/refresh_token:.*/"
      - SecRuleUpdateTargetById 932130 "!ARGS_POST:/json\..*/"
      - SecRuleUpdateTargetById 942100 "!ARGS_POST:/json\..*/"
      - SecRuleRemoveByID 911100
      - SecRuleRemoveByTag language-php
      - SecRuleRemoveByTag platform-windows
      - SecRuleRemoveByTag platform-mssql
  selector:
    matchLabels:
      waf: keycloak
  url: oci://ghcr.io/corazawaf/coraza-proxy-wasm:0.5.0

Envoy log =>


2024-06-20T13:56:30.218347Z     error   envoy lua external/envoy/source/extensions/filters/http/lua/lua_filter.cc:926   script log: [string "function envoy_on_response(response_handle)..."]:2: attempt to index global 'rh' (a nil value)       thread=34
2024-06-20T13:56:30.300833Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:104    Caught Segmentation fault, suspect faulting address 0x160       thread=34
2024-06-20T13:56:30.300868Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:91     Backtrace (use tools/stack_decode.py to get line numbers):      thread=34
2024-06-20T13:56:30.300873Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:92     Envoy version: 6e17fceba9bf16510d80f98b3523672b58f2ce5e/1.29.5-dev/Clean/RELEASE/BoringSSLthread=34
2024-06-20T13:56:30.302519Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #0: __restore_rt [0x7fd877951810]       thread=34
2024-06-20T13:56:30.336513Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #1: Envoy::Http::Http1::StreamEncoderImpl::endEncode() [0x5620b089c614] thread=34
2024-06-20T13:56:30.355874Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #2: Envoy::Http::Http1::StreamEncoderImpl::encodeData() [0x5620b089c80e]        thread=34
2024-06-20T13:56:30.377176Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #3: Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData() [0x5620b0875b34]     thread=34
2024-06-20T13:56:30.397158Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #4: Envoy::Http::FilterManager::encodeData() [0x5620b09b70f6]   thread=34
2024-06-20T13:56:30.419243Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #5: Envoy::Http::ActiveStreamFilterBase::commonContinue() [0x5620b09aefc5]      thread=34
2024-06-20T13:56:30.437730Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #6: Envoy::Http::ActiveStreamFilterBase::commonHandleAfterDataCallback() [0x5620b09afa21]thread=34
2024-06-20T13:56:30.456267Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #7: Envoy::Http::FilterManager::encodeData() [0x5620b09b6bcb]   thread=34
2024-06-20T13:56:30.476421Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #8: Envoy::Router::UpstreamRequest::decodeData() [0x5620b093cb1d]       thread=34
2024-06-20T13:56:30.476445Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #9: Envoy::Http::FilterManager::encodeData() [0x5620b09b70f6]   thread=34
2024-06-20T13:56:30.496187Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #10: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x5620b06a6fc2] thread=34
2024-06-20T13:56:30.496212Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #11: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x5620b06a6fc2] thread=34
2024-06-20T13:56:30.518067Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #12: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x5620b089fff1]    thread=34
2024-06-20T13:56:30.537038Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #13: Envoy::Http::Http1::ClientConnectionImpl::dispatch() [0x5620b089fa1d]      thread=34
2024-06-20T13:56:30.557265Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #14: Envoy::Http::CodecClient::onData() [0x5620b0746a50]        thread=34
2024-06-20T13:56:30.576734Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #15: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x5620b0748de5]       thread=34
2024-06-20T13:56:30.595987Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #16: Envoy::Network::FilterManagerImpl::onContinueReading() [0x5620b0b14795]    thread=34
2024-06-20T13:56:30.618484Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #17: Envoy::Network::ConnectionImpl::onReadReady() [0x5620b0ace527]     thread=34
2024-06-20T13:56:30.637322Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #18: Envoy::Network::ConnectionImpl::onFileEvent() [0x5620b0acb54f]     thread=34
2024-06-20T13:56:30.656151Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #19: std::__1::__function::__func<>::operator()() [0x5620b0abf1f1]      thread=34
2024-06-20T13:56:30.675770Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #20: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x5620b0ac075d]      thread=34
2024-06-20T13:56:30.695443Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #21: event_process_active_single_queue [0x5620b0cc1200] thread=34
2024-06-20T13:56:30.715772Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #22: event_base_loop [0x5620b0cbfb41]   thread=34
2024-06-20T13:56:30.738449Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #23: Envoy::Server::WorkerImpl::threadRoutine() [0x5620b032fad0]        thread=34
2024-06-20T13:56:30.759765Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #24: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::__invoke() [0x5620b0d34573] thread=34
2024-06-20T13:56:30.760338Z     critical        envoy backtrace external/envoy/source/server/backtrace.h:96     #25: start_thread [0x7fd87799bc91]      thread=34
ActiveStream 0x5620b9c93c00, stream_id_: 7323083958643283079&filter_manager_:
  FilterManager 0x5620b9c93ca8, state_.has_1xx_headers_: 0
  filter_manager_callbacks_.requestHeaders():
    ':path', '/background.png'
    ':method', 'GET'
    ':scheme', 'https'
    'pragma', 'no-cache'
    'cache-control', 'no-cache'
    'sec-ch-ua', '"Not/A)Brand";v="8", "Chromium";v="126", "Google Chrome";v="126"'
    'sec-ch-ua-mobile', '?0'
    'sec-ch-ua-platform', '"macOS"'
    'upgrade-insecure-requests', '1'
    'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36'
    'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7'
    'sec-fetch-site', 'none'
    'sec-fetch-mode', 'navigate'
    'sec-fetch-user', '?1'
    'sec-fetch-dest', 'document'
    'accept-encoding', 'gzip, deflate, br, zstd'
    'accept-language', 'en-GB,en-US;q=0.9,en;q=0.8,fr;q=0.7'
    'priority', 'u=0, i'
    'x-forwarded-for', '88.xx.yy.zz'
    'x-envoy-external-address', '88.xx.yy.zz'
    'x-b3-traceid', '7f22272fa09e75ac64deeb820d39701f'
    'x-b3-spanid', 'e489df6c6f925503'
    'x-b3-parentspanid', '64deeb820d39701f'
    'x-b3-sampled', '0'
  filter_manager_callbacks_.requestTrailers():   null
  filter_manager_callbacks_.responseHeaders():
    ':status', '200'
    'cache-control', 'max-age=86400'
    'content-type', 'image/png'
    'referrer-policy', 'no-referrer'
    'strict-transport-security', 'max-age=31536000; includeSubDomains'
    'x-content-type-options', 'nosniff'
    'x-xss-protection', '1; mode=block'
    'date', 'Thu, 20 Jun 2024 13:56:29 GMT'
  filter_manager_callbacks_.responseTrailers():   null
  &streamInfo():
    StreamInfoImpl 0x5620b9c93de8, protocol_: 1, response_code_: 200, response_code_details_: response_payload_too_large, attempt_count_: 1, health_check_request_: 0, getRouteName(): default    upstream_info_:
      UpstreamInfoImpl 0x5620b841b4b8, upstream_connection_id_: 59
    OverridableRemoteConnectionInfoSetterStreamInfo 0x5620b9c93de8, remoteAddress(): 88.xx.yy.zz:0, directRemoteAddress(): 10.10.1.212:52982, localAddress(): 10.10.10.93:8080
Http1::ConnectionImpl 0x5620b9c96010, dispatching_: 1, dispatching_slice_already_drained_: 0, reset_stream_called_: 1, handling_upgrade_: 0, deferred_end_stream_headers_: 0, processing_trailers_: 0, buffered_body_.length(): 0, header_parsing_state_: Done, current_header_field_: , current_header_value_:
absl::get<ResponseHeaderMapPtr>(headers_or_trailers_): null
Dumping corresponding downstream request: null
, current_dispatching_buffer_: nullConnectionImpl 0x5620b8e38480, connecting_: 0, bind_error_: 0, state(): Closed, read_buffer_limit_: 1048576
socket_:
  ListenSocketImpl 0x5620bd3ef200, transport_protocol_:
  connection_info_provider_:
    ConnectionInfoSetterImpl 0x5620ba81fe18, remote_address_: 10.10.10.93:8080, direct_remote_address_: 10.10.10.93:8080, local_address_: 127.0.0.6:38947, server_name_:
2024-06-20T13:56:30.773530Z     error   Envoy exited with error: signal: segmentation fault
2024-06-20T13:56:30.773950Z     info    ads     ADS: "@" keycloak-664c55589d-hcsz5.zel-zzzzzzzz-2 terminated
2024-06-20T13:56:30.774209Z     info    ads     ADS: "@" keycloak-664c55589d-hcsz5.zel-zzzzzzzz-1 terminated
2024-06-20T13:56:30.774260Z     info    sds     SDS server for workload certificates started, listening on "./var/run/secrets/workload-spiffe-uds/socket"
YvesZelros commented 5 months ago

About body response parsing of PNG image => Is here IsResponseBodyProcessable must be checked as is done on coraza/http/interceptor.go ?

- if !tx.IsResponseBodyAccessible() {
+ if !i.tx.IsResponseBodyAccessible() || !i.tx.IsResponseBodyProcessable() {
M4tteoP commented 5 months ago

About body response parsing of PNG image => Is here IsResponseBodyProcessable must be checked as is done on coraza/http/interceptor.go ?

Hello, I think that it is definitely a check we can do, and it is very welcomed, especially on this memory-delicate wasm env. I have a dev branch here with the change: https://github.com/corazawaf/coraza-proxy-wasm/tree/IsResponseBodyProcessable. If handier, I pushed the image to oci://docker.io/deltag/coraza-proxy-wasm:respbproc. Please, consider this image/branch just for testing purposes, but an early feedback would be very welcomed

YvesZelros commented 5 months ago

@M4tteoP Thanks for the PR, Make sens to update coraza.conf-recommended.conf by adding application/json on SecResponseBodyMimeType ?

M4tteoP commented 5 months ago

Make sens to update coraza.conf-recommended.conf by adding application/json on SecResponseBodyMimeType?

Good point. Apart from changes that are really proxy-wasm specific, the idea is to keep the basic conf file as much as possible aligned with the upstream Coraza one. We could move that idea upstream, but the config is meant to be customized for each deployment, so, by default, I would keep the least possible elements

YvesZelros commented 5 months ago

As JSON parsing is actived for request but not for response, I propose to change it. Let see

jcchavezs commented 5 months ago

My 2p here: IMHO by default response body access should be disabled and the default should be raw. It is very extensive and if any should be only enabled carefuly because on every request we will be parsing a json. Same for request. Maybe we can discuss all this in a broader issue but having json by default sounds overkill given the experience of big memory consumption of coraza in wasm envs.