ANSI control character in captured strings break MDSD ingestion

[andyatwork]

ANSI control characters that may be used to colorize a TTY, when captured by fluentd and forwarded to mdsd cause the ingestion to fail with an ACK_DECODE_ERROR.

Such strings are for instance seen when containers workloads such as bitnami/openresty docker images start up and log strings such as the following to stdout/stderr:

\u001b[38;5;6mopenresty \u001b[38;5;5m21:25:41.40 \u001b[0m\u001b[38;5;2mINFO \u001b[0m ==> Initializing OpenResty

Note that the above representation is an encoded string that escapes the ANSI control characters.

In order to avoid ingestion to fail, custom filters such as the following have to be added to sanitize the string.

<filter *.*>
  @type record_transformer
  enable_ruby
  <record>
    log ${ record["log"].gsub(/[\u001b\u009b][\[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/, '') }
  </record>
</filter>

As a user of the mdsd plugin I expect the plugin to sanitize or escape any invalid characters out from an FT_STRING before it is passed to mdsd based on what mdsd deems valid, without requiring custom filters to be written.

[WqMsft]

I added above conf but got a fail to expand error for gsub method shown below: 1c 2023-10-05 23:02:00 +0000 [warn]: #0 dump an error event: error_class=RuntimeError error="failed to expand record[\"log\"].gsub(/[\\u001b\\u009b][\\[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/, '') : error = undefined method gsub' for nil:NilClass" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.3/lib/fluent/plugin/filter_record_transformer.rb:310:inrescue in expand'" tag="k8scontainers.quotecoreservice" time=2023-10-05 23:02:00.074292416 +0000 record={"time"=>"2023-10-05T23:02:00.074292416Z", "Type"=>"CacheHelper", ...

why use "log" to match the record, if it returned nil then it matches this error.