[Okta]: Ingest pipeline not parsing `okta.debug_context.debug_data.flattened.tunnels`

[procule]

Integration Name

Okta [okta]

Dataset Name

No response

Integration Version

3.0.0

Agent Version

8.15.3

Agent Output Type

elasticsearch

Elasticsearch Version

8.15.3

OS Version and Architecture

COS 117 LTS (GKE)

Software/API Version

No response

Error Message

No response

Event Original

No response

What did you do?

Configured normally to retrieve Okta logs via API.

I was able to create a runtime field that show the text with:

def tunnels = doc['okta.debug_context.debug_data.flattened.tunnels'];
if (tunnels != null) {
    emit(tunnels.value)
}

Also, using this Ingest pipeline correctly parse the text into an "object":

{
  "json": {
    "field": "okta.debug_context.debug_data.flattened.tunnels",
    "target_field": "okta.debug_context.debug_data.flattened.tunnels",
    "if": "ctx.okta?.debug_context?.debug_data?.flattened?.tunnels != null"
  }
}

What did you see?

Not the full original but only the relevant part.

Note that the tunnels attribute is only present when okta.security_context.is_proxy == true

      "debug_context": {
        "debug_data": {
          "flattened": {
            "tunnels": "[{\"anonymous\":true,\"operator\":\"NORD_VPN\",\"type\":\"VPN\"}]",
            "authnRequestId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "oktaUserAgentExtended": "okta-auth-js/7.8.1 okta-signin-widget-7.25.0",
            .....

    "okta.debug_context.debug_data.flattened": [
      {
        "tunnels": "[{\"anonymous\":true,\"operator\":\"NORD_VPN\",\"type\":\"VPN\"}]",
        "authnRequestId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "oktaUserAgentExtended": "okta-auth-js/7.8.1 okta-signin-widget-7.25.0",
       ...

What did you expect to see?

The JSON text being parsed in the flattened field and a new okta.debug_context.debug_data.tunnels field:

"debug_context": {
  "debug_data": {
    "flattened": {
      "tunnels": [
        {
          "anonymous": true,
          "operator": "NORD_VPN",
          "type": "VPN"
        }
      ],
...

"okta.debug_context.debug_data.tunnels": [
    {
      "anonymous": true,
      "operator": "NORD_VPN",
      "type": "VPN"
    }
 ]

Anything else?

Related: https://github.com/elastic/integrations/pull/11396

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[procule]

From the Okta System Logs:

...
  "published": "2033-01-01T22:44:35.333Z",
  "securityContext": {
    "isProxy": true
  },
  "severity": "INFO",
  "debugContext": {
    "debugData": {
      "tunnels": "[{\"anonymous\":true,\"operator\":\"NORD_VPN\",\"type\":\"VPN\"}]",
      "authnRequestId": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
...

[efd6]

@procule Have you tried using "Remove flattened debug data"? When this is set to true, the data are stored in a non-flattened object in okta.debug_context.debug_data.

[procule]

@procule Have you tried using "Remove flattened debug data"? When this is set to true, the data are stored in a non-flattened object in okta.debug_context.debug_data.

I changed the setting but still, the tunnels field is not parsed:

      "debug_context": {
        "debug_data": {
          "tunnels": "[{\"anonymous\":true,\"operator\":\"WARP_VPN\",\"type\":\"VPN\"}]",

As I shown here, Okta's logs for that attribute is a text field containing JSON data. Of course, that would be best if they parsed it before but at the moment, it looks like they don't.

[procule]

Oh, forgot to add that one:

    "okta.debug_context.debug_data.tunnels": [
      "[{\"anonymous\":true,\"operator\":\"WARP_VPN\",\"type\":\"VPN\"}]"
    ],

[procule]

I tried some simulations of the "logs-okta.system" ingest pipeline and that would work for both "flattened" and "not flattened":

    {
      "pipeline": {
        "if": "ctx._conf?.remove_flattened_debug != true",
        "name": "logs-okta.system-2.13.0-use_flattened_debug"
      }
    },
    {
      "pipeline": {
        "if": "ctx._conf?.remove_flattened_debug == true",
        "name": "logs-okta.system-2.13.0-no_use_flattened_debug"
      }
    },
    {
      "rename": {
        "field": "json.debugContext.debugData.tunnels",
        "target_field": "okta.debug_context.debug_data.tunnels",
        "if": "ctx.json?.debugContext?.debugData?.tunnels != null"
      }
    },
    {
      "json": {
        "field": "okta.debug_context.debug_data.tunnels",
        "target_field": "okta.debug_context.debug_data.tunnels",
        "if": "ctx.okta?.debug_context?.debug_data?.tunnels != null"
      }
    },
    {
      "json": {
        "field": "okta.debug_context.debug_data.flattened.tunnels",
        "target_field": "okta.debug_context.debug_data.flattened.tunnels",
        "if": "ctx.okta?.debug_context?.debug_data?.flattened?.tunnels != null"
      }
    }

If _conf.remove_flattened_debug == true:

If _conf.remove_flattened_debug == false:

[efd6]

OK. Thanks.