search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
103 stars 4.92k forks source link

Duplicated Google Workspace log entries by Filebeat #39859

Open rlevytskyi opened 5 months ago

rlevytskyi commented 5 months ago

Many moths ago, we’ve noticed that some Google Workspace logs received by Filebeat got duplicated.

I’ve searched the internet for possible cause and find one similar issue here at Elastic Discuss, Google Workspace module using wrong field to avoid duplicates telling that "json.id.time", "json.id.uniqueQualifier", "json.id.applicationName", "json.id.customerId" are used to generate the _id.

After updating Filebeat to the most recent version (8.10.2 run from docker.elastic.co/beats/filebeat:8.10.2) I found that the same issue has different _id.

The issue was posted at forum https://discuss.elastic.co/t/duplicated-google-workspace-log-entries-by-filebeat/344374

Several days ago I've upgraded Filebeat to 8.14.0 and it didn't helped.

Here are two examples. First:

{
  "_index": "google_ws-2023.10.04",
  "_id": "4rrh-YoBiE7xzynem1Cm",
  "_source": { 
    "json": {
      "id": {
        "time": "2023-10-04T08:50:13.677Z"
      },
      "etag": "\"rQ3qpTrpjMqlOD9Fi6ZCgnpo6zAdUtM4Y4wU0J6c8Yw/UiNqGB-f4anaOLIVD9ya9Z-pAP0\"",
      "events": {},
      "actor": {}
    },
    "event": {
      "id": "-8909398197392254316",
      "created": "2023-10-04T08:50:25.347Z",
      "original": "{\"id\":{\"applicationName\":\"drive\",\"customerId\":\"C00hvn0vt\",\"time\":\"2023-10-04T08:50:13.677Z\",\"uniqueQualifier\":\"-8909398197392254316\"}"}"
    },
    "@timestamp": "2023-10-04T08:50:13.677Z",
  },
}

Second:

{
  "_index": "google_ws-2023.10.04",
  "_id": "QePm-YoBq7bjVLXLMFU_",
  "_source": {
    "json": {
      "id": {
        "time": "2023-10-04T08:50:13.677Z"
      },
      "etag": "\"rQ3qpTrpjMqlOD9Fi6ZCgnpo6zAdUtM4Y4wU0J6c8Yw/UiNqGB-f4anaOLIVD9ya9Z-pAP0\"",
      "events": {},
      "actor": {}
    },
    "event": {
      "created": "2023-10-04T08:55:25.376Z",
      "original": "{\"id\":{\"applicationName\":\"drive\",\"customerId\":\"C00hvn0vt\",\"time\":\"2023-10-04T08:50:13.677Z\",\"uniqueQualifier\":\"-8909398197392254316\"}"}"

    },
    "@timestamp": "2023-10-04T08:50:13.677Z",
  }
}

I.e. there is no uniqueQualifier, applicationName, customerId under the “json.id” key, as supposed to be, while they all still exists under the “event.original.id” key.

So could you please tell how this can be fixed?

elasticmachine commented 5 months ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

ShourieG commented 4 months ago

Hi @rlevytskyi, The deduplication fix for this was merged quite a while back in this PR based on this feedback. The unique id for deduplication is no longer in the json.id object but it's a finger print that lies in the _id field of the document. The main issue here seems to be it's generating unique _id's for duplicate events which is weird. Could you confirm if this is the case or not ?

rlevytskyi commented 4 months ago

Yes I can confirm it's the case, different _id for the same event.

ShourieG commented 4 months ago

After investigation of this and similar issues, we've observed the following:

  1. Duplicate issues were significantly reduced and fixed for the most part after this PR was merged.

  2. Duplication issues seem to be more relevant with the workspace module when compared with the workspace integration.

  3. The google workspace module uses a fingerprint processor that does not support canonical ordering of the event object keys, this was recently fixed with this PR and should help reduce duplication going forward.

  4. The duplication issue talked about in this current issue seems to stem from issues outside our control and the involvement of Logstash or some issue that is causing the ingest pipeline to not work as expected as the presence of the "_source" object in the resulting documents suggest that the pipeline did not remove them correctly. Also fields inside the "_source" are missing which is leading to different fingerprints for the same document.

We are keeping this issue open to see if the duplication issue persists following the recent PR fix and we will also introduce an enhancement for adding conditional canonical sorting of keys to the fingerprint processor soon. cc: @narph

rlevytskyi commented 4 months ago

Thank you! I'll test once it will be merged to the new version.

rlevytskyi commented 3 months ago

Just updated filebeat to 8.14.3 and don't see duplicated messages now. Will update you guys on Monday.

narph commented 3 months ago

@rlevytskyi , any updates here? can we close the issue?

rlevytskyi commented 3 months ago

Sorry for the delay, I see no duplicates now. Filebeat works properly.

rlevytskyi commented 2 months ago

It happened again.

maksimsaroka commented 2 months ago

The issue is still exist

rlevytskyi commented 2 months ago 
{
  "_index": "google_ws-2024.35",
  "_id": "WQj5m5EBYPKiM--Ll6eM",
  "_version": 1,
  "_score": null,
  "_source": {
    "office_name": "nl2",
    "google_workspace": {
      "kind": "admin#reports#activity",
      "login": {
        "challenge_method": [
          "none"
        ],
        "is_suspicious": false,
        "type": "reauth"
      },
      "event": {
        "type": "login"
      }
    },
    "source": {
      "ip": "76.185.108.167",
      "user": {
        "id": "113905123137720729621",
        "name": "mboustridge",
        "email": "mboustridge@exadel.com",
        "domain": "exadel.com"
      }
    },
    "user": {
      "name": "mboustridge",
      "id": "113905123137720729621",
      "domain": "exadel.com"
    },
    "ecs": {},
    "service": {
      "type": "google_workspace"
    },
    "fileset": {
      "name": "login"
    },
    "json": {
      "actor": {},
      "id": {
        "time": "2024-08-29T01:00:48.152Z"
      },
      "etag": "\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\"",
      "events": {}
    },
    "agent": {
      "name": "k8s-ams-filebeat"
    },
    "event": {
      "id": "-9198098292682564254",
      "dataset": "google_workspace.login",
      "provider": "login",
      "original": "{\"actor\":{\"email\":\"mboustridge@exadel.com\",\"profileId\":\"113905123137720729621\"},\"etag\":\"\\\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\\\"\",\"events\":{\"name\":\"login_success\",\"parameters\":[{\"name\":\"login_type\",\"value\":\"reauth\"},{\"multiValue\":[\"none\"],\"name\":\"login_challenge_method\"},{\"boolValue\":false,\"name\":\"is_suspicious\"}],\"type\":\"login\"},\"id\":{\"applicationName\":\"login\",\"customerId\":\"C00hvn0vt\",\"time\":\"2024-08-29T01:00:48.152Z\",\"uniqueQualifier\":\"-9198098292682564254\"},\"ipAddress\":\"76.185.108.167\",\"kind\":\"admin#reports#activity\"}",
      "category": [
        "authentication",
        "session"
      ],
      "action": "login_success",
      "created": "2024-08-29T02:31:46.568Z",
      "module": "google_workspace",
      "type": [
        "start"
      ],
      "outcome": "success"
    },
    "type": "logs",
    "@timestamp": "2024-08-29T01:00:48.152Z",
    "related": {
      "ip": [
        "76.185.108.167"
      ],
      "user": [
        "mboustridge"
      ]
    },
    "@version": "1",
    "organization": {
      "id": "C00hvn0vt"
    },
    "input": {},
    "tags": [
      "forwarded",
      "logstash-cv107",
      "logstash-k8s"
    ]
  },
  "fields": {
    "json.id.time": [
      "2024-08-29T01:00:48.152Z"
    ],
    "@timestamp": [
      "2024-08-29T01:00:48.152Z"
    ],
    "event.created": [
      "2024-08-29T02:31:46.568Z"
    ]
  },
  "highlight": {
    "event.original": [
      "{\"@opensearch-dashboards-highlighted-field@actor@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@email@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@mboustridge@/opensearch-dashboards-highlighted-field@@@opensearch-dashboards-highlighted-field@exadel.com@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@profileId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@113905123137720729621@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@etag@/opensearch-dashboards-highlighted-field@\":\"\\\"@opensearch-dashboards-highlighted-field@JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw@/opensearch-dashboards-highlighted-field@/@opensearch-dashboards-highlighted-field@H6NnWqcZznaFd6xmuxV63yJUDpM@/opensearch-dashboards-highlighted-field@\\\"\",\"@opensearch-dashboards-highlighted-field@events@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_success@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@parameters@/opensearch-dashboards-highlighted-field@\":[{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_type@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@value@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@reauth@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@multiValue@/opensearch-dashboards-highlighted-field@\":[\"@opensearch-dashboards-highlighted-field@none@/opensearch-dashboards-highlighted-field@\"],\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_challenge_method@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@boolValue@/opensearch-dashboards-highlighted-field@\":@opensearch-dashboards-highlighted-field@false@/opensearch-dashboards-highlighted-field@,\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@is_suspicious@/opensearch-dashboards-highlighted-field@\"}],\"@opensearch-dashboards-highlighted-field@type@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@id@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@applicationName@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@customerId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@C00hvn0vt@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@time@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@2024@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@08@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@29T01@/opensearch-dashboards-highlighted-field@:@opensearch-dashboards-highlighted-field@00@/opensearch-dashboards-highlighted-field@:@opensearch-dashboards-highlighted-field@48.152Z@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@uniqueQualifier@/opensearch-dashboards-highlighted-field@\":\"-@opensearch-dashboards-highlighted-field@9198098292682564254@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@ipAddress@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@76.185.108.167@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@kind@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@admin@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@reports@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@activity@/opensearch-dashboards-highlighted-field@\"}"
    ]
  },
  "sort": [
    1724893248152
  ]
}

and this

{
  "_index": "google_ws-2024.35",
  "_id": "Ag7-m5EBYPKiM--LLzV_",
  "_version": 1,
  "_score": null,
  "_source": {
    "agent": {
      "name": "k8s-ams-filebeat"
    },
    "input": {},
    "related": {
      "ip": [
        "76.185.108.167"
      ],
      "user": [
        "mboustridge"
      ]
    },
    "tags": [
      "forwarded",
      "logstash-cv107",
      "logstash-k8s"
    ],
    "user": {
      "id": "113905123137720729621",
      "name": "mboustridge",
      "domain": "exadel.com"
    },
    "source": {
      "ip": "76.185.108.167",
      "user": {
        "id": "113905123137720729621",
        "name": "mboustridge",
        "email": "mboustridge@exadel.com",
        "domain": "exadel.com"
      }
    },
    "google_workspace": {
      "login": {
        "is_suspicious": false,
        "challenge_method": [
          "none"
        ],
        "type": "reauth"
      },
      "event": {
        "type": "login"
      },
      "kind": "admin#reports#activity"
    },
    "@timestamp": "2024-08-29T01:00:48.152Z",
    "json": {
      "id": {
        "time": "2024-08-29T01:00:48.152Z"
      },
      "etag": "\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\"",
      "events": {},
      "actor": {}
    },
    "event": {
      "id": "-9198098292682564254",
      "dataset": "google_workspace.login",
      "provider": "login",
      "category": [
        "authentication",
        "session"
      ],
      "action": "login_success",
      "module": "google_workspace",
      "type": [
        "start"
      ],
      "original": "{\"actor\":{\"email\":\"mboustridge@exadel.com\",\"profileId\":\"113905123137720729621\"},\"etag\":\"\\\"JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw/H6NnWqcZznaFd6xmuxV63yJUDpM\\\"\",\"events\":{\"name\":\"login_success\",\"parameters\":[{\"name\":\"login_type\",\"value\":\"reauth\"},{\"multiValue\":[\"none\"],\"name\":\"login_challenge_method\"},{\"boolValue\":false,\"name\":\"is_suspicious\"}],\"type\":\"login\"},\"id\":{\"applicationName\":\"login\",\"customerId\":\"C00hvn0vt\",\"time\":\"2024-08-29T01:00:48.152Z\",\"uniqueQualifier\":\"-9198098292682564254\"},\"ipAddress\":\"76.185.108.167\",\"kind\":\"admin#reports#activity\"}",
      "created": "2024-08-29T02:36:46.556Z",
      "outcome": "success"
    },
    "office_name": "nl2",
    "service": {
      "type": "google_workspace"
    },
    "fileset": {
      "name": "login"
    },
    "@version": "1",
    "organization": {
      "id": "C00hvn0vt"
    },
    "type": "logs",
    "ecs": {}
  },
  "fields": {
    "json.id.time": [
      "2024-08-29T01:00:48.152Z"
    ],
    "@timestamp": [
      "2024-08-29T01:00:48.152Z"
    ],
    "event.created": [
      "2024-08-29T02:36:46.556Z"
    ]
  },
  "highlight": {
    "event.original": [
      "{\"@opensearch-dashboards-highlighted-field@actor@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@email@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@mboustridge@/opensearch-dashboards-highlighted-field@@@opensearch-dashboards-highlighted-field@exadel.com@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@profileId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@113905123137720729621@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@etag@/opensearch-dashboards-highlighted-field@\":\"\\\"@opensearch-dashboards-highlighted-field@JX6NVNYagoMncd_ui5IaKzerPU54GzNJg5WsuXIkYzw@/opensearch-dashboards-highlighted-field@/@opensearch-dashboards-highlighted-field@H6NnWqcZznaFd6xmuxV63yJUDpM@/opensearch-dashboards-highlighted-field@\\\"\",\"@opensearch-dashboards-highlighted-field@events@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_success@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@parameters@/opensearch-dashboards-highlighted-field@\":[{\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_type@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@value@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@reauth@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@multiValue@/opensearch-dashboards-highlighted-field@\":[\"@opensearch-dashboards-highlighted-field@none@/opensearch-dashboards-highlighted-field@\"],\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login_challenge_method@/opensearch-dashboards-highlighted-field@\"},{\"@opensearch-dashboards-highlighted-field@boolValue@/opensearch-dashboards-highlighted-field@\":@opensearch-dashboards-highlighted-field@false@/opensearch-dashboards-highlighted-field@,\"@opensearch-dashboards-highlighted-field@name@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@is_suspicious@/opensearch-dashboards-highlighted-field@\"}],\"@opensearch-dashboards-highlighted-field@type@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@id@/opensearch-dashboards-highlighted-field@\":{\"@opensearch-dashboards-highlighted-field@applicationName@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@login@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@customerId@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@C00hvn0vt@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@time@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@2024@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@08@/opensearch-dashboards-highlighted-field@-@opensearch-dashboards-highlighted-field@29T01@/opensearch-dashboards-highlighted-field@:@opensearch-dashboards-highlighted-field@00@/opensearch-dashboards-highlighted-field@:@opensearch-dashboards-highlighted-field@48.152Z@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@uniqueQualifier@/opensearch-dashboards-highlighted-field@\":\"-@opensearch-dashboards-highlighted-field@9198098292682564254@/opensearch-dashboards-highlighted-field@\"},\"@opensearch-dashboards-highlighted-field@ipAddress@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@76.185.108.167@/opensearch-dashboards-highlighted-field@\",\"@opensearch-dashboards-highlighted-field@kind@/opensearch-dashboards-highlighted-field@\":\"@opensearch-dashboards-highlighted-field@admin@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@reports@/opensearch-dashboards-highlighted-field@#@opensearch-dashboards-highlighted-field@activity@/opensearch-dashboards-highlighted-field@\"}"
    ]
  },
  "sort": [
    1724893248152
  ]
}
msaroka-hpe commented 1 week ago

Hello Guys,

Any progress on this?