[Google Workspace] Support All Event Types

[jamiehynds]

Our Workspace integration currently supports ingest of events from several Workspace services, however we don't support all event types, resulting in blindspots. These events (or Activity Reports to use Google terms) are supported by the same Reporting API that we currently rely upon and based on user demands, need to be added to our integration as new data streams.

Events to Support

• Access Transparency
• GCP
• Groups Enterprise
• Mobile/Device
• Oauth/Token
• Context Aware Access
• Gmail (need to determine if Reporting API supports Gmail Events)
• ChromeOS
• Data Studio
• Keep
• Calendar
• Chat
• GPlus
• Jamboard
• Meet

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[SHolzhauer]

More logging from Google Workspace would be a great addition.

[terrancedejesus]

At the moment I am working on additional SIEM detection rules for Google Workspace regarding email. I have found the visibility into Gmail, outside from the global settings applied by an Admin, to be very minimal. My current setup is using the Business Standard license, whereas I believe there may be an additional Gmail data source available with an Enterprise Plus or Education Plus license as described here.

As a result I am unsure if the lack of Gmail logging is caused by licensing or non-existing capabilities or visibility by Google. If anyone has any further information on this, it would be appreciated.

Also, do we as Elastic have a specific Enterprise Plus license for developer purposes that can be shared or where I may be added as a user for rule development? I understand there are many risks associated with it as I would be playing with all settings of Google Workspace.

Also, when reviewing some documentation for logs -> BigQuery, it is evident that there is a lot more Gmail data and verbosity than we may currently be obtaining. Reference: https://support.google.com/a/answer/12384955?hl=en&ref_topic=9079469

This data itself would be extremely useful for detection rule development.

[jamiehynds]

@SHolzhauer we recently added support for DLP audit log and Workspace Alerts. In addition to the three sources (Mobile, Token and Group Enterprise) within this issue, are there other event types you'd like to see added to the integration. We'd like to eventually support all event types from the Reports API, but prioritising the highest value/requested sources at the moment.

[jamiehynds]

@terrancedejesus Awesome blog post last week - I've already shared it with several users. I'm working on getting access to a Workspace sandbox for development/testing purposes - will touch base with you offline.

The data mentioned in the BigQuery looks similar to mail trace data was that available in an Email Audit API, but they've now deprecated infavor of the Reports API. Can't seem to find the relevant set of events with the Reports API docs, but will do some digging.

[terrancedejesus]

Add Gmail Log Ingestion

@jamiehynds - Thanks for the follow up and reading the blog! Doing some additional digging, I had a hunch that there are more "data sources" available but are restricted depending on license. So I decided to upgrade my current subscription to Enterprise Plus and have several new data sources available via the Reports API, including Gmail Logs!! The data looks pretty decent from a detection rule perspective. I'd be happy to share any data I have to help with getting the integration updated to support pulling gmail logs. While there are other data sources included, Gmail visibility has the most potential value for the SIEM product. Let me know if there is anything else I can provide.

It is also good to hear about a potential sandbox for those looking to explore, please let me know at your earliest convenience when/if I can have access. From TRaDE's rule development I think at least 2 users like TRADE-Alice and TRADE-Bob would be fine to emulate activity between two internal users whereas we can interact with those users via pseudonym accounts outside of the organization. This should cover most of our rule research and development. We can always discuss those details at a later time though.

[terrancedejesus]

@jamiehynds @vinit-elastic - Thanks again for taking a look into adding the Gmail data source into this integration! As discussed, the gmail-related logs that are available from the Admin API are good for visibility into global Gmail settings for the domain or organization, changes, and so fourth, but do not give visibility into specific email data that could be extremely valuable for threat detection. As an example, with the Admin logs we can see when a GWS administrator sets a global route for Gmail as flagged with this rule.

However, what we need to capture are email specifics for each user where we can create logic based on email subject lines, email attachment types and naming, hyperlinks within the email body, original email source, etc. With this visibility, we have the opportunity to write detection logic for very important and popular adversary tactics used for spearphishing and malspam.

While I do not have all of the context related, this Splunk rule is a great example of visibility and detections we lack with GWS. Anything I can be of help with regarding this, please feel free to reach out!

[terrancedejesus]

Add OAuth Log Ingestion

@jamiehynds @vinit-elastic - I have been doing some research on abusing Google's Apps Script platform with container-bound scripts and phishing emails. It does not appear that Apps Script has any specific logging capabilities at the moment unless you interact directly with script.google.com and pull execution logs.

Aside from this, there is some logs in google_workspace.drive where we can filter on as shown below.

However, this only shows when a Google Script is created, edited, changed or deleted. We ideally would want to identify when a script is executed. Not sure if your team has reviewed this feature and how to capture logging but I do not believe any other SIEM is doing so at the moment and it could be a big advantage. Additionally, the ability to triage and capture script executions with document creations or copies is essential to detecting email phishing with container-bound scripts.

This lead me down the rabbit hole of attempting to capture some activity from a phishing simulation I conducted. Unfortunately, it does not appear that Google Workspace logs when a container-bound script is created when a document is copied that contains one in the Drive API. However, it does capture OAuth from the victim who would need to authorize the permissions the script is seeking.

However, it seems we do not collect these logs either. These logs are very important to credential access tactics and techniques where we focus on OAuth activity in relation to user and object being requested.

Hypothically, we would identify access granted OAuth events for administrators where a correlating application ID created event could not be found, thus the script is from an external source. There are other situations where OAuth will come in handy for detection rules as well.

Similar to Gmail, OAuth is another data source available in the admin console, so I assume there is an API where this data can be ingested from the integration.

As always, please let me know if you have questions or I can help with moving this forward.

[terrancedejesus]

@vinit-chauhan or @jamiehynds - Do we know of a rough ETA for at least getting the OAuth and Gmail logs ingested with the integration? Any chance we can target 8.8?

The TRaDE team is running into some roadblocks with creating more detection rules with these two specific sources of data missing as well as some blogs and conference talks.

If there is anything we can do to help, please let us know as we are always happy to help if possible!

[jamiehynds]

@terrancedejesus the Gmail logs seem like a much heavier lift on our end as they aren't covered by the Report API we currently rely on, and there isn't an elegant way (at least that I can see) to pull them.

The OAuth logs are probably doable though, as they are similar to the events our integration currently supports. Can you confirm if these are the OAuth logs you're referring to? If so, is your Workspace instance with Oauth logs still available for us to access?

[terrancedejesus]

The OAuth logs are probably doable though, as they are similar to the events our integration currently supports. Can you confirm if these are the OAuth logs you're referring to? If so, is your Workspace instance with Oauth logs still available for us to access?

@jamiehynds Yes these are the OAuth logs. The importance of them is tracking applications with OAuth access such as third party apps or anything from GCP. Additionally any time a user accepts "This doc/application would like to access these permissions" that is logged here which is where we want to help identify phishing.

It's very odd that the gmail logs are not available via the same Reports API as the others since it is available in the same Reporting > Audit and investigation > Gmail logs location as the other data sources. I'll dig into this a bit more on my end to see if there are additional options.

My environment is using Enterprise Plus and still has OAuth logs as a data source. To avoid additional costs and limit risks only a couple user accounts are still available so if @vinit-chauhan or someone else needs access I am happy to set that back up.

[terrancedejesus]

It's very odd that the gmail logs are not available via the same Reports API as the others since it is available in the same Reporting > Audit and investigation > Gmail logs location as the other data sources. I'll dig into this a bit more on my end to see if there are additional options.

@jamiehynds - Yep, it appears gmail is not accessible via the Reports API. It is a shot in the dark, but I submitted feature request to them to add this. It seems like it is ready to go since it is already available under Audit and investigation in tabular format similar to other logs. I just dont think they have added it to the Reports API yet.

Reference: https://issuetracker.google.com/issues/274983760

[terrancedejesus]

Confirmed OAuth logs are being ingested! Thanks @vinit-chauhan wonderful work.

[vinit-chauhan]

Hey @terrancedejesus - Great to hear that our last release has your use case covered. Excited to hear your feedback on the integration. Let us know if you have any other use cases in your mind. 😄

[terrancedejesus]

@vinit-chauhan - So far the data is great!

One thing I noticed so far is that event.category is not given for the OAuth logs. Since we use EQL in some of our rules, we need an event category to start the logic with. Since this stream does not add one, we have to use any like in the example below.

sequence by source.user.email with maxspan=3m
[file where event.dataset == "google_workspace.drive" and event.action == "copy" and

    /* Should only match if the object lives in a Drive that is external to the user's GWS organization */
    google_workspace.drive.owner_is_team_drive == "false" and google_workspace.drive.copy_type == "external" and

    /* Google Script, Forms, Sheets and Document can have container-bound scripts */
    google_workspace.drive.file.type: ("script", "form", "spreadsheet", "document")]

[any where event.dataset == "google_workspace.token" and event.action == "authorize" and

    /* Ensures application ID references custom app in Google Workspace and not GCP */
    google_workspace.token.client.id : "*apps.googleusercontent.com"]

Reviewing the existing event categories, I think these would align properly with authentication. Reference: https://www.elastic.co/guide/en/ecs/8.7/ecs-allowed-values-event-category.html#ecs-event-category-authentication

Therefore the previous query's second sequence would be updated as such:

[authentication where event.dataset == "google_workspace.token" and event.action == "authorize" and

    /* Ensures application ID references custom app in Google Workspace and not GCP */
    google_workspace.token.client.id : "*apps.googleusercontent.com"]

Thoughts?

[jamiehynds]

@piyush-elastic could someone on the team have a look at Terrance's feedback above? We recently added additional data streams to Workspace, but seemed to have missed some ECS categorisation for the Oauth/Token events.

[terrancedejesus]

@piyush-elastic could someone on the team have a look at Terrance's feedback above? We recently added additional data streams to Workspace, but seemed to have missed some ECS categorisation for the Oauth/Token events.

Thanks Jamie! If needed here is an example event for the token datastream.

Example JSON

```sql { "_index": ".ds-logs-google_workspace.token-default-2023.03.28-000001", "_id": "zrqUGXC7mfTmfqT63dUGOhlj+TM=", "_version": 1, "_score": 0, "_source": { "agent": { "name": "ubuntu-server-tdejesus", "id": "f4011165-a1b8-4d8e-b902-56ba8835cc28", "type": "filebeat", "ephemeral_id": "4c4fa27f-35e7-465b-8383-0b6e6cea1884", "version": "8.6.2" }, "elastic_agent": { "id": "f4011165-a1b8-4d8e-b902-56ba8835cc28", "version": "8.6.2", "snapshot": false }, "source": { "ip": "34.74.46.242", "user": { "domain": "dejesusarcheology.com", "name": "terrance", "id": "115903088752625509360", "email": "terrance@dejesusarcheology.com" } }, "tags": [ "forwarded", "google_workspace-token" ], "cloud": { "availability_zone": "us-east1-b", "instance": { "name": "ubuntu-server-tdejesus", "id": "1709224677170316971" }, "provider": "gcp", "service": { "name": "GCE" }, "machine": { "type": "e2-medium" }, "project": { "id": "elastic-security-dev" }, "account": { "id": "elastic-security-dev" } }, "input": { "type": "httpjson" }, "@timestamp": "2023-03-31T14:35:36.295Z", "ecs": { "version": "8.6.0" }, "related": { "hosts": [ "dejesusarcheology.com" ], "ip": [ "34.74.46.242" ], "user": [ "115903088752625509360", "terrance", "terrance@dejesusarcheology.com" ] }, "google_workspace": { "kind": "admin#reports#activity", "etag": "\"1rpbeKGlFXXrXfE6I4jSKiwRKeqWfVrVpA_lfqSodas/wF6QggzmwJNygQnjqH1S2yPCAdE\"", "event": { "type": "auth" }, "token": { "app_name": "elastic-agent", "method_name": "reports.activities.list", "api_name": "admin", "num_response_bytes": 280, "client": { "id": "116687977707671314889", "type": "WEB" }, "product_bucket": "GSUITE_ADMIN" } }, "data_stream": { "namespace": "default", "type": "logs", "dataset": "google_workspace.token" }, "organization": { "id": "C00qtspd5" }, "event": { "agent_id_status": "verified", "ingested": "2023-03-31T14:42:50Z", "provider": "token", "created": "2023-03-31T14:42:49.688Z", "kind": [ "event" ], "action": "activity", "id": "-8482777507028773979", "dataset": "google_workspace.token" }, "user": { "domain": "dejesusarcheology.com", "name": "terrance", "id": "115903088752625509360", "email": "terrance@dejesusarcheology.com" } }, "fields": { "elastic_agent.version": [ "8.6.2" ], "source.user.email": [ "terrance@dejesusarcheology.com" ], "cloud.availability_zone": [ "us-east1-b" ], "source.user.name.text": [ "terrance" ], "source.ip": [ "34.74.46.242" ], "agent.name": [ "ubuntu-server-tdejesus" ], "google_workspace.token.method_name": [ "reports.activities.list" ], "event.agent_id_status": [ "verified" ], "event.kind": [ "event" ], "google_workspace.event.type": [ "auth" ], "user.id": [ "115903088752625509360" ], "cloud.instance.name.text": [ "ubuntu-server-tdejesus" ], "input.type": [ "httpjson" ], "data_stream.type": [ "logs" ], "related.user": [ "115903088752625509360", "terrance", "terrance@dejesusarcheology.com" ], "google_workspace.token.client.type": [ "WEB" ], "tags": [ "forwarded", "google_workspace-token" ], "cloud.machine.type": [ "e2-medium" ], "cloud.provider": [ "gcp" ], "event.provider": [ "token" ], "agent.id": [ "f4011165-a1b8-4d8e-b902-56ba8835cc28" ], "cloud.service.name": [ "GCE" ], "ecs.version": [ "8.6.0" ], "event.created": [ "2023-03-31T14:42:49.688Z" ], "google_workspace.token.app_name": [ "elastic-agent" ], "organization.id": [ "C00qtspd5" ], "agent.version": [ "8.6.2" ], "related.hosts": [ "dejesusarcheology.com" ], "source.user.name": [ "terrance" ], "google_workspace.token.num_response_bytes": [ 280 ], "user.name": [ "terrance" ], "cloud.instance.id": [ "1709224677170316971" ], "agent.type": [ "filebeat" ], "event.module": [ "google_workspace" ], "user.email": [ "terrance@dejesusarcheology.com" ], "related.ip": [ "34.74.46.242" ], "elastic_agent.snapshot": [ false ], "source.user.id": [ "115903088752625509360" ], "user.domain": [ "dejesusarcheology.com" ], "google_workspace.token.product_bucket": [ "GSUITE_ADMIN" ], "google_workspace.token.client.id": [ "116687977707671314889" ], "elastic_agent.id": [ "f4011165-a1b8-4d8e-b902-56ba8835cc28" ], "data_stream.namespace": [ "default" ], "google_workspace.kind": [ "admin#reports#activity" ], "event.action": [ "activity" ], "event.ingested": [ "2023-03-31T14:42:50.000Z" ], "@timestamp": [ "2023-03-31T14:35:36.295Z" ], "google_workspace.etag": [ "\"1rpbeKGlFXXrXfE6I4jSKiwRKeqWfVrVpA_lfqSodas/wF6QggzmwJNygQnjqH1S2yPCAdE\"" ], "cloud.account.id": [ "elastic-security-dev" ], "data_stream.dataset": [ "google_workspace.token" ], "google_workspace.token.api_name": [ "admin" ], "agent.ephemeral_id": [ "4c4fa27f-35e7-465b-8383-0b6e6cea1884" ], "source.user.domain": [ "dejesusarcheology.com" ], "event.id": [ "-8482777507028773979" ], "event.dataset": [ "google_workspace.token" ], "cloud.instance.name": [ "ubuntu-server-tdejesus" ], "cloud.project.id": [ "elastic-security-dev" ], "user.name.text": [ "terrance" ] } } ```

[piyush-elastic]

@terrancedejesus thanks for the feedback, will analyze and update it soon.