Parse Additional Fields from CloudTrail

[vikas891]

I'm fairly new to SOF-ELK (don't use it daily but on and off). I want to push some documentation around investigations (because THAT is what I'm good at) AWS CloudTrail. I've been trying to populate the ErrorCode and ErrorMessage but they don't get parsed, I think because the fields don't exist in some entries or noisy enough to be parsed as a field. But when it comes to forensics I think it's VERY valuable! For e.g. this attempt failed and I'd like to see that as a parsed field or atleast in there.

"errorCode": "AccessDenied", "errorMessage": "User: arn:aws:iam::107513503799:root is not authorized to perform: organizations:ListDelegatedAdministrators on resource: * because no resource-based policy allows the organizations:ListDelegatedAdministrators action",

If this doesn't qualify as an Issue, I'm sorry.

But from Splunk, I built a nice storyline as all the fields were parsed.

Thanks, Vikas

[vikas891]

Thanks @philhagen 🙂 I'd love to run tests or contribute in any form for SOF-ELK! When I started to mind map the Blog, I wanted it to be around SOF-ELK but then I had to 🚧 the SOF-ELK portion which I'll update later - https://www.linkedin.com/feed/update/urn:li:activity:7120406727144980480/

[philhagen]

~do you have a sample that you can share that has these fields? even a small and/or redacted set would be helpful. If you need to share that privately via email, that's fine too!~ I found some samples

[philhagen]

this should be ready for testing on the develop branch. in my testing, these fields were successfully retained and it was a straightforward fix so I'm comfortable promoting this to prod but if you wanted to give it a try before I do that, see https://github.com/philhagen/sof-elk/wiki/How-to-Test-New-Features for instructions on how to test them

[vikas891]

Thank you so much Phil! I am happy to share the test data but I didn't have your email :( I'll test the Dev one tonight!

Sent from my iPhone, please ignore typos if any.

On Fri, 27 Oct 2023 at 10:11 PM, Phil Hagen @.***> wrote:

this should be ready for testing on the develop branch. in my testing, these fields were successfully retained and it was a straightforward fix so I'm comfortable promoting this to prod but if you wanted to give it a try before I do that, see https://github.com/philhagen/sof-elk/wiki/How-to-Test-New-Features for instructions on how to test them

— Reply to this email directly, view it on GitHub https://github.com/philhagen/sof-elk/issues/293#issuecomment-1783208080, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJML3UV4GUM5SPGDNUV7AV3YBPP2TAVCNFSM6AAAAAA577PE6GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBTGIYDQMBYGA . You are receiving this because you authored the thread.Message ID: @.***>

[vikas891]

Good news! Thanks Phil, all the fields are now being parsed with the magic you spun.

The positives,

I was a little curious so as to:

1. Why the Available Fields say that?
2. I know the timeframe of the test data isn't that big but some percentages or unique values would be nice no?
3. Filter for Value makes our lives easier. I'm not sure if it's my data or something I'm missing.

Also, is there any chance we could further parse the highlighted JSON because.. if someone Assumes a Role, the Response Elements contain the Assumed Role ARN which immediately becomes a Person Of Interest

For e.g., in Splunk:

Test Data: ReadyToIngest.json

[philhagen]

from what I can tell via the screenshots, this is a byproduct of how ES handles strings. by default it will tokenize fields, e.g. arn. I configure a separate subfield called arn.keyword that contains the entire untokenized string. Tokenized fields cannot be used for analysis, only keywords, integers, etc. so to do analysis (percentage breakdowns etc, you should get what you're looking for in the e.g. aws_resource_name.keyword field.

With some of those JSON fields, such as raw_request_parameters, I keep them as JSON to keep the indices from growing to have too many fields. since that field in particular contains an arbitrary structure of arbitrary fields, values, and types, I discussed that with the FOR509 team who initially requested the parser and we determined that having it available in some form was acceptable and they teach how to pivot from the SOF-ELK view to the original log source itself for the lowest-level detail findings.

I think that answers your questions, but if I missed something, please let me know. Thanks for the test data - I'll verify that all alongside the other FOR509 evidence I used at first and will merge this to the latest public/v* branch before closing the issue.

thanks for submitting this!