[FEATURE] - API Host Variable

scwxsljohnson commented 8 months ago

Is your feature request related to a problem? Please describe. As the author of Secureworks Taegis XDR normalizers for Akamai products via ULS, the lack of an API host variable for use in the ULS_TCPUDP_FORMAT option complicates the insertion of a source value to indicate the cloudy origin in which events were sourced. At this time the complication is handled by manual insertion/configuration of the API host in the ULS_TCPUDP_FORMAT option string.

Example: ULS_TCPUDP_FORMAT='{"gc_hostname": "lab01.guardicore.com", "ulsfeed": "Akamai-GC-NETLOG", "event": %s}'

Describe the solution you'd like To reduce the configuration complexity, I desire ULS read the EDGERC host value and the value be accessible as a variable in the ULS_TCPUDP_FORMAT option.

Example:

%h = EDGERC host variable

ULS_TCPUDP_FORMAT='{"gc_hostname": %h, "ulsfeed": "Akamai-GC-NETLOG", "event": %s}'

Describe alternatives you've considered The current alternative is manual insertion of the host value.

Additional context Add any other context or screenshots about the feature request here. The above FORMAT schema is being utilized by Secureworks Taegis XDR for all ULS sourced Akamai products for systematic identification and normalization providing the highest user experience.

MikeSchiessl commented 8 months ago

Hi @scwxsljohnson , thanks for raising this topic. Just before I have a closer look at this - would it be an option for you to provide the EDGERC credentials for GC via ENV vars ? The CLI allows it and therefore we could probably fix forward pretty "easily".

Currently, there is some "strict" segmentation between the CLI part and the ULS overlay, so ULS itself doesn't have the "edgerc" data by now.

Anyhow, please correct me, if I am off here:

The data could be set at ULS start - as ULS is pinned to a single .edgerc / section by default - so "hard" setting it in the ULS_TCPUDP_FORMAT should behave identically to what we could with the proposed variable approach
If we implement this request, instead of ingesting 'edgerc' data, personally I'd tend to go down a more variable way like allowing the usage of ENV vars to be set within that FROMAT variable - would this also help on your particular issue ? We could easily provide some "script" that upfront sets any choosable ENV var to the desired value (i.e. data from the EDGERC file)

I feel i am not 100% getting the use case, you're trying to solve but maybe the above is already targeting the right direction.

Please provide me a couple more details so I am able to support on your request.

Best regards Mike

scwxsljohnson commented 8 months ago

I should have stated in my original submission. I'm utilizing Docker Compose to collect for multiple products (EAA, ETP, and GC) vs Python.

Your proposal to make the host value from the EDGERC sections available as a variable for use in the ULS_TCPUDP_FORMAT option is acceptable.

I noticed currently the host definition varies per product (gc_hostname, host, eaa_api_host). Would it be possible to utilize a common variable in the ULS_TCPUDP_FORMAT option making it easy for customers to utilize said variable?

ULS_TCPUDP_FORMAT examples utilizing %h as variable:

GC ULS_TCPUDP_FORMAT='{"gc_hostname": %h, "ulsfeed": "Akamai-GC-NETLOG", "event": %s}'

EAA ULS_TCPUDP_FORMAT='{"eaa_hostname": %h, "ulsfeed": "Akamai-EAA-ACCESS", "event": %s}'

ETP ULS_TCPUDP_FORMAT='{"etp_hostname": %h, "ulsfeed": "Akamai-ETP-DNS", "event": %s}'

NOTE: The above format offers precise matching of Akamai events by Taegis XDR when transmitted by ULS.

MikeSchiessl commented 8 months ago

Hi @scwxsljohnson, I have implemented something you could try using the "development" branch (and the corresponding docker tag).

The variable representing the "API hostname" is called {api_hostname}.

I've also added this new feature to the docs: https://github.com/akamai/uls/blob/development/docs/ARGUMENTS_ENV_VARS.md#replacement-vars-in-http--tcpudp-format

Please let me know if this is helping you going forward. Please be aware that for the rest of this week my answers might be delayed. I will pick up things again starting from next week.

Some background information:

for performance reasons, we're doing the the "transition" only once at startup - so ULS will not be able to handle dynamic changes (but I think there would be tons of additional issues when dynamically changing values in edgerc :) )
The implementation is a bit "clunky" as the different classes handling input / output are not sharing too much of the variables, but this shouldn't bother your at all
I needed to use the input class to find the "correct" hostname value, as you actually could have a couple defined in one config section - so this is why i had to think a little loner on this topic

Let me know if this works for you

Best regards Mike

scwxsljohnson commented 8 months ago

I pulled the development branch and utilized the variable. The variable doesn't result in the host value from the EDGERC file.

CONTAINER REPOSITORY TAG IMAGE ID SIZE uls-guardicore-GC-INCIDENT-1 akamai/uls development eafe7587a163 417MB uls-guardicore-GC-NETLOG-1 akamai/uls development eafe7587a163 417MB

ULS_TCPUDP_FORMAT='{"gc_hostname": {api_hostname}, "ulsfeed": "Akamai-GC-NETLOG", "event": %s}'

TCPdump 18:59:35.027351 IP 192.168.8.34.34954 > 192.168.8.13.601: Flags [P.], seq 1:2282, ack 1, win 502, options [nop,nop,TS val 1655697780 ecr 884694548], length 2281 E. ..h@.?.P....".......YQNA...V.....R...... b..t4.^.{"api_host": {api_hostname}, "ulsfeed": "Akamai-GC-NETLOG", "event": {"flow_id":

MikeSchiessl commented 8 months ago

Hi @scwxsljohnson, please excuse the delay in my response.

I am running ULS with the following command line:

python3.12 bin/uls.py -i etp -f dns -o tcp --host 127.0.0.1 --port 1234 --tcpudpformat '{"api_host": "{api_hostname}", "ulsfeed": "Akamai-ETP-DNS", "event": %s}' -l debug

running a local netcat listener, I am seeing the following:

nc -l 127.0.0.1 1234
...
{"api_host": "akab-xxxxx.luna.akamaiapis.net", "ulsfeed": "Akamai-ETP-DNS", "event": {"id": "519", ....
{"api_host": "akab-xxxxx.luna.akamaiapis.net", "ulsfeed": "Akamai-ETP-DNS", "event": {"id": "520", ....
{"api_host": "akab-xxxxx.luna.akamaiapis.net", "ulsfeed": "Akamai-ETP-DNS", "event": {"id": "521", ....
...

I also double checked the development branch has the same state as my local state and everything seems to be synced. Could you please double check the ULS Version:

bin/uls.py -v
Akamai Unified Log Streamer Version information
ULS Version             1.7.3-beta

scwxsljohnson commented 8 months ago

I'm utilizing containers, but here is my detail.

root@uls01:~/docker-compose/uls-guardicore# docker compose images CONTAINER REPOSITORY TAG IMAGE ID SIZE uls-guardicore-GC-INCIDENT-1 akamai/uls development eafe7587a163 417MB uls-guardicore-GC-NETLOG-1 akamai/uls development eafe7587a163 417MB

root@uls01:~/docker-compose/uls-guardicore# docker compose logs GC-NETLOG-1 | {"dt": "2024-03-25T15:31:36.244391", "uls_product": "GC", "uls_feed": "NETLOG", "uls_output": "TCP", "uls_version": "1.7.3", "uls_runtime": 300, "event_count": 17, "event_count_interval": 17, "event_ingested_interval": 17, "event_bytes_interval": 37038, "event_rate": 0.06, "mon_interval": 300} GC-INCIDENT-1 | {"dt": "2024-03-25T15:31:36.191944", "uls_product": "GC", "uls_feed": "INCIDENT", "uls_output": "TCP", "uls_version": "1.7.3", "uls_runtime": 300, "event_count": 1, "event_count_interval": 1, "event_ingested_interval": 1, "event_bytes_interval": 3032, "event_rate": 0.0, "mon_interval": 300}

MikeSchiessl commented 8 months ago

hi @scwxsljohnson ,

can you please pull the image again?

According to DockerHub, the digest should be: sha256:86767abb6677a30a3a4a425ebd8aba7fc6369b9d55510ccca800f1f2ac5ea261

➜  docker image ls --digests | grep akamai/uls | grep devel
akamai/uls                                                                development   sha256:86767abb6677a30a3a4a425ebd8aba7fc6369b9d55510ccca800f1f2ac5ea261   2ef41f39fa98   6 days ago      417MB

scwxsljohnson commented 8 months ago

Aha! Pulled the image again and I now have the expected outcome.

root@uls01:~/docker-compose/uls-guardicore# docker compose images CONTAINER REPOSITORY TAG IMAGE ID SIZE uls-guardicore-GC-INCIDENT-1 akamai/uls development 2ef41f39fa98 417MB uls-guardicore-GC-NETLOG-1 akamai/uls development 2ef41f39fa98 417MB

{"api_host": "labxx-x.xx.guardicore.com", "ulsfeed": "Akamai-GC-NETLOG", "event":

MikeSchiessl commented 8 months ago

Wohoooo ;) This is great. Please let me know if this does, what you've expected it to do - Feel free to close the ticket as well, it will find its way into the latest release coming up, soon !

scwxsljohnson commented 8 months ago

From end-to-end I'm producing the desired outcome.

A side note. The documentation for the enhancement is missing double quotes around the variable. I did not catch this until I compared your latest update with what I had scraped from the docs. - https://github.com/akamai/uls/blob/development/docs/ARGUMENTS_ENV_VARS.md#replacement-vars-in-http--tcpudp-format

MikeSchiessl commented 8 months ago

Brilliant, I am happy this is working as expected.

You're correct the example generates invalid JSON - thx for the hint. I have fixed it in the testing branch on our repo - will likely jump towards "devel" within the next couple of days.

bitonio commented 8 months ago

Thanks @MikeSchiessl ! I took the liberty or reworking the doc section a little bit beyond just the syntax fix.

MikeSchiessl commented 8 months ago

I have just merged the latest developments into "Devel"

For your interest: I have added 2 additional variables (ULS FEED and ULS INPUT) - so you can use that also as a dynamic variable in the output generation ;)

Best Mike

scwxsljohnson commented 8 months ago

Final format I'll be utilizing:

ULS_TCPUDP_FORMAT='{"api_host": "{api_hostname}", "ulsfeed": "Akamai-{uls_input}-{uls_feed}", "event": %s}'

MikeSchiessl commented 8 months ago

hi @scwxsljohnson , we just released ULS v1.7.3 which includes your change request + an OS ENV VAR substitution in addition to your request.

akamai / uls

[FEATURE] - API Host Variable #57