search

Azure / Azure-Sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.
https://azure.microsoft.com/en-us/services/azure-sentinel/
MIT License
4.63k stars 3.04k forks source link

Update connectors to use the AMA agent and "Log Ingestion API" #10067

Closed o-l-a-v closed 8 months ago

o-l-a-v commented 9 months ago

Is your feature request related to a problem? Please describe.

OMSAgent → AMA

The old Log Analytics agent will stop working 31st of August 2024:

Example of connectors that must be updated:

Example of already updated connectors:

HTTP Data Collector API → Log Ingestion API

The HTTP Data Collector API will stop working 14th of September 2026:

Example of connectors that must be updated:

Describe the solution you'd like

  1. Update connectors to use AMA and "Log Ingestion API" long before August 2024 to give us time to migrate.
  2. Create a roadmap to keep us updated on the progress for various affected connectors.

Describe alternatives you've considered

None.

Additional context

None.

v-sudkharat commented 9 months ago

Hi @o-l-a-v, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 06-03-2024. Thanks!

v-rusraut commented 9 months ago

Hi @o-l-a-v, We are checking with the respective team, we will provide you an update soon. Thanks

v-rusraut commented 8 months ago

Hi @o-l-a-v, Still, we are waiting for response from respective team, we will provide you an update till 22 Mar 2024. Thanks

v-rusraut commented 8 months ago

Hi @o-l-a-v, We got response from respective team. They will be working on these connectors soon, they don't provide any ETA. If we got any update related to this, we will inform you. currently closing this issue. Thanks

lux209 commented 7 months ago

+1 for the meraki connector and also for the VMware one that also only support OMS at the moment

InfiniteInsight commented 6 months ago

Any ETA yet? @v-rusraut

subhashkrishnan-ai commented 5 months ago

Hey, is there any way we can get this logs from data collector endpoints ? MS link- https://learn.microsoft.com/en-us/azure/azure-monitor/logs/custom-logs-migrate

JustinGrote commented 4 months ago

Stuck here too, the OMS agent goes EOL in 1 month and there is no solution that has been specified for bringing Cisco Meraki data into Sentinel via the AMA. I guess we'll have to go off the reservation and report back.

pemontto commented 3 months ago

The API method is not a replacement. The OMS/Syslog method needs to be updated for use with AMA...

image
JustinGrote commented 3 months ago

As an unsupported workaround, I have used logstash with the new DCR based connector to get syslogs into Sentinel. This works on both linux and windows. https://learn.microsoft.com/en-us/azure/sentinel/connect-logstash-data-connection-rules

pemontto commented 3 months ago

@JustinGrote yeah, we’re using Fluent Bit now with direct kind DCRs.

Any chance you’ve tackled ASA logs into CommonSecurityLog without OMS/MMA/OMS? I want to avoid recreating the wheel with whatever black box parsing MS is doing to get them into CEF

JustinGrote commented 3 months ago

@JustinGrote yeah, we’re using Fluent Bit now with direct kind DCRs.

Any chance you’ve tackled ASA logs into CommonSecurityLog without OMS/MMA/OMS? I want to avoid recreating the wheel with whatever black box parsing MS is doing to get them into CEF

Not particularly, I'm mostly making things to match the ASIM schemas to make things easier on our hunters

o-l-a-v commented 3 months ago

Hi @o-l-a-v, We got response from respective team. They will be working on these connectors soon, they don't provide any ETA. If we got any update related to this, we will inform you. currently closing this issue. Thanks

What's the status? 31st of August is approaching, you've closed this issue and provided no new info, which you promised us to do.

There are still connectors that must be updated, like Cisco Meraki.

@v-rusraut @v-sudkharat @v-muuppugund

lux209 commented 3 months ago

I was informed today that a new connector just came out to support VMware, Meraki and many other log types, check the data connector named "Custom Logs AMA", I did not test it yet but it is looking to be the solution for AMA

lux209 commented 3 months ago

@JustinGrote yeah, we’re using Fluent Bit now with direct kind DCRs. Any chance you’ve tackled ASA logs into CommonSecurityLog without OMS/MMA/OMS? I want to avoid recreating the wheel with whatever black box parsing MS is doing to get them into CEF

Not particularly, I'm mostly making things to match the ASIM schemas to make things easier on our hunters

For ASA you can use the Common Event Format (CEF) via AMA connector as far as I know

InfiniteInsight commented 3 months ago

I was informed today that a new connector just came out to support VMware, Meraki and many other log types, check the data connector named "Custom Logs AMA", I did not test it yet but it is looking to be the solution for AMA

I don't see the "Custom Logs AMA" connector you mentioned. Do you mean the "Common Event Format (CEF) via AMA" connector?

o-l-a-v commented 3 months ago

Maybe this one?

InfiniteInsight commented 3 months ago

Maybe this one?

This does look hopeful, but maybe I am being dense because I don't see this in the Data Connectors page in the Sentinel Web UI

lux209 commented 3 months ago

I mean this: https://learn.microsoft.com/en-us/azure/sentinel/unified-connector-custom-device?tabs=rsyslog

I have it on my side on the sentinel content hub named "Custom Logs AMA"

pemontto commented 3 months ago

@JustinGrote yeah, we’re using Fluent Bit now with direct kind DCRs. Any chance you’ve tackled ASA logs into CommonSecurityLog without OMS/MMA/OMS? I want to avoid recreating the wheel with whatever black box parsing MS is doing to get them into CEF

Not particularly, I'm mostly making things to match the ASIM schemas to make things easier on our hunters

For ASA you can use the Common Event Format (CEF) via AMA connector as far as I know

We have some machines that can't use AMA because that would require the tenant they're loading into to manage the machines via Arc. Without the magical AMA black box parsing though we'll have to manually parse ASA into CEF 😢

JustinGrote commented 3 months ago

Agreed, huge limitation of AMA, it requires Azure Arc to work. Lots of MSSPs have probe VMsfor syslog in other client tenants, and it's not supported to onboard an Azure VM to Arc in another tenant, making it a huge gap (apparently MS assumes noone runs things in Azure, funny enough)

For us, Logstash/Fluent Bit was the only solution to this (Fluent Bit being way less resource intensive). We originally tried "tunneling" client syslogs back to an azure collector on our side using dtlspipe, but the solution was too flaky and didn't allow for onsite log filtering to reduce traffic.

pemontto commented 3 months ago

@JustinGrote completely agreed 🤦‍♂️. We considered jailing or containerising an AMA agent inside a forwarder but the overhead and reliability didn't seem worth it. We've used a lot of Logstash in the past but went with Fluent Bit which has proven very capable so far.

Would be interested to hear more about your solution? We're running everything through rsyslog into files then tailing files with Fluent Bit. Yet to be fully load tested but we're successfully using Lua to transform CEF logs into what the CommonSecurityLog table expects, including shoehorning any non-standard extension fields into the AdditionalExtensions field.

lux209 commented 3 months ago

Agreed, huge limitation of AMA, it requires Azure Arc to work. Lots of MSSPs have probe VMsfor syslog in other client tenants, and it's not supported to onboard an Azure VM to Arc in another tenant, making it a huge gap (apparently MS assumes noone runs things in Azure, funny enough)

For us, Logstash/Fluent Bit was the only solution to this (Fluent Bit being way less resource intensive). We originally tried "tunneling" client syslogs back to an azure collector on our side using dtlspipe, but the solution was too flaky and didn't allow for onsite log filtering to reduce traffic.

I m also working for a mssp but we are not getting the logs in our own sentinel. Each client has it own instance in their tenant and they grant us the access through an azure app.

For your case I would most likely use rsyslog to forward the logs to your own collector with ama. Rsyslog is pretty reliable, you just need a secure connection for the traffic.

JustinGrote commented 3 months ago

" you just need a secure connection for the traffic."

This is not a trivial detail, good luck setting up IPSec tunnels with every MSSP customer. That's why we use Fluent Bit, we just ingest directly into a DCR via HTTPS, no secure connection or overhead required (DCRs are free). You can even make DCRs direct now without a DCE, using kind: direct and the latest API

lux209 commented 3 months ago

You can use secure syslog using tls

JustinGrote commented 3 months ago

Would be interested to hear more about your solution? We're running everything through rsyslog into files then tailing files with Fluent Bit. Yet to be fully load tested but we're successfully using Lua to transform CEF logs into what the CommonSecurityLog table expects, including shoehorning any non-standard extension fields into the AdditionalExtensions field.

We just ingest directly into Fluent Bit with its syslog listener. On customer premises it's either:

  1. Running as a windows service (some customers still don't let us deploy linux probes)
  2. Running in a linux VM
  3. Running in a container for our customers that support it (or our more advanced probes)

This allows us to do client-side filtering to minimize ingestion. We use a bicep template tied to a deploy button to create DCRs. Most customers want to "own" their sentinel workspace so we set up the DCR in their environment.

We do some limited formatting at the Fluent Bit level, and then DCR transformation rules written in KQL do most of the heavy lifting, either dumping them to Microsoft-Syslog or, for specific logs, converting them to ASIM format and dumping the logs there. In some cases we have multiple DCR streams and fluent bit passes them to the different streams based on filtering criteria just to keep the KQL transformation rules from getting crazy.

KQL transformation is great, and its free compute, it can even filter up to 50% of the logs without being charged for ingestion.

JustinGrote commented 3 months ago

You can use secure syslog using tls

You're still saying a rsyslog vm at the customer site, then tls'd to some public ip of an azure vm on our side, just to then get it into the collector with AMA (which you would need 1 collector per customer and they couldn't run in azure because you couldn't tie it to the customer workspace)

vs.

a single windows/linux vm/container at the customer site running Fluent Bit that ingests directly to a DCR via the Log Ingestion output plugin and can filter locally, minimizing bandwidth to both.

yeah I'll take the second options thanks.

pemontto commented 3 months ago

@JustinGrote cheers for the explanation.

We have rsyslog in place because we may have other processes subscribing to the output e.g. Splunk, Elastic Beats, etc. We ended up using ansible to deploy tables (including basic tier versions) and DCRs then generate the Fluent Bit config with dynamic inputs/outputs all in one go.

lux209 commented 3 months ago

But if you finally send the log to the customer workspace why not just use arc and ama in the customer collector hosted in their environment?

Anyway this is not the subject of the thread

pemontto commented 3 months ago

But if you finally send the log to the customer workspace why not just use arc and ama in the customer collector hosted in their environment?

Anyway this is not the subject of the thread

If you're managing a forwarding host with your own ~IP~ intellectual property/key material, it's not really acceptable to have every customer with full control/RCE over it.

lux209 commented 3 months ago

I can understand, on our side we have nothing to hide on the collectors, they just have the ama agent with full standard configuration.