Azure Log Analytics DCR support

[rhavenn]

Azure Log Analytics is releasing some new features (currently in preview) around allowing 3rd party logging tools to send in data to the Microsoft controlled tables. However, it looks like this will require support for handling new endpoints for submission.

see: https://docs.microsoft.com/en-us/azure/azure-monitor/logs/custom-logs-overview https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/data-collection-rule-transformations

This should allow FluentBit to send data to a DCR endpoint and have its data be translated into one of the Syslog, WindowsEvents, SecurityEvents, or CommonSecurityLog tables. This would be epic and make it a much more straightforward way to get data into those tables and allow, for example, Sentinel consumption without having to deal with the MS omsagent.

It looks like the Header and Body of the request are the same format as traditional LA endpoints.

[agup006]

Ah this is nice, perhaps we could add a new endpoint configuration option in Azure plugin

[iamvighnesh]

Do we have any updates on this please?

[agup006]

@iamvighnesh would you be able to help contribute this feature/functionality?

[iamvighnesh]

@agup006 Wish I could but I am afraid I can't help with my very limited experience with C lang.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[ashmilhussain]

any update on this?

[HarleyB123]

@agup006 was this ever implemented? would be nice to have

[agup006]

i've re-opened and removed stale - Looks like some Microsoft might be able to help us get this implemented!

[agup006]

Adding @pragmatical who is going to be looking at this in upcoming releases

[kforeverisback]

Hey @agup006 I'm currently looking into it. My plan is basically replicate the Azure LA plugin with added functionality. And for now just looking into Shared Key based Auth

[kforeverisback]

Hey y'all Created azure_logs_ingestion plugin. See https://github.com/kforeverisback/fluent-bit/tree/feature/logs_ingestion Feel free to try it out! Here is a sample conf with azure_logs_ingestion plugin:

[INPUT]
    Name tail
    Path /path/to/sample_access.log
    Tag sample
    Key RawData

[FILTER]
    Name modify
    Match sample
    Add Application FBLogGen

[OUTPUT]
    Name stdout
    Match sample
[OUTPUT]
    Name            azure_logs_ingestion
    Match           sample
    client_id       00000000-0000-0000-0000-000000000000
    tenant_id       00000000-0000-0000-0000-000000000000
    client_secret   abcdabcdacbdcbadcbadcdicasiunclaisnclsdc
    dce_url         https://dce-abcd.westus3-1.ingest.monitor.azure.com
    dcr_id          dcr-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
    table_name      ladcr_CL    #----> _CL must be appended here
    time_generated  true
    time_key        Time
    Compress        true

For testing I've followed this tutorial and this Logs Ingestion REST API reference to create DCE, DCR and sample data (with and without GZip compression and Time key).

I have tested it with fluentbit without any crash/mem-leak.

I'll do a long-term test, if successful I'll probably do a PR as well!

[agup006]

Really cool @kforeverisback !! Let’s try to get it in for 2.1 in time for KubeCon EU

[kforeverisback]

@agup006 Do we have a timeline/deadline for the 2.1 release? or KubeCon EU?

[kforeverisback]

Just created a PR for the DCR output plugin. https://github.com/fluent/fluent-bit/pull/7155

[rhavenn]

Sweet. Thank you.

[seeavosh]

I am not sure if I am making a mistake or if recent changes are breaking this

It seems the stream name and table name now have the extra word "Input" Take a look at this If I provide just the table name it will not find it when I add input I get 204 Error - Thanks

    "dataFlows": [
        {
            "streams": [
                "Custom-Input-TableName_CL"
            ],
            "destinations": [
                "myloganalyticsworkspace"
            ],
            "outputStream": "Custom-TableName_CL"
        }