Missing Output to Log Analytics Workspace in latest Version

ddbnl / office365-audit-log-collector

Collect / retrieve Office365, AzureAD and DLP audit logs and output to PRTG, Azure Log Analytics Workspace, SQL, Graylog, Fluentd, and/or file output.

https://ddbnl.github.io/office365-audit-log-collector/

MIT License

106 stars 40 forks source link

Missing Output to Log Analytics Workspace in latest Version #61

Open Gill-Bates opened 8 months ago

Gill-Bates commented 8 months ago

Thanks a lot for your effort to keep this tool running!

I am collecting the O365 Logs for over 10,000 Users and put them into a Log Analytics workspace. I would really appreachiate to have this feature in your latest Rust-Version.

ddbnl commented 8 months ago

Hiya,

No worries, thanks for using it :)

I can add the interface, I just wasn't sure which ones were actually being used. I'll have time to work on it monday or tuesday, so you can expect it in the coming days. I'll update here when it's available!

ddbnl commented 8 months ago

A new version is available that supports the Azure Log Analytics interface. Due to the full rewrite some small changes might be required:

Command line arguments need full syntax
OMS interface had a shared key config option which is insecure, instead the key should be passed as a command line argument "--oms-key"
Container available in github container registry if you want it (see README)
Schedule option has been removed, since it's probably better to use cron/task scheduler, or run the container on a schedule

New syntax would look like:

OfficeAuditLogCollector --tenant-id "11111111-1111-1111-1111-1111111111111" --client-id "111111111-1111-1111-11111111111111111111" --secret-key "12345" --config logAnalytics.yaml --oms-key "12345"

Let me know how it works for you. It's a full rewrite but I tested it as much as I could by myself.

Release: https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3

Gill-Bates commented 7 months ago

@ddbnl I have tested your recent version against my LogAnalytics-Workspace and must confirm, that no Tables inside Log Analytics workspace appear.

However, I have noticed the following points:

It takes two minutes for the data to be visible in the LogAnalytics workspace. This should be mentioned as a comment in the config. This is an Azure behaviour and affects all applications.

The log file is not generated as required. My config.yaml looks like this:

log: 
path: '/root/nextgen/collector.log'
debug: True
collect:
workingDir: '/root/nextgen'

Console Output

./LINUX-OfficeAuditLogCollector --tenant-id "*****" --client-id "*****" --secret-key "*****" --config config.yaml
SEND
Blobs found: 315
Blobs successful: 315
Blobs failed: 0
Blobs retried: 0
Logs saved: 26747

The Custom Log Table are still missing:

ddbnl commented 7 months ago

@Gill-Bates,

I'll take a look asap. I can at least confirm the logging issue, I'll fix that and link the new release here when it's done.

In regards to log analytics, do you also have the output defined in the config, and the log analytics key as a command line arg? The config (based on your post) would be:

log: 
  path: '/root/nextgen/collector.log'
  debug: True
collect:
  workingDir: '/root/nextgen' 
output:
  azureLogAnalytics:
    workspaceId: 11111111-1111-1111-1111-1111111111111

And the command would be:

./LINUX-OfficeAuditLogCollector --tenant-id "*****" --client-id "*****" --secret-key "*****" --oms-key "12345" --config config.yaml

You can get the log analytics key as follows using AZ CLI:

az monitor log-analytics workspace get-shared-keys --resource-group my-rg --workspace-name my-oms --query "primarySharedKey"

If you already had all that then I will try to reproduce.

Thanks for testing!

Gill-Bates commented 7 months ago

@ddbnl i can confirm that I have configured the Collector as described above. For me it looks like that the Logs are fetched correctly. But when shifting them to the log analytics workspace, it will fail.

ddbnl commented 7 months ago

So far I have not been able to reproduce, for me the tables show up. My test is creating a log analytics workspace (just next, next, finish) and putting the workspace ID in the config. Then getting the shared key:

az monitor log-analytics workspace get-shared-keys --resource-group my-rg --workspace-name my-oms --query "primarySharedKey"

And putting the result in the '--oms-key' parameter. This gives the following result:

The result config is (also to enable logging):

log:
  path: './log.txt'
  debug: True
collect:
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.Exchange: True
    Audit.SharePoint: True
    DLP.All: True
output:
  azureLogAnalytics:
    workspaceId: 11111-1111111-11111111-111111111-11111111

I have released a new version with fixed logging and also extended logging, hopefully we'll be able to capture the error:

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.1

Let me know what it does for you.

ddbnl commented 7 months ago

I just fixed a bug that caused issues with the 'hoursToCollect' parameter. If you were using it, you might want to try with the latest release.

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.2

It also added an interactive interface that can be used for testing. If you have the new release, you can run the command as you did before, but add the '--interactive' command line parameter. This allows you to run the collector and immediately see the logs for any errors.

Screenshot