No logs received

[flotpg]

Hi,

I'm just starting with this nice solution but I'm stuck and I can't get any logs.

Auditing is enabled in tenant

Get-AdminAuditLogConfig | FL UnifiedAuditLogIngestionEnabled
UnifiedAuditLogIngestionEnabled : True

Starting run @ 2022-04-27 16:26:48.850581. Content: ['Audit.General', 'Audit.AzureActiveDirectory', 'Audit.Exchange', 'Audit.SharePoint', 'DLP.All'].
Retrieving Audit.General. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:26:49.
Retrieving Audit.AzureActiveDirectory. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:26:49.
Retrieving Audit.Exchange. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:26:49.
Retrieving Audit.SharePoint. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:26:49.
Retrieving DLP.All. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:26:49.
Finished. Total logs retrieved: 0. Total logs with errors: 0. Run time: 0:00:02.546457.
GraylogInterface reports: 0 successfully sent, 0 errors

Strangely the log file "collector.log" is not created when I run the collector: ./LINUX-OfficeAuditLogCollector-V1.3 AAD-Tenant-ID AAD-App-ID AAD-App-SecretKey --config ./fullConfig.yaml

fullConfig.yaml

log:  # Log settings. Debug will severely decrease performance
  path: 'collector.log'
  debug: True
collect:  # Settings determining which audit logs to collect and how to do it
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.Exchange: True
    Audit.SharePoint: True
    DLP.All: True
  maxThreads: 50
  retries: 3  # Times to retry retrieving a content blob if it fails
  retryCooldown: 3  # Seconds to wait before retrying retrieving a content blob
  autoSubscribe: True  # Automatically subscribe to collected content types. Never unsubscribes from anything.
  skipKnownLogs: True  # Remember retrieved log ID's, don't collect them twice
  resume: True  # Remember last run time, resume collecting from there next run
  hoursToCollect: 72  # Look back this many hours for audit logs (can be overwritten by resume)
filter:  # Only logs that match ALL filters for a content type are collected. Leave empty to collect all
  Audit.General:
  Audit.AzureActiveDirectory:
  Audit.Exchange:
  Audit.SharePoint:
  DLP.All:
output:
  graylog:
    enabled: true
    address: 127.0.0.1
    port: 5555

[flotpg]

Got a log by adding -l ./collector.log -d

Starting run @ 2022-04-27 16:31:26.894792. Content: ['Audit.General', 'Audit.AzureActiveDirectory', 'Audit.Exchange', 'Audit.SharePoint', 'DLP.All'].
Making API request using URL: "https://manage.office.com/api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/list"
Starting new HTTPS connection (1): login.microsoftonline.com:443
https://login.microsoftonline.com:443 "POST /TENANT-ID-TENANT-ID/oauth2/token HTTP/1.1" 200 1510
Logged in
Starting new HTTPS connection (1): manage.office.com:443
https://manage.office.com:443 "GET /api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/list HTTP/1.1" 200 342
Getting available content for type: "Audit.General"
Getting available content for type: "Audit.AzureActiveDirectory"
Retrieving Audit.General. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:31:27.
Getting available content for type: "Audit.Exchange"
Retrieving Audit.AzureActiveDirectory. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:31:27.
Getting available content for type: "Audit.SharePoint"
Making API request using URL: "https://manage.office.com/api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log"
Getting available content for type: "DLP.All"
Retrieving Audit.Exchange. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:31:27.
Making API request using URL: "https://manage.office.com/api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log"
Retrieving Audit.SharePoint. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:31:27.
Starting new HTTPS connection (1): login.microsoftonline.com:443
Retrieving DLP.All. Start time: 2022-04-26T14:22:09. End time: 2022-04-27T14:31:27.
Making API request using URL: "https://manage.office.com/api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log"
Making API request using URL: "https://manage.office.com/api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log"
Starting new HTTPS connection (1): login.microsoftonline.com:443
Making API request using URL: "https://manage.office.com/api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log"
Starting new HTTPS connection (1): login.microsoftonline.com:443
Starting new HTTPS connection (1): login.microsoftonline.com:443
Starting new HTTPS connection (1): login.microsoftonline.com:443
https://login.microsoftonline.com:443 "POST /TENANT-ID-TENANT-ID/oauth2/token HTTP/1.1" 200 1510
Logged in
https://login.microsoftonline.com:443 "POST /TENANT-ID-TENANT-ID/oauth2/token HTTP/1.1" 200 1510
Starting new HTTPS connection (1): manage.office.com:443
Logged in
https://login.microsoftonline.com:443 "POST /TENANT-ID-TENANT-ID/oauth2/token HTTP/1.1" 200 1510
https://login.microsoftonline.com:443 "POST /TENANT-ID-TENANT-ID/oauth2/token HTTP/1.1" 200 1510
Starting new HTTPS connection (1): manage.office.com:443
https://login.microsoftonline.com:443 "POST /TENANT-ID-TENANT-ID/oauth2/token HTTP/1.1" 200 1510
Logged in
Logged in
Logged in
Starting new HTTPS connection (1): manage.office.com:443
Starting new HTTPS connection (1): manage.office.com:443
Starting new HTTPS connection (1): manage.office.com:443
https://manage.office.com:443 "GET /api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log HTTP/1.1" 400 301
Got 1 content blobs of type: "Audit.General"
https://manage.office.com:443 "GET /api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log HTTP/1.1" 400 301
https://manage.office.com:443 "GET /api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log HTTP/1.1" 400 301
Got 1 content blobs of type: "Audit.Exchange"
Got 1 content blobs of type: "Audit.SharePoint"
https://manage.office.com:443 "GET /api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log HTTP/1.1" 400 301
Got 1 content blobs of type: "DLP.All"
https://manage.office.com:443 "GET /api/v1.0/TENANT-ID-TENANT-ID/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2022-04-26T14:22:09&endTime=2022-04-27T14:31:27&PublisherIdentifier=/tmp/_MEIFNesQT/AuditLogCollector.log HTTP/1.1" 400 301
Got 1 content blobs of type: "Audit.AzureActiveDirectory"
Finished. Total logs retrieved: 0. Total logs with errors: 0. Run time: 0:00:01.433801.
GraylogInterface reports: 0 successfully sent, 0 errors

[flotpg]

ManiFest of my AAD App registration:

{
    "id": "6a1c9f37-c846-4136-9f98-2dfdfbdb5e8a",
    "acceptMappedClaims": null,
    "accessTokenAcceptedVersion": null,
    "addIns": [],
    "allowPublicClient": null,
    "appId": "c270d7e2-ce98-47a6-a557-b12975e2c40c",
    "appRoles": [],
    "oauth2AllowUrlPathMatching": false,
    "createdDateTime": "2022-04-27T13:43:59Z",
    "description": null,
    "certification": null,
    "disabledByMicrosoftStatus": null,
    "groupMembershipClaims": null,
    "identifierUris": [],
    "informationalUrls": {
        "termsOfService": null,
        "support": null,
        "privacy": null,
        "marketing": null
    },
    "keyCredentials": [],
    "knownClientApplications": [],
    "logoUrl": "https://aadcdn.msftauthimages.net/c1c6b6c8-r55h-wjqdm8wgmnfjuaemwehvzqkq7kd6vr4rnalwsc/appbranding/tlvf10jwni-fzny9vuol0zdbbgj2blqj-6gextg450i/1033/bannerlogo?ts=637866639079187665",
    "logoutUrl": null,
    "name": "graylog-office365-audit-log-collector-TI-7307",
    "notes": "TI-7307\nhttps://github.com/ddbnl/office365-audit-log-collector",
    "oauth2AllowIdTokenImplicitFlow": false,
    "oauth2AllowImplicitFlow": false,
    "oauth2Permissions": [],
    "oauth2RequirePostResponse": false,
    "optionalClaims": null,
    "orgRestrictions": [],
    "parentalControlSettings": {
        "countriesBlockedForMinors": [],
        "legalAgeGroupRule": "Allow"
    },
    "passwordCredentials": [
        {
            "customKeyIdentifier": null,
            "endDate": "2024-04-27T14:20:27.175Z",
            "keyId": "776cfbf7-e4f4-4bbe-8bc0-66cbf3e30411",
            "startDate": "2022-04-27T14:20:27.175Z",
            "value": null,
            "createdOn": "2022-04-27T14:20:46.5669135Z",
            "hint": "qLm",
            "displayName": "LINUX-OfficeAuditLogCollector-V1.3"
        }
    ],
    "preAuthorizedApplications": [],
    "publisherDomain": "theprojectgroup.com",
    "replyUrlsWithType": [],
    "requiredResourceAccess": [
        {
            "resourceAppId": "c5393580-f805-4401-95e8-94b7a6ef2fc2",
            "resourceAccess": [
                {
                    "id": "4807a72c-ad38-4250-94c9-4eabfe26cd55",
                    "type": "Role"
                },
                {
                    "id": "594c1fb6-4f81-4475-ae41-0c394909246c",
                    "type": "Role"
                },
                {
                    "id": "e2cea78f-e743-4d8f-a16a-75b629a038ae",
                    "type": "Role"
                }
            ]
        }
    ],
    "samlMetadataUrl": null,
    "signInUrl": "https://github.com/ddbnl/office365-audit-log-collector",
    "signInAudience": "AzureADMyOrg",
    "tags": [],
    "tokenEncryptionKeyId": null
}

[ddbnl]

I've located the bug and it happens to coincide with the next feature I wanted to implement.

The issue is that the management API has two constraints:

1. You can only retrieve logs in a time span of 24 hours or less;
2. You cannot retrieve logs further back than 7 days.

So when you entered 'hoursToCollect: 72', this is impossible unless it's done in 3 separate runs (each 24 hours). The program should be able to do this for you but I'm still implementing this logic (ETA this weekend). It should have at least thrown an error so you would've known, that's my bad.

Could you try changing hoursToCollect to 24 and let me know if this workaround works for you? I will give you an update when the new version drops that supports >24 hour time spans.

Edit: Also delete the 'last_run_times' and 'known_content' files (if they exist) before trying again, for a clean start.

[ddbnl]

Went faster than expected. I have a new version ready for you to test if you want to. The new versions allows 'hoursToCollect' to be set up to 168 hours (hard limit on API). If you keep your config file set to 72 hours, you will see it now performs 3 separate runs (each 24 hours) automatically. This will happen only once as you have 'resume' set to true, so the next timespan will be the difference between your last- and current run. The new version should also fix the logging error you had.

Use this link to download the Linux executable for version 1.4 (I will add it as an official release later after we test it).

Thanks for opening the issue :) Let me know if this works now.

[flotpg]

Hey @ddbnl, switching to 24 hours and deleting the two files (last_run_times known_content) works!

Starting run @ 2022-04-29 21:51:15.618675. Content: ['Audit.General', 'Audit.AzureActiveDirectory', 'Audit.Exchange', 'Audit.SharePoint', 'DLP.All']. Retrieving Audit.General. Start time: 2022-04-28T19:51:15. End time: 2022-04-29T19:51:15. Retrieving Audit.AzureActiveDirectory. Start time: 2022-04-28T19:51:15. End time: 2022-04-29T19:51:15. Retrieving Audit.Exchange. Start time: 2022-04-28T19:51:15. End time: 2022-04-29T19:51:15. Retrieving Audit.SharePoint. Start time: 2022-04-28T19:51:15. End time: 2022-04-29T19:51:16. Retrieving DLP.All. Start time: 2022-04-28T19:51:15. End time: 2022-04-29T19:51:16. Finished. Total logs retrieved: 6054. Total logs with errors: 0. Run time: 0:02:11.611529. GraylogInterface reports: 6054 successfully sent, 0 errors

[flotpg]

With the v1.4 I get this error (even with 24hours) /LINUX-OfficeAuditLogCollector-V1.4: 8: Syntax error: newline unexpected

[ddbnl]

I just had the same issue, using WGET to download the new client. Turns out I forgot to give you the raw content link. My bad!

Try this link instead:

https://github.com/ddbnl/office365-audit-log-collector/raw/master/Linux/LINUX-OfficeAuditLogCollector-V1.4

Good to hear the workaround is working at least.

[flotpg]

Mate! Awesome - That works. Thanks a lot.

By the way, it seems the -l (log path) option got dumped?

LINUX-OfficeAuditLogCollector-V1.4: error: unrecognized arguments: -l ./collector.log

Output:

Starting run @ 2022-04-29 22:59:14.741016. Content: deque(['Audit.General', 'Audit.AzureActiveDirectory', 'Audit.Exchange', 'Audit.SharePoint', 'DLP.All']).
Retrieving Audit.General. Start time: 2022-04-26T20:59:14. End time: 2022-04-27T20:59:14.
Retrieving Audit.General. Start time: 2022-04-27T20:59:14. End time: 2022-04-28T20:59:14.
Retrieving Audit.General. Start time: 2022-04-28T20:59:14. End time: 2022-04-29T20:59:14.
Retrieving Audit.AzureActiveDirectory. Start time: 2022-04-26T20:59:14. End time: 2022-04-27T20:59:14.
Retrieving Audit.AzureActiveDirectory. Start time: 2022-04-27T20:59:14. End time: 2022-04-28T20:59:14.
Retrieving Audit.AzureActiveDirectory. Start time: 2022-04-28T20:59:14. End time: 2022-04-29T20:59:14.
Retrieving Audit.Exchange. Start time: 2022-04-26T20:59:14. End time: 2022-04-27T20:59:14.
Retrieving Audit.Exchange. Start time: 2022-04-27T20:59:14. End time: 2022-04-28T20:59:14.
Retrieving Audit.Exchange. Start time: 2022-04-28T20:59:14. End time: 2022-04-29T20:59:14.
Retrieving Audit.SharePoint. Start time: 2022-04-26T20:59:14. End time: 2022-04-27T20:59:14.
Retrieving Audit.SharePoint. Start time: 2022-04-27T20:59:14. End time: 2022-04-28T20:59:14.
Retrieving Audit.SharePoint. Start time: 2022-04-28T20:59:14. End time: 2022-04-29T20:59:14.
Retrieving DLP.All. Start time: 2022-04-26T20:59:15. End time: 2022-04-27T20:59:15.
Retrieving DLP.All. Start time: 2022-04-27T20:59:15. End time: 2022-04-28T20:59:15.
Retrieving DLP.All. Start time: 2022-04-28T20:59:15. End time: 2022-04-29T20:59:14.
Finished. Total logs retrieved: 37942. Total logs with errors: 0. Run time: 0:00:38.571161.
GraylogInterface reports: 37942 successfully sent, 0 errors

[flotpg]

By the way a small side note for graylog input. I created two extractors on the input

Json extractor > message > extracts all fields of the message like

There is a problem with the timestamps. They look like this: CreationTime": "2022-04-28T05:46:37

I added an input extractor to convert this and copy the value to the timestamp field: Extractor type: Copy input Source field: CreationTime Add a converter of "Convert to date type" with this value: yyyy-MM-dd'T'HH:mm:ss

We should note this somewhere or premium solution: ship the creationDate as graylog / elasticsearch compatible timestamp :)

[ddbnl]

I dropped the command line switches in favor of the config file because with all the new outputs the amount of cmd line args were exploding; if this turns out to be a hindrance to people I will consider putting them back in.

I definitely agree with presenting a Graylog compatible timestamp. Unfortunately my company stopped using Graylog so I'll need some time to set up a test. Alternatively, do you know how Graylog prefers the timestamps? I could add a field

timestamp: seconds (UNIX epoch time)

which is converted from "CreationTime" to all logs specifically for the Graylog interface. If you think that would do it I could throw it in a commit for you to test.

[flotpg]

Cool.

Here an examples from my other logs: Timestamp: 2022-04-29 23:36:06.023

[ddbnl]

Thanks for the example. Decided to do this now since it could be done relatively quickly. Below is a link to the new executable (1.4.1). It should add timestamp fields for the Graylog output: 'timestamp': '2022-04-29 07:05:33.000000'. If you have an opportunity to test sometime let me know if it works :)

https://github.com/ddbnl/office365-audit-log-collector/raw/master/Linux/LINUX-OfficeAuditLogCollector-V1.4.1

[flotpg]

hm. not sure why it's not working but must be something in the formate which graylog doesn't like: gl2_processing_error Replaced invalid timestamp value in message with current time - Value <2022-04-29 15:29:33.000000> caused exception: Invalid format: "2022-04-29 15:29:33.000000" is malformed at "000".

[ddbnl]

Ah I think I see the issue already. It insists on having three microseconds precision, rather than 6 (which python gives you by default). It's a small fix so I've made a new executable (1.4.2). Link below. Thanks for the help with troubleshooting, it's a bit of trial and error but I imagine it should work now and be useful to the others using the Graylog output.

https://github.com/ddbnl/office365-audit-log-collector/raw/master/Linux/LINUX-OfficeAuditLogCollector-V1.4.2

For reference, timestamps are now formatted as "2022-04-30 11:40:38.000", instead of "2022-04-30 11:40:38.000000".

[flotpg]

Awesome, but I get a 404... EDIT: trying https://raw.githubusercontent.com/ddbnl/office365-audit-log-collector/master/Linux/LINUX-OfficeAuditLogCollector-V1.5

Now i just need to get time zone right ;)

[ddbnl]

Argh, a new commit pushed out the link. I'll use perma links next time such as the one below which should work:

https://github.com/ddbnl/office365-audit-log-collector/raw/6bf0c608d26c269a0a49c4fd82be689316a07545/Linux/LINUX-OfficeAuditLogCollector-V1.4.2

[flotpg]

1.4.2 also works... My zone is Europe/Berlin and this seems to also be correct:

Great work mate, much appreciated!

[ddbnl]

Awesome! No worries, happy that this tool can be of use to other people as well. I'll close this issue; if you have any other issues/questions/requests then feel free to open another one in the future.

[flotpg]

Thanks a lot. Just setting up a Cron job ;) ... Do you know how frequent MS updates the audit log?