search

ddbnl / office365-audit-log-collector

Collect / retrieve Office365, AzureAD and DLP audit logs and output to PRTG, Azure Log Analytics Workspace, SQL, Graylog, Fluentd, and/or file output.
https://ddbnl.github.io/office365-audit-log-collector/
MIT License
105 stars 40 forks source link

error in latest version #63

Open vvhor opened 6 months ago

vvhor commented 6 months ago

Hello,

I'm trying the latest version but I got this error

thread 'main' panicked at src\api_connection.rs:59:33:
Could not parse API login reply: error decoding response body: missing field `access_token` at line 1 column 623
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

the request is:

./OfficeAuditLogCollector.exe --tenant-id "xxxxxxx" --client-id "xxxxx" --secret-key "xxxxx" --config config.yaml

config file:

collect:
  skipKnownLogs: True
  workingDir: ./
  maxThreads: 50
  globalTimeout: 5
  retries: 3
  hoursToCollect: 168
  contentTypes:
    Audit.General: True
    Audit.AzureActiveDirectory: True
    Audit.Exchange: True
    Audit.SharePoint: True
    DLP.All: True 
output:
  file:
    path: 'output.csv'
    separateByContentType: True
    separator: ';'

I'm using the client on window system

ddbnl commented 6 months ago

Hi @vvhor,

Have you successfully used the older version(s) with the same app registration before, or are you trying for the first time? If it's the first time you could check if the API permissions are properly set, and if auditing is enabled for the tenant (this might take a while to sync after enabling it). Both these actions are described in README.md.

If it was already working before then we'll have to figure where it's coming from. I'm currently working on a new release with improved logging, so once that's out in the coming days I will link it here. Then hopefully we can see more with the increased logs.

vvhor commented 6 months ago

Hi,

I've used it in previous version on other tenant. With this tenant it's the first time.

I've followed all of the steps in the READEME some days ago

vvhor commented 6 months ago

I'm currently working on a new release with improved logging, so once that's out in the coming days I will link it here. Then hopefully we can see more with the increased logs.

did you have an estimation for this release?

ddbnl commented 6 months ago

I have released the new version with fixed logging and also extended logging, hopefully we'll be able to capture the error:

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.1

Make sure to also enable logging in the config:

log:
  path: './log.txt'
  debug: True

If you get it working consider disabling debug again, it's very noisy. Let me know what it does for you.

vvhor commented 6 months ago

Hi,

many thanks. Now the log is very helpful. I'll do some test and let you know

ddbnl commented 6 months ago

As a heads up, there's a new release that added an interactice interface that can be used for testing. If you have the new release, you can run the command as you did before, but add the '--interactive' command line parameter. This allows you to test the connection and immediately see the logs for any errors.

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.2

Screenshot

vvhor commented 6 months ago

Hello,

I'm now having different errors in "Run Collector":

[00:00:01.339] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.339] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.339] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.339] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.341] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.341] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.349] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.349] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.350] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.350] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.350] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.350] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.352] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.352] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.352] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.352] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.356] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.356] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.358] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.358] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.398] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.398] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.401] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.401] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.403] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.403] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.407] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.407] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.408] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.408] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.409] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.409] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.411] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.411] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-23T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.412] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.412] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.412] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.412] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.412] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.412] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.414] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.414] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-18T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.416] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.416] (5f60) WARN   Retry blob 1 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.416] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.416] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.SharePoint&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.417] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.417] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.AzureActiveDirectory&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.418] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.418] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-21T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.420] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.420] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-22T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.420] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.420] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=DLP.All&startTime=2024-03-19T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.420] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.420] (5f60) WARN   Retry blob 2 https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.Exchange&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123
[00:00:01.422] (52b4) INFO   Blobs found: 0
Blobs successful: 0
Blobs failed: 0
Blobs retried: 34
Logs saved: 0

[00:00:01.422] (2e58) WARN   Err getting blob JSON error decoding response body: invalid type: map, expected a sequence at line 1 column 0
[00:00:01.424] (2e58) ERROR  Err getting blob response error sending request for url (https://manage.office.com/api/v1.0/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx/activity/feed/subscriptions/content?contentType=Audit.General&startTime=2024-03-20T11:38:02Z&endTime=2024-03-25T11:38:02Z&PublisherIdentifier=12345678-1234-1234-1234-123456789123): connection error: Either the application has not called WSAStartup, or WSAStartup failed. (os error 10093)
[00:00:01.424] (2e58) ERROR  Could not resend failed blob, dropping it: send failed because receiver is gone

Non error in "Test API Connection"

ddbnl commented 6 months ago

That's odd, so far I'm not able to reproduce. Best we can do is improve logging. I've added the full output of the JSON response as a debug log, in the section where you are receiving the error. This should give us the full response you are getting from the API. Can you run it again with the latest release, and enabling debug logging?

Also, just to ensure you are not being rate limited, could you use a publisher ID? For the ID you can just use your tenant ID again. This will isolate your requests to avoid rate limiting as much as possible. You can the executable like before, but adding "--publisher-id %tenant-id%"

https://github.com/ddbnl/office365-audit-log-collector/releases/tag/v2.3.3

vvhor commented 5 months ago

PS. The new version not write the log in interactive mode.

in attach the full log log.txt

vvhor commented 5 months ago

Hi,

just to ask if you have any news about the errors..

vvhor commented 5 months ago

can I do any other test to help you?