counteractive / o365beat

Elastic Beat for fetching and shipping Office 365 audit events
Other
66 stars 27 forks source link

Publisher ID is not set #17

Closed jjkoster closed 4 years ago

jjkoster commented 4 years ago

When using o365beat I receive the error:

{"code":"AF429","message":"Too many requests. Method=GetContents, PublisherId=00000000-0000-0000-0000-000000000000"}

It looks like Publisher ID is not set from the config file from directory_id so the request get rate limited with the other 00000 etc Publisher IDs being used. I believe Publisher ID can be arbitrary from other Azure AD API scripts I have used.

My current config looks like this (hard coded values): tenant_domain: removed client_secret: removed client_id: removed # aka application id (GUID) directory_id: removed # aka tenant id (GUID) registry_file_path: ./o365beat.state

I have tried to set the directory as an environment variable as well.

Thanks for any help.

chris-counteractive commented 4 years ago

Thanks for the feedback! Could you confirm which version you're using, on which platform, and what calls are getting throttled? We fixed a version of this in a063d06722205678b79bc1051e1685bd5e4dc3b7, it should be properly appended to all throttle-able calls for a while now, but I might have missed one!

If you'd prefer to send logs or other info via email, you can reach me at chris@counteractive.net. Thanks again!

jjkoster commented 4 years ago

Hi Chris,

I'm using 7.4.0, on Oracle Linux 7.6

The throttled request is for contenttype=Audit.AzureActiveDirectory

If you need more info, I'll send to your email address.

jjkoster commented 4 years ago

Sorry version 1.4.2

chris-counteractive commented 4 years ago

That's the current version, so there's definitely something else going on. To diagnose I'll need to see the logging output with all the debugging on, you can log to stderr (on the command line) with:

# and whatever other flags you're using when you have the issue
./o365beat -e -d "*" 

Please send as much of that output via email as you can, and if you could send your config file too (you can of course remove anything sensitive), that'll help. Thanks so much for reaching out, I'm looking forward to sorting out what's going on!

chris-counteractive commented 4 years ago

Sure enough, it turns out some tenancies can be so busy that even asking what content is available gets throttled! Thank you for reporting the issue, it should be a straightforward fix: every API call needs PublisherIdentifier appended, not just the high-volume calls. Sorry for not catching that sooner, we'll get it pushed in a new version this week.

jjkoster commented 4 years ago

Thanks Chris. That'd be great. Much appreciated.

On Tue, 26 Nov 2019 at 16:05, Chris Hendricks notifications@github.com wrote:

Sure enough, it turns out some tenancies can be so busy that even asking what content is available gets throttled! Thank you for reporting the issue, it should be a straightforward fix: every API call needs PublisherIdentifier appended, not just the high-volume calls. Sorry for not catching that sooner, we'll get it pushed in a new version this week.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/counteractive/o365beat/issues/17?email_source=notifications&email_token=AIXLJGMMVKJXPC6W3Z4FIG3QVSVA3A5CNFSM4JRRFJL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFEXHBY#issuecomment-558461831, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIXLJGIX337IKAC3YZEGW43QVSVA3ANCNFSM4JRRFJLQ .

chris-counteractive commented 4 years ago

@jjkoster: I pushed the Publisher ID into all API requests, so you shouldn't see this error once we issue a new release. I'm going to try to include a couple other fixes before we push that out, but it should be very soon. Thanks again for the feedback!

jjkoster commented 4 years ago

Thanks Chris, looking forward to it.

chris-counteractive commented 4 years ago

Thanks for your patience @jjkoster: release v1.4.3 contains this fix, please let me know if you run into anything else.