Open jorritfolmer opened 4 years ago
+1 O365 is moving in this same direction.
Microsoft has set a date of October 2022.
Today, we are announcing that, effective October 1, 2022, we will begin to permanently disable Basic Auth in all tenants, regardless of usage
Assuming the add-in is only collecting reports on a daily basis, the code can probably be kept simple and throw away the access token after each fetch (it will have expired by the time we reconnect to the mailbox). We just need to securely persist the access token and the initial secret (or just get a new access token each day)?
Thanks for the heads up! Getting a token from the Azure AD authorize endpoint should be easy using the MSAL Python lib. I just verified the IMAPClient library from Menno Finlay-Smits supports the XOAUTH2 method needed to submit tokens against the IMAP service. Looks like a fun little project for the summer to get it working for O365 and Google Workplace.
This Microsoft deadline seems to be holding for October so I expect the O365 mailbox pickups to break soon. Have most other people switched to file-based pickup?
If I were to take a crack at adding this capability:
a) Is there an AOB-friendly version of the TA could import to my local Splunk dev environment?
b) Is there an approach you had planned? Given the extra complexity/config options required for OAuth, I was envisioning a new/separate input type: dmarc_imap_oauth
. That would allow for extra fields to collect those options, e.g.:
"authority": "https://login.microsoftonline.com/Enter_the_Tenant_Name_Here",
"scope": ["https://graph.microsoft.com/.default"],
"endpoint": "https://graph.microsoft.com/v1.0/users"
Then a small refactoring of the Imap2Dir.get_dmarc_messages would allow basic or OAuth (based on the presence of the settings above)
Awesome, sounds good!
I've updated the repo to include TA-dmarc.aob_meta
which should allow you to import into AoB using these steps:
Package imported. I had to fiddle with the tar commands a bit to exclude .git files. For my own future reference, the (Windows) commands are:
SET ARCHIVE=%CD%\ta-dmarc-add-on-for-splunk_AOB.tar.gz
pushd \<PARENT_FOLDER_OF_TA>
DEL %ARCHIVE%
tar -cz --exclude .git --exclude .gitignore -f %ARCHIVE% TA-dmarc
popd
REM open to verify that the app folder is at the top of the archive
tar ztvf %ARCHIVE% | more
Any thoughts on the code structure/approach from b) above?
b) Is there an approach you had planned? Given the extra complexity/config options required for OAuth, I was envisioning a new/separate input type:
dmarc_imap_oauth
. That would allow for extra fields to collect those options, e.g.:"authority": "https://login.microsoftonline.com/Enter_the_Tenant_Name_Here", "scope": ["https://graph.microsoft.com/.default"], "endpoint": "https://graph.microsoft.com/v1.0/users"
Then a small refactoring of the Imap2Dir.get_dmarc_messages would allow basic or OAuth (based on the presence of the settings above)
Yes, agreed on all counts!
Unrelated to the OAuth matter, I'm trying to learn how to prepare add-ons for GitHub. Which version of the aob_meta file is the one you added to this project?
When I look in a different add-on I'm building, I see three
/etc/apps/splunk_app_addon-builder/local/validation/TA-NAME
/var/data/tabuilder/package/TA-NAME
/var/data/tabuilder/export_ta/TA-NAME
The first two seem to be updated when I run the AOB UI's Validate operation. The second two are updated when I run the Export operation. Based on this, I'm assuming that /var/data/tabuilder/package/TA-NAME
(always updated) is the one to grab for inclusion in a share-able project.
The addon builder generates everything in the TA-dmarc app you need to run it anywhere. This doesnt include the aob meta stuff but thats only needed to modify the addon forms. I would suggest to forget about the aob meta stuff until you have a working version, and only then (once) grab the aob meta file from the “export add-on” button in the builder.
The only thing to be aware of it to regularly move conf files and other stuff from the local/ directory to the default/ before comitting. Local/ is included in .gitignore. To verify if I have everything I usually do a git clone on a separate empty and fresh Splunk instance to see if I’ve included everthing in default/.
You are right. I found the AOB maddening and never could get on-disk .conf changes to to appear in the UI. I was trying to copy the existing dmarc_imap input to dmarc_imap_oauth2. I gave up and started work on the OAuth2 bits.
First draft here:
https://github.com/jorritfolmer/TA-dmarc/compare/master...hkelley:imap-oauth2?expand=1
Some items to note:
opt_imap_mailbox
(e.g. dmarc-reports@xyz.com) and added the opt_imap_folder
(e.g. INBOX).Hah! Looks good! Some short pointers in the readme would be nice indeed, as well as including the msal library under bin/. I’ll do some testing this weekend on my Google Workplace account.
I've updated the readme and added msal. I also took a crack at "re-wiring" the UI parameters for the existing IMAP basic input so that they will work with the new imap2dir.py methods.
If all works as hoped, the existing inputs will behave as-is and you can simply: 1) clone the dmarc_imap input as dmarc_imap_oauth2 2) add the params for mailbox, authority, and scope to the cloned input
Thanks! I've added the new input based on your commits in the oauth2
branch.
My time ran out so I only managed to test whether or not the old IMAP input works in a fresh install and upgrade scenario and it does. However, I had to revert your imap_folder rename to prevent breaking old IMAP inputs in upgrade scenario's. That leaves the new imap_mailbox field which I ended up renaming to imap_username as it most closely resembles a username to access the mailbox with the previously acquired token.
Can you checkout and test the oauth2
branch with your O365 setup?
I'm sure this is a rookie mistake, but I can't seem to get the TA installed. I have:
Uninstalled+reinstalled Splunk
Cloned the branch you created
# git status
On branch oauth2
Your branch is up to date with 'origin/oauth2'.
Tar'd it into a file for import
tar -czf \temp\TA-dmarc.tar.gz TA-dmarc
Imported the .gz file. Receive this Message
Unable to initialize modular input "dmarc_pop3" defined in the app "TA-dmarc": Introspecting scheme=dmarc_pop3: script running failed (exited with code 255)..
The app partially installs but the Add button for a credential is missing.
On the bright side, my outside-of-Splunk test harness runs fine with the imap2dir.py package.
You can skip the tar step and git clone directly into etc\apps.
But TBH it looks like an issue with the TA. Not sure what exactly, it worked yesterday on Linux and a fresh Splunk 8.2.5.
I will try it on v8. This testing was on v9 (Windows)
Trying to keep it simple, I grabbed Splunk v8 and the latest TA-dmarc from Splunkbase. Installing the app on Windows, I am able to add a credential but not to create an input. I don't get any visible error messages.
Any idea which logs I should be peering into?
After I upgrade/overwrite to the new app version, the same error from Splunk v9 appears:
Unable to initialize modular input "dmarc_pop3" defined in the app "TA-dmarc": Introspecting scheme=dmarc_pop3: script running failed (exited with code 255)..
I tried swapping the sequence of the inputs in inputs.conf.spec
. Whichever one is first in the file (after dmarc_directory, which is fine) is the one reported in the UI's messages when I restart Splunk and open the app.
Unable to initialize modular input "dmarc_imap_oauth2" defined in the app "TA-dmarc": Introspecting scheme=dmarc_imap_oauth2: script running failed (exited with code 255)..
I can't reproduce the errors on a fresh Windows 2016 with fresh Splunk 8.2.8.
git clone https://github.com/jorritfolmer/TA-dmarc.git
in c:\program files\splunk\etc\apps
git checkout oauth2
index=_internal sourcetype="ta:dmarc:log"
Glad to hear that. I'll get a clean VM and retry (assuming we still need to verify the OAuth import).
Please do, I tried to use OAuth2 against my Google Workplace account, but the msal library won't allow it saying:
ValueError: Unable to get authority configuration for https://accounts.google.com/o/oauth2/v2/auth. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant Also please double check your tenant name or GUID is correct.
This was after adding validate_authority = False
to the msal.ConfidentialClientApplication()
statement. So somewhere in the future there will be a need to replace the msal library with a generic oauth2 library, or maybe vendor specific IMAP Oauth2 inputs.
I'll wait for your confirmation that the current setup works against O365 before publishing to Splunkbase.
I have used a brand new Windows server and a fresh install of Splunk. I'm still having the same issue.
If I install the app from the tar file, I get a timeout while installing.
Error connecting to /services/apps/local: The read operation timed out
If I manually copy the app, I get this Message in the UI.
Unable to initialize modular input "dmarc_imap_oauth2" defined in the app "TA-dmarc": Introspecting scheme=dmarc_imap_oauth2: script running failed (exited with code 255)..
I tried swapping the sequence of the inputs in inputs.conf.spec
. Whichever one is first in the file (after dmarc_directory, which is fine) is the one reported in the UI's Messages when I restart Splunk and open the app.
Can you try to use the source from Github? That way we can eliminate a number of variables. My suggestion would be to install Git from https://git-scm.com/download/win so you can omit a number of tar and/or copy steps.
git clone https://github.com/jorritfolmer/TA-dmarc.git in c:\program files\splunk\etc\apps
git checkout oauth2
The direct git checkout worked. I'm not sure why a file copy from my workstation's folder (on the same branch) didn't work, but I'll be the first to admit that I'm still learning a lot about Splunk dev practices.
I have it working now (still need to figure out why they aren't being indexed, but that's probably a config error of mine)
tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_messages: 76972 messages in folder Inbox
Two fixes are needed (this may also address the Google login failures you saw). I also have a lot of extra debug logging in imap2dir.py that we might as well include. I checked it into my fork.
Fixes:
opt_oauth2_scope = helper.get_arg('oauth2_scope')
Data ingestion issue found and logged at #43 . Unrelated to OAuth2 changes, just a side-effect of my testing on Windows.
Thanks! I've seen you committed the helper.get_arg fix. Can you be more specific what the authority url example should be in the setup screen?
It looks that's the final issue to clear before releasing this to Splunkbase?
In the case of Office 365, the desired authority and scope are:
opt_oauth2_authority https://login.microsoftonline.com/<tenant_id>
opt_oauth2_scope https://outlook.office365.com/.default
I may have previously mentioned a v2 for the authority, but in my testing, that is not accepted by O365
Thanks again for your efforts! I've updated the screens and docs. v4.1.0 is pending validation and cloud vetting on Splunkbase. It should be available somewhere next week for download. In the meantime, it's available here on Github in master with release tag 4.1.0.
Update: the new release won't make it to Splunk Cloud because Splunkbase found some library vulnerabilities in the stuff that the Splunk add-on builder generates. See #45. This doesn't seem to be easily fixable without a new add-on builder release.
Notes. For office365 the username and password should be replaced with Application ID and secret and not username/password.
I found that imap2dir.py line 131 calls msal.ConfidentialClientApplication which indicates "Acquires token for the current confidential client, not for an end user."
I am also not getting data into splunk after switching over to oauth.
I have enabled debug logging and ta_dmarc_dmarc_imap_oauth2.log looks like it is processing data.
Yes the Splunk add-on builder has fantastic feature for entering, storing and using credentials. You can add an account tab to an add-on through a single click! However that's all the customisability there is, so we'll probably have to live with the fact that for client id/secret the terminology doesn't match. To compensate, it is covered in the README that user/pass should be read as client id/secret.
Can you share more details about the IMAP Oauth2 input not working properly?
It was not clear looking at README to use an application id and secret.
As far as not working. I have not seen any DMARC reports injected into splunk since I switched from basic auth to OAuth. I have enabled debugging within the application.
oh wait!. I am wondering if it is starting back from the beginning of the mailbox......... and not limiting to things ( say unread )
looking at the logs ta_dmarc_dmarc_imap_oauth2.log it does look like it is processing message from back in 2019. Well. looks like I will have a while to wait. or I can move things to a new folder. I will report back sorry.
So to summarize the current state of this issue:
I'll leave this issue open to track future work for Google Workspace or others.
Thank you. Mine is deployed to Splunk cloud and is ingesting now. Am I correct that the KV Store "history log" is global, so my new input (using OAuth2) won't duplicate any reports that the old input had already ingested?
In case it helps anyone else making the transition from BASIC to OAuth, an error message like the following (visible from index=_internal sourcetype"ta:dmarc:log"
), might mean that you have an old O365 client access rule blocking your auth to the mailbox (even though Azure AD authentication of the service principal is working).
imaplib.IMAP4.error: SELECT command error: BAD [b'User is authenticated but not connected.']
Am I correct that the KV Store "history log" is global, so my new input (using OAuth2) won't duplicate any reports that the old input had already ingested?
I think the reports will be duplicated. It will only skip already ingested messages if the servername and username haven’t changed. The key on line 319 takes server, user and uid: https://github.com/jorritfolmer/TA-dmarc/blob/ba207caddcfb3b27bbebff87746cc714f5ebd3ac/bin/dmarc/imap2dir.py#L318-L323
Unfortunately the key will be different since you need to specify a client id for Oauth2 and an email for plain IMAP.
Because e.g. Gmail is dropping support for user/pass authentication: https://gsuiteupdates.googleblog.com/2019/12/less-secure-apps-oauth-google-username-password-incorrect.html