msgraph DeviceCode auth needs new permission after #334 fixes

[jdghub]

I'm using the #334 build to get the credentials cache and batch size fixes, and using DeviceCode auth because I'm logging on to what was originally a shared mailbox. But while I can authenticate via DeviceCode successfully using the original 8.3.0 PyPI build, after logging on using the #334 build I get a prompt that the mailbox

needs permission to access resources in your organization that only an admin can grant. 
Please ask an admin to grant permission to this app before you can use it.

ParseDMARC already has User.Read and Mail.ReadWrite permissions (which allows 8.3.0 to authenticate successfully.) What additional permission is needed as a result of the #334 changes?

[nathanthorpe]

could be related to Mail.ReadWrite.Shared permission. Are you logging on as another user?

[jdghub]

Just getting back to this now.

No, logging in as the shared mailbox, which allows direct logon.

I just tried again. After entering the device code and then logging in as the shared mailbox, the prompt is:

[domain]
needs permission to access resources in your organization that only an admin can grant. Please ask an admin to grant permission to this app before you can use it.

Have an admin account? Sign in with that account
Return to the application without granting consent

In the past I've just canceled at this point, but this time I clicked on the first link and signed in with a global admin account. This produced a new prompt to consent to permissions:

[admin]@[domain]
Permissions requested
ParseDMARC
[domain]

This application is not published by Microsoft.

This app would like to:
 Read and write mail you can access
 Maintain access to data you have given it access to
 View your basic profile

Consent on behalf of your organization

Accepting these permissions means that you allow this app to use your data as specified in their terms of service and privacy statement. You can change these permissions at https://myapps.microsoft.com.
Important: Only accept if you trust the publisher and if you downloaded this app from a store or website you trust. Microsoft is not involved in licensing this app to you.
Does this app look suspicious? Report it here

I consented, which was accepted, and logged out.

But now there's a new issue, which may be temporary but if so is there a way to speed up the resolution?

I then ran ParseDMARC again but got a new error ERROR:cli.py:948:Mailbox Error: 'value' which I assumed was due to my being still authenticated as the admin rather than the shared mailbox. I ended all the admin sessions, but now ParseDMARC gives a new error:

DeviceCodeCredential.get_token failed: Interactive authentication is required to get a token. Call 'authenticate' to begin.
Content: {"error":"invalid_grant","error_description":"AADSTS50173: The provided grant has expired due to it being revoked, a fresh auth token is needed. The user might have changed or reset their password. The grant was issued on...

This makes sense, except why isn't ParseDMARC automatically requesting a new token in this case? Has it cached the previous token somewhere and if so how can I clear it so I can start again as the shared mailbox?

[nathanthorpe]

This error Mailbox Error: 'value' came up in another issue but I haven't been able to replicate it. I'm curious as to how that issue happens.

Try deleting the .token file maybe? It does have a cache but it may need some work.

Also according to Microsoft you aren't supposed to directly log on as the shared mailbox so that case wasn't accounted for. The logic to detect whether to add the Mail.ReadWrite.Shared might need to be changed.

[jdghub]

Thanks, The good news is that deleting the .token file allowed re-authentication.

The bad news is that in all subsequent attempts DeviceCode authentication succeeds (including MFA), but ParseDMARC then fails with the same ERROR:cli.py:948:Mailbox Error: 'value'

I have tested logging in both as the shared mailbox and as a delegate with full access to it, but get the same Mailbox Error in both cases. I checked the app permissions in Azure AD and the list is now:

User.Read
Mail.ReadWrite
Mail.ReadWrite.Shared
offline_access
openid
profile

with the last 3 presumably having been added when I consented yesterday.

Under API permissions it was showing only the first 2 as Configured permissions with the other 4 showing as Other permissions granted for [tenant]. It recommended adding the last 4 as Configured permissions which I did. But both direct and delegate logins still give the same Mailbox Error.

I originally setup the app in Azure AD to not require user assignment, but also tried assigning it to both the shared mailbox and the delegate, but that didn't resolve the error.

DeviceCode auth was working with 8.3.0 before the #334 changes (though with the caching and batching issues that #334 fixed.) I was going to retest that but found that PyPI is now at 8.3.1, so has both the fixes and the Mailbox Error.

I will see if I it will work for us to convert the shared mailbox to a regular one, which should allow use of ClientSecret flow/auth.

[nathanthorpe]

Oh ok I will do some more testing in the next couple of days to see if I can reproduce that error.

I am actually using client secret auth with a shared mailbox so it should work for you as well if you use that flow, although you should run the scope limit command in the docs.

[jdghub]

I tried using ClientSecret auth initially, but when I ran the New-ApplicationAccessPolicy -AccessRight RestrictAccess ... command I got an error that pointed to this not working for a shared mailbox since it can't be a security principal. It was at that point that I switched to DeviceCode.

How did you get ClientSecret working for a shared mailbox?

[nathanthorpe]

Oh yeah for shared mailboxes you need to add them to a mail-enabled security group. I just have a shared mailbox called dmarc-reports@domain.com that is a member of a mail security group called report-group@domain.com. And then use the security group in the parameter.

Under the -PolicyScopeGroupID param, it talks about this https://learn.microsoft.com/en-us/powershell/module/exchange/new-applicationaccesspolicy?view=exchange-ps

[nathanthorpe]

I think I found the issue with value, if you don't have a batch_size in the config it throws that error. I will make a PR to fix. In the meantime you can add batch_size and it should work or use ClientSecret

[jdghub]

Thanks for that. I see the fix for ClientSecret, and can try that. But I'm still using DeviceCode and have run into a new issue. I'm now using the PyPI 8.3.1 release.

I first tried batch_size = 0 to use no limit but that didn't work - it reported 0 messages in Inbox (even though the Inbox has 81 emails.)

So I changed to batch_size = 1000, which does start successfully. But after processing 10/81 messages (currently in test mode), and logging the sending of 2 keepalive cmds, the cached authentication seems to fail:

DeviceCodeCredential.get_token failed: Interactive authentication is required to get a token. Call 'authenticate' to begin.
  Content: {"error":"invalid_grant","error_description":"AADSTS50196: The server terminated an operation because it encountered a client request loop. 
    Please contact your app vendor.\r\n
    Trace ID: cec73cbe-589f-4f92-ab09-b3e841435700\r\n
    Correlation ID: dd9ecef6-0225-47e1-8fce-a9c18a6a64ce\r\n
    Timestamp: 2022-09-17 21:13:44Z","error_codes":[50196],"timestamp":"2022-09-17 21:13:44Z","trace_id":"cec73cbe-589f-4f92-ab09-b3e841435700","correlation_id":"dd9ecef6-0225-47e1-8fce-a9c18a6a64ce","error_uri":"https://login.microsoftonline.com/error?code=50196"}
ERROR:cli.py:969:Mailbox Error: Interactive authentication is required to get a token. Call 'authenticate' to begin.
  [same details as above]

I deleted the .token file and re-authenticated for another try, but this run failed at the same point with the same error.

I can try ClientSecret if you think it will avoid this token caching issue, but if it's likely to be common I'll stick with DeviceCode for now.

[nathanthorpe]

ClientSecret does bypass the cache so might be good to try.

I was able to reproduce that error after 10 messages. Currently investigating...

[nathanthorpe]

Ok I think I got it, that error from MS is caused by fetching a new token to frequently. It was unable to find an existing token because the scopes are automatically determined by the graph client.

I added the scopes to the graph client initialization to force it to use those and it looks like that fixed the issue.

[jdghub]

As suggested I tried ClientSecret auth, via a mail-enabled security group with the DMARC Reports shared mailbox as a member. The New-ApplicationAccessPolicy command worked when I applied to the mail-enabled security group. I already had a client secret defined from my earlier attempt, so I updated the INI file to

[mailbox]
delete = False
watch = False
test = True
batch_size = 1000

[msgraph]
auth_method = ClientSecret
tenant_id = [ID]
client_id = [ID]
client_secret = [secret]
mailbox = dmarcreports@[domain]

But now when I run 8.3.1 it gives ERROR:cli.py:969:Mailbox Error: 'value' (same error as when I tried it before moving to DeviceCode.)

Unlike DeviceCode I don't get prompted to login to the shared mailbox, but I think that's expected when using ClientSecret. And for that reason MFA enabled for the shared mailbox shouldn't matter.

ClientSecret is working for you, any thoughts on what I may be missing?

[nathanthorpe]

Can you use the code in #352 instead of 8.3.1. Just do a git checkout of the branch and then run python setup.py install.

This will show better error messages.

You updated the app permissions in Azure AD right, can you screenshot it?

[jdghub]

Thanks, I've tested with 352 and the error is now logged as:

    INFO:cli.py:777:Starting parsedmarc
0it [00:00, ?it/s]
   ERROR:cli.py:969:Mailbox Error
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\cli.py", line 952, in _main
    reports = get_dmarc_reports_from_mailbox(
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\__init__.py", line 1085, in get_dmarc_reports_from_mailbox
    messages = connection.fetch_messages(reports_folder, batch_size=batch_size)
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\mail\graph.py", line 136, in fetch_messages
    folder_id = self._find_folder_id_from_folder_path(folder_name)
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\mail\graph.py", line 205, in _find_folder_id_from_folder_path
    return self._find_folder_id_with_parent(folder_name, None)
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\mail\graph.py", line 215, in _find_folder_id_with_parent
    folders = folders_resp.json()['value']
KeyError: 'value'

My pd.ini doesn't specify reports_folder but the default worked for DeviceCode. I tried adding reports_folder = Inbox but that didn't help.

Here's a screenshot of the app permissions, which are still as they were for DeviceCode auth. The API permissions page shows the same list in a different view.

[nathanthorpe]

Can you give it application permissions for Mail.ReadWrite instead of delegated?

[jdghub]

Thanks, the application permission was what was required, and I see why now.

There is one small new issue with this version: the output files aren't getting created before being written to. The first run failed with

  Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python310\Scripts\parsedmarc.exe\__main__.py", line 7, in <module>
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\cli.py", line 975, in _main
    process_reports(results)
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\cli.py", line 81, in process_reports
    save_output(results, output_directory=opts.output,
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\__init__.py", line 1325, in save_output
    append_json(os.path.join(output_directory, aggregate_json_filename),
  File "C:\Program Files\Python310\lib\site-packages\parsedmarc\__init__.py", line 1265, in append_json
    with open(filename, "r+", newline="\n", encoding="utf-8") as output:
FileNotFoundError: [Errno 2] No such file or directory: 'X:\\Scripts\\ParseDMARC\\Logs\\20220920\\agg.json'

I manually created empty output files in the already created output folder, moved the emails back to the Inbox and ran it again, and this time it completed without error. So there is a workaround, but this wasn't required when I was using IMAP access.

Thanks again.