[FEATURE REQUEST] MS365 OAuth2 Mailbox-Import

[thomasklosinsky]

Describe the requested feature We have a few MS365 accounts with old emails which are not in our piler yet. We want to import those. Since MS365 does not support simple IMAP anymore, the import function would need to support OAuth2.

Maybe that's already supported, but we didn't find out how? If not yet supported, this is a feature request...

Piler 1.4.4

[Smig0l]

AFAIK we stumbled upon the same issue. we downloaded a .pst using the 365 compliance portal. (u can delimit by date ranges) and we imported it using readpst and pilerimport. see docs

[jsuto]

imapfetch.py supports --oauth2-token cli option that expects a file with the oauth2 access token inside. Can you test it?

[thomasklosinsky]

Thanks for the hints! the imapfetch.py is not integrated in the docker (compose) container, so I need to install it and dependencies manually, correct? Or use the host machine for that...

[jsuto]

I think imapfetch.py can run just fine in the container. At least it doesn't complain about any dependencies. However, you may run it on the host as well.

[thomasklosinsky]

OK, getting further with this:

copied a script for generating an (application) access token in the piler container, successfully generated a file containing the token, copied the imapfetch script to /opt/ in the container and tried to use it with the following variables:

python3 imapfetch.py -s outlook.office365.com -u [email-address] --oauth2-token /opt/access_t oken.txt -v -P 993 Skipped folder list: ['junk', 'trash', 'spam', 'draft', '"[Gmail]"'] Traceback (most recent call last): File "/opt/imapfetch.py", line 230, in main() File "/opt/imapfetch.py", line 198, in main conn.authenticate("XOAUTH2", lambda x: generate_auth_string( File "/usr/lib/python3.10/imaplib.py", line 444, in authenticate raise self.error(dat[-1].decode('utf-8', 'replace')) imaplib.IMAP4.error: AUTHENTICATE failed.

any ideas what I did wrong?

[thomasklosinsky]

of course the imap access rights are set...

[Smig0l]

copied a script for generating an (application) access token in the piler container, successfully generated a file containing the token

can u share the script that generates the access_token...depending on that u might need different API permission. if is using the client_credentials flow u need this permission:

[thomasklosinsky]

script is this:

  import requests
  import json

  # Deine MS365 Anwendungs-Registrierungsdaten
  client_id = 'DEINE_CLIENT_ID'
  client_secret = 'DEIN_CLIENT_SECRET'
  tenant_id = 'DEINE_TENANT_ID'

  # URL für den Tokenabruf
  token_url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token"

  # Die Daten für die POST-Anfrage
  data = {
    'grant_type': 'client_credentials',
    'client_id': client_id,
    'client_secret': client_secret,
    'scope': 'https://graph.microsoft.com/.default'
  }

  # Eine POST-Anfrage senden, um den AccessToken zu erhalten
  response = requests.post(token_url, data=data)
  response_json = response.json()

  # Überprüfen, ob der Token erfolgreich abgerufen wurde
  if 'access_token' in response_json:
    access_token = response_json['access_token']
    # Den AccessToken in eine Datei schreiben
    with open('access_token.txt', 'w') as token_file:
        token_file.write(access_token)
    print("AccessToken wurde erfolgreich abgerufen und gespeichert.")
  else:
    print("Fehler beim Abrufen des AccessTokens:", response_json)

So, maybe the scope is not corect in this script?

[Smig0l]

So, maybe the scope is not corect in this script?

yeah the scope should be https://outlook.office365.com/.default ..follow this MSFT guide. it is important to create a service principal and give him fullaccess permission to the mailbox u need. then in the clientid u should use the objectid inside enterprise app section.

[jsuto]

Be sure to include the working script, perhaps it's better to replace util/get-token.py with this one.

[thomasklosinsky]

Not really concerning this issue... but stuck at: New-ServicePrincipal -AppId [app-id] -ObjectId [object-id-of-app]

Write-ErrorMessage : |System.Exception|Fehler: AADServicePrincipalNotFound.

Will have a look into that tomorrow.

[thomasklosinsky]

Aight. got it working with my own get token script.

The Principal creation is not very appropriate and not well documented by MS. For everyone who is hitting the same problems:

I had to read out the ObjectID of the registered App in PowerShell, not using the one which is shown in the Azure Portal.

Get-AzureADServicePrincipal -Filter "appId eq 'ObejctID'"

Then get the Pricipal ID

Get-ServicePrincipal | fl

And with that ID I could modify the Mailbox-Permissions.

Afterwards I could run the imapfetch.py script successfully.

[jsuto]

I'm attempting to write a tool that can mass download emails from mailboxes, and I wonder if either of you could test it, and help me to fine tune it? My problem is that microsoft has suspended the enrollment for an o365 sandbox.

[Smig0l]

My problem is that microsoft has suspended the enrollment for an o365 sandbox.

I have a 365 developer tenant. I will gladly help you😁

[jsuto]

Awesome. I've just created a python script to do that, see https://github.com/jsuto/piler/pull/56/files The objective is to acquire all emails in all folders for the dev tenant. Also the downloaded message format is interesting, I'd like to see a few smaller files.

[jsuto]

Created a new issue #57 for this feature

[thomasklosinsky]

I am reopening, again with a MS365 concerning issue. The access token is running into timeout when importing a large mailbox. Already tried these steps but they are not working: https://learn.microsoft.com/en-us/entra/identity-platform/configure-token-lifetimes

Any ideas? Does the new script solve that token timeout issue?

[jsuto]

Did you see this PR? https://github.com/jsuto/piler/pull/56/files#diff-cd95da568fa8631fb163d3b1c5e9879fe0a5f6f43e89cf7192d97431ad5e975e

[thomasklosinsky]

Yes, I did. That's why I am asking. Is the token timeout problem solved with that script?

[jsuto]

I think so.