localWebServer: request not found

[ArthurBorsboom]

I am trying to setup a Microsoft 365 mail account remotely by SSH on a Linux server.

Whichever configuration I try, mailctl keeps ending up in a browser window stating "request not found".

Roughly these are the steps I am doing.

Allow incoming traffic through the firewall (iptables): TCP/8080 Edit /etc/hosts file making sure localhost refers to the public IP, so the webserver of mailctl attaches to the public IP. Start mailctl in a SSH window.

Start desktop browser and enter the URL: http://monitor.xxxxxxxx.com:8080/start A Microsoft password form shows in the browser.

I enter the password followed by [Sign in].

The browser gets redirected to: http://monitor.xxxxxxxx.com:8080/common/login

and the following response appears in the browser window.

localWebServer: request not found ... Request {requestMethod = "POST", httpVersion = HTTP/1.1, rawPathInfo = "/common/login", rawQueryString = "", requestHeaders = [("Host","monitor.xxxxxxxx.com:8080"),("User-Agent","Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0"),("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"),("Accept-Language","en-US,en;q=0.5"),("Accept-Encoding","gzip, deflate"),("Referer","http://monitor.xxxxxxxx.com:8080/"),("Content-Type","application/x-www-form-urlencoded"),("Content-Length","1330"),("Origin","http://monitor.xxxxxxxx.com:8080"),("DNT","1"),("Connection","keep-alive"),("Upgrade-Insecure-Requests","1")], isSecure = False, remoteHost = 86.70.118.233:47858, pathInfo = ["common","login"], queryString = [], requestBody = , vault = , requestBodyLength = KnownLength 1330, requestHeaderHost = Just "monitor.xxxxxxxx.com:8080", requestHeaderRange = Nothing}

services.yaml

microsoft:
  auth_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
  # auth_endpoint: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/authorize
  auth_http_method: GET
  auth_params_mode: query-string
  token_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
  # token_endpoint: https://login.microsoftonline.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/oauth2/v2.0/token
  token_http_method: POST
  token_params_mode: request-body-form
  redirect_uri: http://localhost:8080
  # redirect_uri: http://monitor.xxxxxxxx.com
  # redirect_uri: http://localhost:8080/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/login
  auth_scope: https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access
  # auth_scope: offline_access https://outlook.office.com/Mail.ReadWrite
  client_id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  client_secret: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

I have tried several configuration options, but the result stays the same.

Do you have any suggestions how to approach this issue?

[pdobsan]

mailctl and the browser are supposed to run on the same machine, the communication is in clear text so you don't really want your credentials sent this way.

Do you have any suggestions how to approach this issue?

Here are some:

• Try to run the whole process of authorization on a Linux machine with the browser running on it and see if that works all the way. This should work with the recent version since other people succeeded with Microsoft 365 account.

• Use the same service config as above but now on your remote machine (without the localhost editing trick). Use a curses based browser on the remote, for example w3m, and see how far you can get. Probably, not too far because of the missing javascript stuff. Nevertheless it could be interesting to see what happens.

• Instead of the editing /etc/localhost, tunnel back the remote localhost:8080 to your local machine over ssh.

[ArthurBorsboom]

Option 3 - tunneling back without editing localhost, I have tried before to no extend. Option 2 - Using a text based browser on the server (lynx/w3m), I have tried but indeed the process stops fast due to Javascript needs. Option 1 - Try on a desktop machine first; I will try this one, since it might give new information.

Thinking about the problem a bit longer combined with the comment of Daniel (about a webpage that should have gone towards MS instead of the local webserver), I am suspecting the multi-factor authentication.

After entering the password I would have expected to see a MFA challenge (Microsoft / Google Authenticator) code. I know the MS 365 account used has MFA.

Anyway, I will first try local on my desktop to see if I can replicate this behavior and I will get back to here.

[ArthurBorsboom]

On my desktop it seems to be the same behavior.

Are there any success stories about MS365 accounts with MFA (to test my theory) ?

[ArthurBorsboom]

This is the used services.yaml (as default as it gets)

microsoft:
  auth_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
  auth_http_method: GET
  auth_params_mode: query-string
  token_endpoint: https://login.microsoftonline.com/common/oauth2/v2.0/token
  token_http_method: POST
  token_params_mode: request-body-form
  redirect_uri: http://localhost:8080
  auth_scope: https://outlook.office365.com/IMAP.AccessAsUser.All https://outlook.office365.com/SMTP.Send offline_access
  client_id: removed
  client_secret: removed

[pdobsan]

Are there any success stories about MS365 accounts with MFA (to test my theory) ?

I am not aware of any. Look at the thread of the (now merged and closed) pull request. You might want to ask its author by referring him in this thread.

Interestingly Daniel's comment disappeared (?).

[ArthurBorsboom]

Interestingly Daniel's comment disappeared (?).

Same thought here.

Look at the thread of the (now merged and closed) pull request. You might want to ask its author by referring him in this thread.

I will have a look at the pull request mentioned.

In the mean time, do you have other suggestions to try ?

[ghost]

I also have the same problem, MFA is not enabled on this account. It looks to me like

token_http_method: POST
token_params_mode: request-body-form

Could be the problem here or related to it, since this is one thing that changed

[ArthurBorsboom]

I have been talking to Phil, a user who has successfully authenticated against Microsoft 365, see issue #11. From this conversation I have concluded the following.

• Our MS365 accounts share several items like MFA and being an organization account
• Therefore, I assume MFA is unlikely to be the cause
• Therefore, I assume an organizational account is unlikely to be the cause.

I guess the focus must be somewhere else.

@pdobsan Since 3 users report the same behavior, it might be worth to debug a bit further.

Do you have suggestions to find clues of the cause?

For example, where should the call .../common/login being send to? To the local webserver or to Microsoft? Is there any way to analyze/debug this?

[pdobsan]

On Wed, Oct 12, 2022 at 01:37:12AM -0700, Arthur Borsboom wrote:

• Therefore, I assume MFA is unlikely to be the cause
• Therefore, I assume an organizational account is unlikely to be the cause.

Agreed. If it worked for @philcowans it should also work for you. If it doesn't, figure out what you are making differently.

I guess the focus must be somewhere else.

@pdobsan Since 3 users report the same behavior, it might be worth to debug a bit further.

Which 3 users are you referring to?

Anyway, instead of "same behavior" "seemingly similar" would be more correct. A good, usable description of the problem should include, mailctl's version and installation method, the config files, the operating system, browser, the exact type of the email account with its configuration (like MFA). A precise description of the process which produced the problem. Including click/selection history, preferably with logs/errors in text format and screen shots only if the text is not possible or does not make sense.

"I have the same problem" is not really helpful.

For example, where should the call .../common/login being send to? To the local webserver or to Microsoft?

Well definitely not to mailctl's web server, why MS sends it there I have no clue.

Here is the process:

Your redirect_uri should be http://localhost:8080/

• you go to http://localhost:8080/start using a browser running on the same machine as mailctl running on.

• mailctl redirects you to whatever you configured as token_endpoint sending the required parameters along with redirection request. The method used and the parameters are determined by your config and the cli arguments given to mailctl authorize ...

• from that point mailctl does not do anything just wait at http://localhost:8080/. Everything happens on MS side, including MFA etc. Whatever you do there might affect the outcome, but it is not in mailctl's control.

• Finally MS is supposed to send the requested credentials to http://localhost:8080/. That is what you configured as redirect_uri and sent over with the initial request to MS. This redirection happens locally in your browser since MS cannot see this uri from the outside.

• If for whatever reason MS redirects something to a different uri like .../common/login mailctl refuses the request since it is not prepared to deal with it and you see the ugly but informational error message in the browser.

Hope that helps with further debugging.

[aragaer]

I have the same issue as above -- but no MFA. I suppose I do have some kind of SSO though.

Looking at the MS login page source I see "urlPost":"/common/login" somewhere in the JS code. I believe pressing the "Sign in" button simply submits all the form data to that address which becomes the "http://localhost:8080/common/login"

[ArthurBorsboom]

Which 3 users are you referring to?

Daniel Nielson who's comment disappeared copycc (above) Arthur Borsboom (me)

Steps to reproduce

Install and setup Arch Linux for normal desktop usage (browser, etc)

yay -S mailctl-bin (install package) mkdir -p /home/user/.config/mailctl mkdir -p /home/user/.local/var/mailctl cp /usr/share/mailctl/configs/config-template.yaml /home/user/.config/mailctl/config.yaml cp /usr/share/mailctl/configs/services-template.yaml /home/user/.config/mailctl/services.yaml

nano /home/user/.config/mailctl/config.yaml

• configure services_file
• configure oauth2_dir

nano /home/user/.config/mailctl/services.yaml

• configure Microsoft ClientId
• configure Microsoft ClientSecret

mailctl authorize microsoft yourmail@organization.com

Visit the authorizatoin URL in the browser: http://localhost:8080/start

Enter password in Microsoft form Click [Sign In]

[ArthurBorsboom]

Maybe it is browser related?

I have tried both Firefox and Chromium on Linux. You guys?

[ghost]

I found out that when you sign in normally to Outlook, once you get to the /common/login screen, you are already authenticated. It only asks you whether you would like to stay signed in or not on that screen. So there should be some potential to sidestep the issue? The thing we're looking for should be found somewhere in the --debug output, though I don't really know where to go from there.

[ArthurBorsboom]

@copycc nice finding.

From my side, I have tried to access the webpage from Windows 10 with MS Edge and it did not change anything.

[ArthurBorsboom]

I also noticed that it doesn't matter if the password entered is correct or not. Entering a wrong password results in the same redirect to /common/login

[ghost]

After following these steps https://github.com/pdobsan/mailctl/pull/11#issuecomment-1276117027 and putting the correct redirect URI in the Application, I got a callback from Microsoft, resulting in this error Message: Application '<My app> is not configured as a multi-tenant application. Usage of the /common endpoint is not supported for such applications created after '10/15/2018'. Use a tenant-specific endpoint or configure the application to be multi-tenant.

[ghost]

Update: Configured app as Multi-tenant and it now works as intended. Hurray!

[danielneilson]

I deleted my earlier comment because it seemed I had drawn conclusions too hastily about what are and aren't the same issue. Even though I was getting the same error message as @ArthurBorsboom, I don't know oauth2 well enough to be sure of the connection.

At the moment I do have a working mailctl config for an institutional microsoft account. I am using the microsoft template from the configs dir of release 0.7.3. I am using Thunderbird oauth2 credentials.

To complete the authorization step, I had to follow ferdinandyb's advice from issue #5. Namely, I started the authorization step with

mailctl authorize microsoft X

where X is something that microsoft will not recognize as an account name. On the same machine, I open http://localhost:8080/start (in opera, which is chrome under the hood). I click through microsoft's SSO to enter my institution name and correct account name (i.e. email address). Then I can complete the authorization. I renamed the resulting token in $HOME/.local/var/mailctl/ to be the email address. I can now get expected behavior with mailctl access $EMAIL etc.

It seems that between now and a possible future v1.0 of mailctl, the README should be updated to reflect what users have figured out about getting through the microsoft auth process. For now the wisdom is here in the issues threads. I'm happy to help with that, but I was trying to fix my little problem and don't necessarily have the big picture. Also the renaming step is kludgy and with time there may be a more straightforward solution.

Thanks @pdobsan for your work on mailctl, which at least for the moment, and at least for me, is doing what I need it to do. I want to give it a few weeks of everyday use.

[aragaer]

Using "X" instead of an actual address did work.

[ArthurBorsboom]

I guess the stars have been aligned today, since it seems all the users here have been able to fix the issue somehow, today! :) After configuring gpg and setting up a https redirect to the localhost (since MS only accepts HTTPS redirect URI's other than localhost), it worked here as well.

Thanks everybody for thinking and helping along. Closing this ticket.

[pdobsan]

It seems that between now and a possible future v1.0 of mailctl, the README should be updated to reflect what users have figured out about getting through the microsoft auth process. For now the wisdom is here in the issues threads. I'm happy to help with that,

Yes please, see issue #13 !

To complete the authorization step, I had to follow https://github.com/pdobsan/mailctl/issues/5#issuecomment-1209008661. ... Also the renaming step is kludgy and with time there may be a more straightforward solution.

This "false email trick" is definitely something to be considered implementing programmatically. I have opened an issue #14 to investigate.