search

IndySockets / Indy

Indy - Internet Direct
https://www.indyproject.org
455 stars 155 forks source link

Add support for XOAUTH2 SASL authentication #192

Closed rlebeau closed 2 weeks ago

rlebeau commented 6 years ago

Outlook/Hotmail/Live, Gmail, and possibly others, support XOAUTH2 authentication over SASL for POP3, SMTP and IMAP. Indy should implement a TIdSASL component to support this. This way, users do not need to create application-specific passwords when 2-step verification is enabled in their accounts.

https://developers.google.com/gmail/imap/xoauth2-protocol

https://msdn.microsoft.com/en-us/library/dn440163.aspx

geoffsmith82 commented 4 years ago

Hi, I have a demo of this at https://github.com/geoffsmith82/GmailAuthSMTP

BretBordwell commented 2 years ago

Remy,

Now that Microsoft has removed all but OAuth2 authentication for IMAP, this is now on our front burner. Any resolution in sight?

rlebeau commented 2 years ago

I do have some new TIdSASL classes in the works for OAUTHBEARER, OAUTH10A and XOAUTH2 that I started awhile back ago, but they have not been finished or tested yet (for instance, response handling is not implemented yet), so they are not ready for release. The SASL classes linked above are similar to what I have, but a bit less fleshed out than what I have. But you can try them in the meantime and see if they work.

rlebeau commented 2 years ago

I have now pushed a new sasl-oauth branch into Indy's repo. It includes a new IdSASLOAuth.pas unit, and updates various client components to include their Port number when authenticating using OAuth.

BretBordwell commented 2 years ago

Thanks Remy,

The above sample by geoffsmith82 works. Confusing for me, but I've been able to implement it into our app. Now debugging.

Just wanted to say thanks.

For anyone that needs help, the following Google link pointed me in the right direction: https://developers.google.com/identity/protocols/oauth2

rlebeau commented 2 years ago

The above sample by geoffsmith82 works

OK, but do things also work when using the new SASL classes I checked in, rather than using geoff's SASL classes?

BretBordwell commented 2 years ago

OK, but do things also work when using the new SASL classes I checked in, rather than using geoff's SASL classes?

Thank you Remy. I gave it a fair shot yesterday, but had compile issues. Differences in procedure calls, declaration of uses in implementation when it belonged in interface (IdGlobals for example), etc. I'm using Alexandria (280), with the project files available for 270, etc (eg:IndyCore270). I'm fairly experienced, but just didn't have the time to work out all the differences and get past this learning curve. Some of this could be path issues, where this branch was trying to load default dcu and dcp packages as released with Delphi. All on me and my inexperience.

I would have preferred a subclass for now vs the implementation in the base classes and this is where geoffsmith82's implementation works. Sorry, I can not say if it works or not. I may give it another shot soon. Thanks again, really.

rlebeau commented 2 years ago

I gave it a fair shot yesterday, but had compile issues. Differences in procedure calls, declaration of uses in implementation when it belonged in interface (IdGlobals for example), etc.

Can you elaborate on the errors?

I'm using Alexandria (280), with the project files available for 270, etc (eg:IndyCore270)

FYI, package files for Alexandria (280) have now been checked in.

I would have preferred a subclass for now vs the implementation in the base classes and this is where geoffsmith82's implementation works.

What do you mean? The new SASL classes are subclasses.

LongDelphiHalfLife commented 2 years ago

Hi Remy I have your branch implemented in my app but it doesnt have any code to generate the token. In TryStartAuthenticate you GetPassword or call the GetAccessToken event. Im considering recreating Geoffs TEnhancedOAuth2Authenticator as a TIdUserPassProvider to return the token via GetPassword, or leaving that blank and using the GetAccessToekn event.

What do you recommend? Have a missed the token generator in your code?

function TIdSASLOAuth2Base.TryStartAuthenticate(const AHost: string; const APort: TIdPort; const AProtocolName : string; var VInitialResponse: String): Boolean; var LToken: String; begin LToken := GetPassword; if (LToken = '') and Assigned(FOnGetAccessToken) then begin FOnGetAccessToken(Self, LToken); end; VInitialResponse := DoStartAuthenticate(AHost, APort, LToken); Result := True; end;

rlebeau commented 2 years ago

I have your branch implemented in my app but it doesnt have any code to generate the token.

Correct, it does not. The user is responsible for obtaining the necessary token first, such as via HTTP to whatever OAuth provider they are working with (Google, Microsoft, etc), and then assign the token to the SASL component.

Have a missed the token generator in your code?

No. Outside of submitting the OAuth token over SASL, I do not have any other OAuth code implemented at this time.

LongDelphiHalfLife commented 2 years ago

Thanks Remy, IIm getting my own tokens now and passing them in with OnGetAccessToken. Geoffs class above was useful to get started but didn't cover the OAuth requirements i had.

hairy77 commented 2 years ago

Hi Remy I have your branch implemented in my app but it doesnt have any code to generate the token. In TryStartAuthenticate you GetPassword or call the GetAccessToken event. Im considering recreating Geoffs TEnhancedOAuth2Authenticator as a TIdUserPassProvider to return the token via GetPassword, or leaving that blank and using the GetAccessToekn event.

What do you recommend? Have a missed the token generator in your code?

function TIdSASLOAuth2Base.TryStartAuthenticate(const AHost: string; const APort: TIdPort; const AProtocolName : string; var VInitialResponse: String): Boolean; var LToken: String; begin LToken := GetPassword; if (LToken = '') and Assigned(FOnGetAccessToken) then begin FOnGetAccessToken(Self, LToken); end; VInitialResponse := DoStartAuthenticate(AHost, APort, LToken); Result := True; end;

FOnGetAccessToken

Hi

Thanks for this. I have downloaded Remy's branch, and can see the IdSASLOAuth.pas unit installed. However there is no component installed for TIdSASLOAuth2Base component in my component suite. (I completely uninstalled Indy from Delphi and deleted all references before installing this so I'm fairly sureI have the latest install).

It this not an actual design time component? Can you please advise if I need to try and use this another way? And are you able to confirm please if you actually were able to get POP3 working with Microsoft oAuth?

Thanks

Adam

rlebeau commented 2 years ago

Thanks for this. I have downloaded Remy's branch, and can see the IdSASLOAuth.pas unit installed. However there is no component installed for TIdSASLOAuth2Base component in my component suite. (I completely uninstalled Indy from Delphi and deleted all references before installing this so I'm fairly sure I have the latest install).

I hadn't yet updated the IdRegister.pas file in the Lib/Protocols folder to register the new SASL components in the IDE palette. I have now updated that file, so you should be able to pull the latest branch, recompile Indy, and see the new components. Although, I still need to update the DCR file to add palette icons for them.

FYI, TIdSASLOAuth2Base is just a base class, it is not meant to be used directly, so you will not see it on the palette. Use the derived classes instead: TIdSASLOAuth2Bearer, TIdSASLOAuth10A, and TIdSASLXOAuth2.

It this not an actual design time component?

It is now.

Can you please advise if I need to try and use this another way?

You can alternatively create the components in code at runtime, like any other component.

And are you able to confirm please if you actually were able to get POP3 working with Microsoft oAuth?

At this time, I haven't tested any of this new code myself.

hairy77 commented 2 years ago

Thanks Remy,

I've updated to the latest branch and indeed have access to the 3 new components. I have added these to a test project and added them to the POP3 SASLMechanisms, but when testing POP3 on Office365 still get the error "Doesn't support AUTH or the specified SASL handlers".

I'm assuming there's still work to be done before Indy is compatible with Microsoft's changes?

LongDelphiHalfLife commented 2 years ago

I've updated to the latest branch and indeed have access to the 3 new components. I have added these to a test project and added them to the POP3 SASLMechanisms, but when testing POP3 on Office365 still get the error "Doesn't support AUTH or the specified SASL handlers".

I have it working for MS IMAP and POP. If you see above I had to write a class to get an OAuth token from the MS service then pass it into the Indy component which uses that token to authorize the IMAP or POP connection. Works for google too but you need a class that can get a token from the google service (similar but just different parameters passed in)

rlebeau commented 2 years ago

I've updated to the latest branch and indeed have access to the 3 new components. I have added these to a test project and added them to the POP3 SASLMechanisms, but when testing POP3 on Office365 still get the error "Doesn't support AUTH or the specified SASL handlers".

That error message means that the POP3 server's CAPA response did not include a SASL line specifying any of the SASL components in the SASLMechanisms.

I'm assuming there's still work to be done before Indy is compatible with Microsoft's changes?

Microsoft uses the XOAUTH2 SASL, which is covered by the TIdSASLXOAuth2 component. All you need to do is assign the desired username to the Username property, and either assign the access token to the Password property or return the token from an OnGetAccessToken event handler.

Unfortunately, Microsoft's SASL documentation only demonstrates IMAP querying the server for OAuth2 support via a CAPABILITY command, it does not demonstrate a similar query for POP3 or SMTP, or maybe Microsoft didn't implement that yet? I don't know.

hairy77 commented 2 years ago

"I've updated to the latest branch and indeed have access to the 3 new components. I have added these to a test project and added them to the POP3 SASLMechanisms, but when testing POP3 on Office365 still get the error "Doesn't support AUTH or the specified SASL handlers"."

I have it working for MS IMAP and POP. If you see above I had to write a class to get an OAuth token from the MS service then pass it into the Indy component which uses that token to authorize the IMAP or POP connection. Works for google too but you need a class that can get a token from the google service (similar but just different parameters passed in)

Thanks for your replies LongDelphiHalfLife and Remy

I must be doing something wrong as I'm trying to follow what LongDelphiHalfLife has done, but the TidSASLXOAuth2's OnGetAccessToken event never fires. I get a "Doesn't support AUTH or the specified SASL Handlers!!" error raised when I perform the POP3's Connect command.

I've double check I have assigned the TidSASLXOAuth2 component to the SASLMechanisms (moved it from Available to Assigned) so I'm at a loss why it's not executing the onGetToken event or why it doesn't believe that the component isn't supported.

It's very encouraging to hear that LDHL has this actually working with Microsoft for POP3 - gives me hope that there's a solution with Indy that I may be able to implement before the Microsoft deadline - I just don't know why it's failing for me.

rlebeau commented 2 years ago

the TidSASLXOAuth2's OnGetAccessToken event never fires. I get a "Doesn't support AUTH or the specified SASL Handlers!!" error raised when I perform the POP3's Connect command.

As I explained in my previous reply, that error means that TIdPOP3 is not able to discover XOAuth2 listed in Microsoft's reply to the CAPA command when TIdPOP3 is trying to login, even though XOAuth2 is really supported by the server.

When logging in to a server, TIdPOP3 sends a CAPA command to discover which authentications the server supports, and then if TIdPOP3.AuthType is patSASL then TIdPOP3.Login() matches that list against TIdPOP3.SASLMechanisms to find a common set of authentications that both parties support, and then it attempts each of those in order until one succeeds or they all fail.

Please check this for yourself. You can either:

The CAPA response should look something like this:

C: CAPA

S: +OK
TOP
UIDL
SASL PLAIN XOAUTH2 // <-- HERE
USER
.

Are you seeing that entry when logging in to your Microsoft server?

I've ... moved it from Available to Assigned

What does that mean?

so I'm at a loss why it's not executing the onGetToken event

Because it is not attempting to login with XOAuth2.

or why it doesn't believe that the component isn't supported.

Because it thinks Microsoft doesn't support XOAuth2.

It's very encouraging to hear that LDHL has this actually working with Microsoft for POP3

I haven't looked at LDHL's implementation, but I suspect it is simply ignoring the CAPA response and attempting XOAuth2 unconditionally. TIdPOP3 does not do that.

hairy77 commented 2 years ago

Good Morning Remy,

Thanks for your reply (and your patience!). I'm not fully familiar with CAPA/XOAuth2 protocols and am undergoing a steep learning curve with this one. I really appreciate your assistance! I think I understand a lot more now.

I've ... moved it from Available to Assigned

What does that mean?

Sorry - what I mean by this is when I click on the idPop3 component in the designer, and open up SASLMechanism my idSASLXOauth2 component is in the Available list (left hand side). I move this to the Assigned side to confirm that it's actually being assigned to the POP3 component.

Please check this for yourself. You can either:

  • look at the contents of the TIdPOP3.Capabilities property after calling TIdPOP3.Connect() (to avoid TIdPOP3.Connect() raising the exception, you can set TIdPOP3.AutoLogin to false, and then call TIdPOP3.Login() afterwards)
  • assign a TIdLog... component to the TIdPOP3.Intercept property to capture the raw POP3 commands/responses.

The CAPA response should look something like this:

C: CAPA

S: +OK
TOP
UIDL
SASL PLAIN XOAUTH2 // <-- HERE
USER
.

Are you seeing that entry when logging in to your Microsoft server?

Thanks very much for the details. Unfortunately no - I'm not seeing that when connecting to the MS Server.

What I have in my debug log is as follows:

S:CAPA
R:+OK
TOP
UIDL
STLS
.
S:STLS
R:+OK Begin TLS negotiation.

...and then I get the Doesn't support AUTH or specified SASL handlers. So it looks like the response is definitely missing the XOAUTH2 that Indy is expecting.

I have checked out the Capabilities property after calling connect as you suggested. I get TOP, UIDL and STLS only.

I thought I'd try and be sneaky and do a force, so I went and added:

idPOP3.Connect;
idPOP3.CAPA;
idPop3.Capabilities.add('SASL PLAIN XOAUTH2'); // Trying to be sneaky here...
idPOP3.Login;

but then in the logs I get:

S:AUTH XOAUTH2 R:-ERR Protocol error. Connection is closed. 10

I'm guessing from this it seems that my approach of manually adding a capability that wasn't returned was successful to force XOAuth2- but then I get the protocol error occurs immediately after that as Microsoft rejects my S:AUTH XOAUTH2 request.

rlebeau commented 2 years ago

I've ... moved it from Available to Assigned

What does that mean?

Sorry - what I mean by this is when I click on the idPop3 component in the designer, and open up SASLMechanism my idSASLXOauth2 component is in the Available list (left hand side). I move this to the Assigned side to confirm that it's actually being assigned to the POP3 component.

Oh, OK. I wasn't aware that there was a custom design-time Form being used to edit the SASLMechanisms, I thought only the the Object Inspector's standard TCollection editor was being used. It hass been a long time since I last dealt with the SASL components at design-time.

Are you seeing that entry when logging in to your Microsoft server?

Unfortunately no - I'm not seeing that when connecting to the MS Server.

Well, then that is why you are getting the error raised.

What I have in my debug log is as follows:

Interesting, I would have expected TIdPOP3 to send another CAPA command after STLS has finished securing the connection, as a server's capabilities may change once the connection has been secured. But looking at TIdPOP3's code, it only sends CAPA one time, in TIdPOP3.Connect() before login. I have now fixed that (https://github.com/IndySockets/Indy/issues/427). I'll bet the XOAUTH2 capability will now show up properly after a successful STLS.

I thought I'd try and be sneaky and do a force, so I went and added:

...

but then in the logs I get:

S:AUTH XOAUTH2
R:-ERR Protocol error. Connection is closed. 10

Odd, considering that is following Microsoft's example of sending AUTH XOAUTH2 without any parameters, waiting for the server to acknowledge the request before then sending the encoded token. I wonder why it think there is a protocol error.

Well, try the latest branch (you can remove your hack), and see if the same error still occurs.

hairy77 commented 2 years ago

I have now fixed that (#427). I'll bet the XOAUTH2 capability will now show up properly after a successful STLS. Odd, considering that is following Microsoft's example of sending AUTH XOAUTH2 without any parameters, waiting for the server to acknowledge the request before then sending the encoded token. I wonder why it think there is a protocol error.

Well, try the latest branch (you can remove your hack), and see if the same error still occurs.

Thanks Remy. You are correct! I've updated and tried that and on the 2nd CAPA request it shows up properly, but it is still unsuccessful.

I think I've found the problem! (Just not sure how to fix it).

I manually tried logging in with OpenSSL typing in the commands myself. If I try and replicate what Indy is doing I get the same error.

If on the other hand I execute AUTH XOAUTH2 as a command, wait for a response, and then paste in the token and send that through as a separate command - it executes successfully.

I notice in IdSASLCollection on line 235 - this is where the problem occurs:

AClient.SendCmd(ACmd + ' ' + String(ASASL.ServiceName) + ' ' + AEncoder.Encode(S), []);//[334, 504]);

Indy is sending through ...

AUTH XOAUTH2 <token>

... as a single transmission and Microsoft doesn't appear to like it.

However it would appear if AUTH XOAUTH2 is sent through, wait for a response and then the Token is sent through after - Microsoft is happy.

I'm not sure if it's of any help, but I've come up with the following code to create things at runtime (to try and make it easier for discussion and to see what I'm actually doing instead of having components on a form at design time):

procedure TForm2.IdSASLXOAuth21GetAccessToken(Sender: TObject; var AccessToken: string);
begin
  AccessToken := EmailOAuthDataModule.FOAuth2_Enhanced.AccessToken;
end;

procedure TForm2.btnCheckMsg2Click(Sender: TObject);
var
  IDPop3: TidPop3;
  xoauthSASL: TIdSASLListEntry;
  msgCount: Integer;
  SASLList: TIdSASLListEntry;
begin
  IdPop3 := TidPop3.create;
  IdPop3.AutoLogin := false;
  IdPOP3.IOHandler := TidSSLioHandlerSocketOpenSSL.create;
  xoauthSASL := IdPOP3.SASLMechanisms.Add;
  xoauthSASL.SASL := TIdSASLXOAuth2.Create(nil);
  TIdSASLXOAuth2(xoauthSASL.SASL).OnGetAccessToken := IdSASLXOAuth21GetAccessToken;
  TIdSASLXOAuth2(xoauthSASL.SASL).UserPassProvider := TIdUserPassProvider.Create();
  TIdSASLXOAuth2(xoauthSASL.SASL).UserPassProvider.Username := microsoft_clientaccount;

  IdPOP3.Host := 'outlook.office365.com';
  IdPOP3.Port := 995;
  IdPOP3.UseTLS := utUseExplicitTLS;

  IdPOP3.AuthType := patSASL;
  IdPOP3.Connect;
  IdPOP3.CAPA;
  IdPOP3.Login;
  msgCount := IdPOP3.CheckMessages;
end;

I hope this is of some help.

KrystianBigaj commented 2 years ago

Hi,

I can confirm that POP3, IMAP4, SMTP with MS OAUTH2 and Indy (sasl-oauth branch from about month ago, Delphi 10.4) works correctly. I had to make 2 fixes: IdReplyIMAP4.pas (add AssignTo, because in case of error during ouath authentication, error message is mssing), fix:

procedure TIdReplyIMAP4.AssignTo(ADest: TPersistent);
begin
  inherited AssignTo(ADest);

  // Extra.Assign must be called after inherited AssignTo (because of Clear)
  if ADest is TIdReplyIMAP4 then
    TIdReplyIMAP4(ADest).Extra.Assign(Extra);
end;

For POP3 connection I had to fix AUTH XOAUTH2 with token in new line (because I got "Protocol error" respone). IdSASLCollection.pas:

function PerformSASLLogin ....
...
  if ACanAttemptIR then begin
    if ASASL.TryStartAuthenticate(AHost, APort, AProtocolName, S) then begin
      { KB
      https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth
      To authenticate a POP server connection, the client will have to respond with an AUTH command split into two lines in the following format:
      }
      if TextIsSame(AProtocolName, IdGSKSSN_pop) then
      begin
        AClient.SendCmd(ACmd + ' ' + String(ASASL.ServiceName) {+ ' ' + AEncoder.Encode(S)}, []);//[334, 504]);
        AClient.SendCmd(AEncoder.Encode(S), []);
      end else
        AClient.SendCmd(ACmd + ' ' + String(ASASL.ServiceName) + ' ' + AEncoder.Encode(S), []);//[334, 504]);
      if CheckStrFail(AClient.LastCmdResult.Code, AOkReplies, AContinueReplies) then begin
        if not TextIsSame(AProtocolName, IdGSKSSN_pop) then begin
          ASASL.FinishAuthenticate;
          Exit; // this mechanism is not supported
        end;
      end else begin
        AuthStarted := True;
      end;
    end;
  end;

I'm not sure it this is required for other OAUTH2 providers.

hairy77 commented 2 years ago

Thanks so much for your confirmation and code snippet. I have now changed line 235 in IdSASLCollection.pas from:

AClient.SendCmd(ACmd + ' ' + String(ASASL.ServiceName) + ' ' + AEncoder.Encode(S), []);//[334, 504]);

to:

     if TextIsSame(AProtocolName, IdGSKSSN_pop) then
      begin
        AClient.SendCmd(ACmd + ' ' + String(ASASL.ServiceName) {+ ' ' + AEncoder.Encode(S)}, []);//[334, 504]);
        AClient.SendCmd(AEncoder.Encode(S), []);
      end else
      AClient.SendCmd(ACmd + ' ' + String(ASASL.ServiceName) + ' ' + AEncoder.Encode(S), []);//[334, 504]);

... and can confirm that I am successfully authenticating with Microsoft 365! Again - thank you!!!

geoffsmith82 commented 2 years ago

@KrystianBigaj I wouldn't make the change in PerformSASLLogin as the the 2 line login for Microsoft isn't used by everyone (specifically Google). I would add an property to the SASL class for a Two line POP authentication.

rlebeau commented 2 years ago

Most POP3 providers support the version of the AUTH command where the initial credentials are included in the 1st line of the request, so PerformSASLLogin() attempts that version first. This feature saves a round-trip. But unlike other protocols, that feature of AUTH is not advertised in POP3's CAPA reply, as it is assumed to be supported since it was introduced in the same RFC 2449 that introduced the CAPA command itself. But, it wasn't formalized until RFC 5034, so not all POP3 providers actually implement that feature of AUTH. So, if it fails the 1st time, then for POP3 only, PerformSASLLogin() falls back to the old logic of sending an AUTH request without sending the initial credentials until the server explicitly asks for them.

THIS IS BY DESIGN. You should NOT have had to make ANY code changes to TIdPOP3 or PerformSASLLogin() for this to work. So please, revert all of those changes.

If you really want to disable the 1st 1-line AUTH attempt for POP3, the correct way to do that is to have TIdPOP3.Login() set ACanAttemptIR=False when it calls LoginSASL(). TIdPOP3 used to do exactly that, until a year ago when the retry logic was implemented in PerformSASL() in PR #354.

I have now added a SASLCanAttemptInitialResponse property to TIdPOP3 in the sasl-oauth branch to control whether ACanAttemptIR is set to True or False on a per-login basis. Similar to the ValidateAuthLoginCapability property in TIdSMTP when AuthType=satDefault.

As for TIdReplyIMAP, I have checked in a fix for it in the main code.

hairy77 commented 2 years ago

Hi Remy,

Thanks for your reply, and thanks for adding in SASLCanAttemptInitialResponse. Just to clarify...

THIS IS BY DESIGN. You should NOT have had to make ANY code changes to TIdPOP3 or PerformSASLLogin() for this to work. So please, revert all of those changes. TIdPOP3 used to do exactly that, until a year ago when the retry logic was implemented in PerformSASL() in PR #354.

I'm confused by this. With the example code shown above on how I connect - I was definitely getting errors with Microsoft until I made code changes Krystian mentioned to the code - however if I understand you correctly - I shoudn't have needed to make that change because the retry logic was implemented in PerformSASL(). So now I'm confused as to why it is failing for me with Microsoft as it certainly didn't seem to be retrying using the second method after failure of the first.

I will revert changes and give SASLCanAttemptInitialResponse a go- but I'm still curious to know that what I'm experiencing (failure to authenticate) does not match up with what your saying (retry logic was implemented and there should be no need to make any changes to the code) - that I shouldn't need SASLCanAttemptInitialResponse at all?

rlebeau commented 2 years ago

I'm confused by this. With the example code shown above on how I connect - I was definitely getting errors with Microsoft until I made code changes Krystian mentioned to the code - however if I understand you correctly - I shoudn't have needed to make that change because the retry logic was implemented in PerformSASL().

Correct. You would have gotten an initial error, but it should have retried automatically, and your code would not see any error unless the retry also failed.

So now I'm confused as to why it is failing for me with Microsoft as it certainly didn't seem to be retrying using the second method after failure of the first.

I would have to see the complete POP3 log to see what is really happening. Under the original code, it should have sent AUTH XOAUTH2 <base64> first, and if that failed then it should have retried by sending AUTH XOAUTH2 followed by just <base64> after receiving + from the server. So, either the retry didn't happen, or the server didn't send +? That is why I would need to see a log.

I will revert changes and give SASLCanAttemptInitialResponse a go- but I'm still curious to know that what I'm experiencing (failure to authenticate) does not match up with what your saying (retry logic was implemented and there should be no need to make any changes to the code) - that I shouldn't need SASLCanAttemptInitialResponse at all?

Ideally, you should not need SASLCanAttemptInitialResponse at all, but since Microsoft is causing problems, you are just going to have to set SASLCanAttemptInitialResponse=False when connecting to Office 365 via POP3 (IMAP and SMTP should be fine) until this can be sorted out.

hairy77 commented 2 years ago

deally, you should not need SASLCanAttemptInitialResponse at all, but since Microsoft is causing problems, you are just going to have to set SASLCanAttemptInitialResponse=False when connecting to Office 365 via POP3 (IMAP and SMTP should be fine) until this can be sorted out.

Hi Remy,

SASLCanAttemptInitialResponse=False works a charm! Thank you so much for all your work in this! This will get me out of trouble!

I would have to see the complete POP3 log to see what is really happening. Under the original code, it should have sent AUTH XOAUTH2 <base64> first, and if that failed then it should have retried by sending AUTH XOAUTH2 followed by just <base64> after receiving + from the server. So, either the retry didn't happen, or the server didn't send +? That is why I would need to see a log.

Hopefully this is of some help:

Stat Connected.
Recv 29/08/2022 12:23:05 PM: +OK The Microsoft Exchange POP3 service is ready. [TQBFxxxxxxxxxx]<EOL>
Sent 29/08/2022 12:23:05 PM: CAPA<EOL>
Recv 29/08/2022 12:23:05 PM: +OK<EOL>TOP<EOL>UIDL<EOL>STLS<EOL>.<EOL>
Sent 29/08/2022 12:23:05 PM: CAPA<EOL>
Recv 29/08/2022 12:23:05 PM: +OK<EOL>TOP<EOL>UIDL<EOL>STLS<EOL>.<EOL>
Sent 29/08/2022 12:23:05 PM: STLS<EOL>
Recv 29/08/2022 12:23:05 PM: +OK Begin TLS negotiation.<EOL>
Sent 29/08/2022 12:23:05 PM: CAPA<EOL>
Recv 29/08/2022 12:23:05 PM: +OK<EOL>TOP<EOL>UIDL<EOL>SASL PLAIN XOAUTH2<EOL>USER<EOL>.<EOL>
Sent 29/08/2022 12:23:05 PM: AUTH XOAUTH2 <xxxxxxx token redacted> <EOL>
Recv 29/08/2022 12:23:05 PM: -ERR Protocol error. Connection is closed. 10<EOL>
Sent 29/08/2022 12:23:05 PM: AUTH XOAUTH2<EOL>
Stat Disconnected.
KrystianBigaj commented 2 years ago

Thanks for SASLCanAttemptInitialResponse update, now works with MS POP3 without my workaround :) As I remember, I have previously got same error "Protocol error. Connection is closed"

KrystianBigaj commented 2 years ago

@rlebeau There is a warning after recent change: [dcc32 Warning] IdPOP3.pas(380): W1000 Symbol 'LoginSASL' is deprecated: 'Use overload with APort parameter' Previously LoginSASL was called with FPort parameter in IdPOP3

rlebeau commented 2 years ago 
Stat Connected.
Recv 29/08/2022 12:23:05 PM: +OK The Microsoft Exchange POP3 service is ready. [TQBFxxxxxxxxxx]<EOL>
Sent 29/08/2022 12:23:05 PM: CAPA<EOL>
Recv 29/08/2022 12:23:05 PM: +OK<EOL>TOP<EOL>UIDL<EOL>STLS<EOL>.<EOL>
Sent 29/08/2022 12:23:05 PM: CAPA<EOL>
Recv 29/08/2022 12:23:05 PM: +OK<EOL>TOP<EOL>UIDL<EOL>STLS<EOL>.<EOL>
Sent 29/08/2022 12:23:05 PM: STLS<EOL>
Recv 29/08/2022 12:23:05 PM: +OK Begin TLS negotiation.<EOL>
Sent 29/08/2022 12:23:05 PM: CAPA<EOL>
Recv 29/08/2022 12:23:05 PM: +OK<EOL>TOP<EOL>UIDL<EOL>SASL PLAIN XOAUTH2<EOL>USER<EOL>.<EOL>
Sent 29/08/2022 12:23:05 PM: AUTH XOAUTH2 <xxxxxxx token redacted> <EOL>
Recv 29/08/2022 12:23:05 PM: -ERR Protocol error. Connection is closed. 10<EOL>
Sent 29/08/2022 12:23:05 PM: AUTH XOAUTH2<EOL>
Stat Disconnected.

Why is CAPA being sent twice before STLS? TIdPOP3 sends CAPA only once after connecting, and then again after STLS. So where did the extra CAPA come from?

There is a warning after recent change: [dcc32 Warning] IdPOP3.pas(380): W1000 Symbol 'LoginSASL' is deprecated: 'Use overload with APort parameter' Previously LoginSASL was called with FPort parameter in IdPOP3

Fixed.

LongDelphiHalfLife commented 2 years ago

remy, you will see in Hairys code above he is calling CAPA himself hence the second call

rlebeau commented 2 years ago

Oh, OK, I didn't notice that earlier, thanks. That call is not needed, though, since Connect() already calls CAPA().

KrystianBigaj commented 2 years ago

As for TIdReplyIMAP, I have checked in a fix for it in the main code.

After that change, now I got error: "Reply Code is not valid: 0"

System._RaiseAtExcept(???,???)
System._RaiseExcept(???)
IdReply.TIdReply.SetCode('0')
IdReply.TIdReply.SetNumericCode(0)
IdReplyIMAP4.TIdReplyIMAP4.AssignTo(???)
System.Classes.TPersistent.Assign(???)
IdIMAP4.SetupErrorReply
IdIMAP4.TIdSASLEntriesIMAP4.LoginSASL_IMAP($2C2A0860)
IdIMAP4.TIdIMAP4.Login
IdIMAP4.TIdIMAP4.Connect(True)

Thanks.

rlebeau commented 2 years ago

After that change, now I got error: "Reply Code is not valid: 0"

Fixed

joostcrommert1 commented 2 years ago

@rlebeau do you perhaps have a test project with SASLMechanisms implemented on a idSMTP or IdIMAP-component? Currently seeking for a alternative to the soon-to-be legacy idSTMP with Basic Authentication mailing.

Thanks!

rlebeau commented 2 years ago

@rlebeau do you perhaps have a test project with SASLMechanisms implemented on a idSMTP or IdIMAP-component?

No, I do not, sorry.

marcin-bury commented 2 years ago

@KrystianBigaj , @hairy77 Guys, would you be so kind to share some working example how tu use TIdSASLXOAuth2 to connect to IMAP account?

TIA

rlebeau commented 2 years ago

I don't have an example to offer. But the use is fairly straight-forward. It works similar to most other SASL components. You need to:

marcin-bury commented 2 years ago

Remy @rlebeau Thanks a lot.

marcin-bury commented 1 year ago

@KrystianBigaj , @hairy77 How do you obtain the acces_token for reading emails. What endpoints do you use and what do you put in "scope"?

TIA Marcin

joostcrommert1 commented 1 year ago

@marcin-bury depends on what type of library (SMTP, Pop3, IMAP) you use in your Delphi-project. In my case i've used the TIdSMTP-component with the following scope; "https://outlook.office.com/IMAP.AccessAsUser.All https://outlook.office.com/POP.AccessAsUser.All https://outlook.office.com/SMTP.Send offline_access".

Used endpoints are https://login.microsoftonline.com/common/oauth2/v2.0/authorize and https://login.microsoftonline.com/common/oauth2/v2.0/token

marcin-bury commented 1 year ago

@joostcrommert1 thanks for the response. Would you share the a piece of code to get what is the correct sequence of calling the endpoints and what I should put in each request body? I need only IMAP - read the messages and move them to 'archive' folder.

TIA Marcin

joostcrommert1 commented 1 year ago

@marcin-bury I haven't used IMAP myself, but I guess the set-up would be the same as POP3;

Providers[ProviderInfo] is based on the example that @geoffsmith82 posted earlier in this thread.

DataModule.Mailbox: TIdPop3;

    xoauthSASL := DataModule.Mailbox.SASLMechanisms.Add;
    xoauthSASL.SASL := TIdSASLXOAuth.Create(nil);

    if xoauthSASL.SASL is TIdSASLXOAuth then
    begin
      TIdSASLXOAuth(xoauthSASL.SASL).Token := Self.GetAccessCode(FSelectedProvider);
      TIdSASLXOAuth(xoauthSASL.SASL).User := FUsername;
      TIdSASLXOAuth(xoauthSASL.SASL).TwoLinePOPFormat := True;
    end;

    DataModule.Mailbox.SASLCanAttemptInitialResponse := False;
    DataModule.Mailbox.AuthType := patSASL;

    DataModule.Mailbox.Connect;
    DataModule.Mailbox.Login;

GetAccessCode:

if ((Now() >= aAccessTokenValidUntil) and (aAccessTokenValidUntil <> 0)) then
  begin
    AccessToken := aAccessToken;
  end
  else
  begin
    if aRefreshToken <> '' then
    begin
      aParams := TStringList.Create;
      aParams.Add('grant_type=refresh_token');
      aParams.Add('client_id='+Providers[ProviderInfo].ClientID);
      aParams.Add('client_secret='+Providers[ProviderInfo].ClientSecret);
      aParams.Add('scope='+Providers[ProviderInfo].Scopes);
      aParams.Add('redirect_uri='+Providers[ProviderInfo].RedirectUrl);
      aParams.Add('refresh_token='+aRefreshToken);

      aResponseString := Self.Request(Providers[ProviderInfo].AccessTokenEndpoint, 'POST', aParams);

      aOauth := TOffice365OAuthClass.FromJsonString(aResponseString);

      aAccessTokenValidUntil := IncSecond(Now(), Round(aOauth.expires_in));

      AccessToken := aOauth.access_token;
    end;

    if AccessToken = '' then
    begin
      FCallBackURL := Providers[ProviderInfo].RedirectUrl;

      AuthURL := Providers[ProviderInfo].AuthorizationEndpoint + '?client_id=%s'
       + '&response_type=code'
       + '&redirect_uri=%s'
       + '&scope=%s';

      AuthURL := Format(AuthURL, [
        Providers[ProviderInfo].ClientID,
        Providers[ProviderInfo].RedirectUrl,
        Providers[ProviderInfo].Scopes
      ]);

      // Show (login) form
      aLoginForm := TFLogin.Create(nil);
      try
        Self.aLoginForm.InitForm(AuthURL, Providers[ProviderInfo].RedirectUrl, Self.processLogin());
      finally
        FreeAndNil(Self.aLoginForm);
      end;

      if FAuthCode = '' then
      begin
        Abort;
      end;

      aParams := TStringList.Create;
      aParams.Add('grant_type=authorization_code');
      aParams.Add('client_id='+Providers[ProviderInfo].ClientID);
      aParams.Add('client_secret='+Providers[ProviderInfo].ClientSecret);
      aParams.Add('scope='+Providers[ProviderInfo].Scopes);
      aParams.Add('redirect_uri='+Providers[ProviderInfo].RedirectUrl);
      aParams.Add('code='+FAuthCode);

      aResponseString := Self.Request(Providers[ProviderInfo].AccessTokenEndpoint, 'POST', aParams);

      aOauth := TOffice365OAuthClass.FromJsonString(aResponseString);

      aAccessTokenValidUntil := IncSecond(Now(), Round(aOauth.expires_in));

      AccessToken := aOauth.access_token;
    end;
  end;
rlebeau commented 1 year ago

@marcin-bury xoauthSASL.SASL := TIdSASLXOAuth.Create(nil);

Have you tried using the TIdSASLXOAuth2 class in this ticket's branch, instead of using a custom class?

if xoauthSASL.SASL is TIdSASLXOAuth then

Why are you using the is operator to check the type you just created? This will always be True, so just omit it.

 TIdSASLXOAuth(xoauthSASL.SASL).Token := Self.GetAccessCode(FSelectedProvider);
 TIdSASLXOAuth(xoauthSASL.SASL).TwoLinePOPFormat := True;

TIdSASLXOAuth2 does not have Token or TwoLinePOPFormat properties.

Also, what is the TwoLinePOPFormat property doing? Is there a problem with the new TIdPOP3.SASLCanAttemptInitialResponse property added in this ticket's branch?

marcin-bury commented 1 year ago

@rlebeau Remy The piece of code, you are refering to was presented by @joostcrommert1 as an example of "flow" to connect to Office365 mailbox.

marcin-bury commented 1 year ago

Btw, @joostcrommert1 , thanks for sharing

marcin-bury commented 1 year ago

Sorry guys for non english post @KrystianBigaj Podpowiedziałbyś jak właściwie pobrać token z login.microsoftonline.com, żeby zalogować się do skrzynki IMAP-owej Office365 (przez TidIMAP4). Mam tenant_Id, client_Id, client_secret, takie mam body przy wywołaniu: tsRequestBody.Add('grant_type=client_credentials'); tsRequestBody.Add('client_id=' + ClientID); tsRequestBody.Add('client_secret=' + ClientSecret); tsRequestBody.Add('scope=https://graph.microsoft.com/.default'); dostaję odpowiedź z access_token, ale już do IMAP zalogować się nie mogę, Dzięki Marcin

joostcrommert1 commented 1 year ago

@rlebeau code is messy indeed, was happy with a working example so didn't bother to clean up the code.

marcin-bury commented 1 year ago

@joostcrommert1 What "scopes" do you use for obtaining access_token - some standard or dedicated ones?