IMAP client Authenticate problems with OAuth2 using Google service account

[frank4040]

I am really having trouble getting through the 'Authenticate' step in MailKit. I need to use a Google Service Account since I read all mails in the background from a Windows Service without any user interface. I can already get a token, but the final 'Authenticate' step still fails. Tried about a thousand things, to no avail. In the Google Cloud console, I created a new custom Role with these permissions: iam.serviceAccounts.get iam.serviceAccounts.getAccessToken iam.serviceAccounts.getOpenIdToken iam.serviceAccounts.implicitDelegation iam.serviceAccounts.list iam.serviceAccounts.signBlob iam.serviceAccounts.signJwt (I selected about all that could even remotely be associated with tokens and service accounts)

Then I added this role to the Service Account, both the gserviceaccount.com as my own. I created a KEY and downloaded the JSON file. The content of that is in my JSONdata variabele. Like this (a bit scrambled here and there)

    private static String JSONdata = @"{
      ""type"": ""service_account"",
      ""project_id"": ""logical-xxxxxx"",
      ""private_key_id"": ""bce4e547b217835b2722c8b1c41b18b4c76b776b"",
      ""private_key"": ""-----BEGIN PRIVATE KEY-----\nMIIEvAIBADA..(more data)...rn0AP9dIkpSsrg==\n-----END PRIVATE KEY-----\n"",
      ""client_email"": ""name_of_serviceAccount@logical-yyyyy.iam.gserviceaccount.com"",
      ""client_id"": ""100517952578543651826"",
      ""auth_uri"": ""https://accounts.google.com/o/oauth2/auth"",
      ""token_uri"": ""https://oauth2.googleapis.com/token"",
      ""auth_provider_x509_cert_url"": ""https://www.googleapis.com/oauth2/v1/certs"",
      ""client_x509_cert_url"": ""https://www.googleapis.com/robot/v1/metadata/x509/name_of_serviceAccount@logical-yyyyy.iam.gserviceaccount.com"",
      ""universe_domain"": ""googleapis.com""
    }";

And this is basically what I have so far:

`

    public async void Method4()
    {

        GoogleCredential googleCredential = GoogleCredential.FromJson(JSONdata);
        ServiceAccountCredential credential = (ServiceAccountCredential)googleCredential.UnderlyingCredential;

        Console.WriteLine("credential.id: " + credential.Id);
        Console.WriteLine("credential.KeyId: " + credential.KeyId);

        credential.Scopes = new[] { "https://mail.google.com/" };
        try
        {
            credential.GetAccessTokenForRequestAsync().Wait();

            // this seems to work OK (I think), since I actually get a token!!
            Console.WriteLine("Token: " + credential.Token.AccessToken.ToString());     
        }
        catch (Exception ex)
        {
            Console.WriteLine("OEPS: " + ex.ToString());
        }

        // credential.Id (service account) instead of the real emailadres has the same outcome
        var oauth2 = new SaslMechanismOAuth2(emailadres, credential.Token.AccessToken);
        Console.WriteLine("IsAuthenticated: " + oauth2.IsAuthenticated);
        Console.WriteLine("MechanismName: " + oauth2.MechanismName);

        Console.WriteLine("open imap client");
        using (var client = new ImapClient())
        {
            client.Connect("imap.gmail.com", 993, SecureSocketOptions.SslOnConnect);
            Console.WriteLine("client.IsConnected: " + client.IsConnected);

            Console.WriteLine("authenticate");
            try
            {
                client.Authenticate(oauth2);
                Console.WriteLine("authenticated");
            }
            catch (Exception ex)
            {
                Console.WriteLine("Auth: " + ex.ToString());
            }

            try
            {
                Console.WriteLine("client.Inbox.Count: " + client.Inbox.Count);
            }
            catch (Exception ex)
            {
                Console.WriteLine("Count: " + ex.Message.ToString());
            }

            client.Disconnect(true);
        }

    }

`

Output:

credential.id: id-xxxxxxxxx@logical-craft-yyyyyyy-h7.iam.gserviceaccount.com (scrambled) credential.KeyId: bce4e547b217835b2722c8b1c41b18b4c76b776b Token: ya29.c.c0ASRK0Gb655ubbb....9kVdr8aRWIWsVrahj7 (shortened, actual value is 1024 bytes) IsAuthenticated: False MechanismName: XOAUTH2 open imap client client.IsConnected: True authenticate Auth: MailKit.Security.AuthenticationException: Authentication failed. bij MailKit.Net.Imap.ImapClient.ProcessAuthenticateResponse(ImapCommand ic, SaslMechanism mechanism) in D:\src\MailKit\MailKit\Net\Imap\ImapClient.cs:regel 1057 bij MailKit.Net.Imap.ImapClient.Authenticate(SaslMechanism mechanism, CancellationToken cancellationToken) in D:\src\MailKit\MailKit\Net\Imap\ImapClient.cs:regel 1161 bij Import_v1.GoogleOauth2.d__10.MoveNext() in C:\Users\frank\source\repos\Import_v1\Import_v1\GoogleOauth2.cs:regel 451 Count: The ImapClient is not authenticated.

I even took these steps with both my personal gmail account as wel as my corporate account, but same results.

It's really driving me crazy, I already spent several days on this. What am I doing wrong here? Hope someone can help!

[Davilarek]

I'm experiencing a similar issue. I also enabled logging to see what's going on.

C: A00000001 AUTHENTICATE XOAUTH2 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: + 
C: 
S: * BYE Too many commands before auth 

Looks like MailKit "forgets" to actually send the authentication info and the connection cuts by the server.

[jstedfast]

@Davilarek If I were to guess, you are trying to reuse a SaslMechanismOAuth2 instance without first resetting it.

If you are going to reuse a SaslMechanism instance, you MUST call SaslMechanism.Reset() first.

@frank4040 it might help if you could get a protocol log - the server response might contain useful info (the base64 encoded blob that it sends back after a failed authentication).

Odds are high that I'm not going to be able to help you, though, because OAuth2 is impossible to debug from the client side. You will likely need to reach out to Google for assistance.

[frank4040]

@jstedfast, this is the log I get:

Connected to imaps://imap.gmail.com:993/ S: OK Gimap ready for requests from aaa.bbb.ccc.ddd cf29mb85450632edb C: A00000000 CAPABILITY S: CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER S: A00000000 OK Thats all she wrote! cf29mb85450632edb C: A00000001 AUTHENTICATE XOAUTH2 **** S: + eyJzdGF0dXMiOiI0MDAiLCJzY2hlbWVzIjoiQmVhcmVyIiwic2NvcGUiOiJodHRwczovL21haWwuZ29vZ2xlLmNvbS8ifQ== C: S: A00000001 NO Lookup failed cf29mb85450632edb C: A00000002 LOGOUT S: * BYE Logout Requested cf29mb85450632edb S: A00000002 OK Quoth the raven, nevermore... cf29mb85450632edb

Of course it's possible my service account is not set up correctly. But if i would know the correct Google cloud settings (in terms of roles, rights, permissions, keys, or whatever) then I can easily set that up since I have full control in the cloud console.

Just a simple roadmap to have the basics working is essentially all I need. Already gone through your page at https://github.com/jstedfast/MailKit/blob/master/GMailOAuth2.md, but there is no service account section.

Hope this logging helps in getting me out of the dirt!

[frank4040]

@jstedfast Decoded the base64 and it says: {"status":"400","schemes":"Bearer","scope":"https://mail/google.com"}

Above log was btw with the service account, this is the log using the real google account. (that should be used in the SaslMechanismOAuth2 instead of the service account, as you stated in some other posts about this subject)

Connected to imaps://imap.gmail.com:993/ S: OK Gimap ready for requests from a.b.c.d h10mb26400215ede C: A00000000 CAPABILITY S: CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN AUTH=OAUTHBEARER S: A00000000 OK Thats all she wrote! h10mb26400215ede C: A00000001 AUTHENTICATE XOAUTH2 **** S: + eyJzdGF0dXMiOiI0MDAiLCJzY2hlbWVzIjoiQmVhcmVyIiwic2NvcGUiOiJodHRwczovL21haWwuZ29vZ2xlLmNvbS8ifQ== C: S: A00000001 NO [AUTHENTICATIONFAILED] Invalid credentials (Failure) C: A00000002 LOGOUT S: * BYE Logout Requested h10mb26400215ede S: A00000002 OK Quoth the raven, nevermore... h10mb26400215ede

Slightly different, but still failing. However, hope this gives some new clues. The returned base64 has the same content: {"status":"400","schemes":"Bearer","scope":"https://mail/google.com"}

[jstedfast]

You could try using credential.ClientEmail. Perhaps that's what it wants.

[frank4040]

@jstedfast In the Google's 'ServiceAccountCredential' object there is no member by that name. There is a "client_email" in the json structure, but that is the service account (SERVICEACCOUNTNAME@PROJECTID.iam.gserviceaccount.com) which I already tried.

So taking one step back... I already receive an accesstoken from google, so that leads me to think that google at least 'opens the gate' for me (sort-a-speak). I guess the 'client.Authenticate(oauth2)' step is where MailKit basically asks google: 'can I please use the inbox for IMAP access with this useraccount and this accesstoken that you previously supplied'. And that is where it seems to go wrong.

Honestly, I think it is entirely possible that I am missing some project setting in the google cloud console, like a hidden checkbox somewhere that needs ticked, a missing permission, etc.

So is there a general/practical usecase where MailKit has actually worked with a google service account? And if so, what should be the settings (if any) in the google cloud console?

At this point I don't really care about security, just functionality. So I have no problem setting everything 'open' to the world, as long as I can read mails. Al the finetunings can come later.

[jstedfast]

In the Google's 'ServiceAccountCredential' object there is no member by that name. There is a "client_email" in the json structure, but that is the service account (SERVICEACCOUNTNAME@PROJECTID.iam.gserviceaccount.com) which I already tried.

👍

So is there a general/practical usecase where MailKit has actually worked with a google service account? And if so, what should be the settings (if any) in the google cloud console?

There's nothing that I know of that I can just point to, unfortunately, because of the nature of open source (kinda can't go checking-in code with credentials into a public repo).

I know for sure that I have personally gotten it to work in the past and I'm pretty sure I just used the normal email address for my account, but this was probably going on 4-5 years ago at this point and is what I based some of my original documentation on.

The best I can suggest to you is to go back to the docs and reread them and follow the steps to make sure you didn't miss anything.

Verify that you set the proper scope (https://mail.google.com and not SMTP.Send or whatever they called it).

You could also try setting up an app-specific password (not sure if that is still an option, but it used to be) to bypass OAuth altogether.

I truly hate OAuth. I get so many questions every week from people who are struggling with setting it up and I can never help them because all I can say is "RTFM". There's absolutely nothing I can do from my end or MailKit's end. It's all up to you to properly configure it. I don't even know what options are available in Google's configuration screens anymore.

FWIW, Both Google and Microsoft essentially consider IMAP/SMTP/etc to be "dead" and don't really want you using those protocols anyway. They want you to use their HTTP REST APIs:

• Google: https://www.nuget.org/packages/Google.Apis.Gmail.v1
• Microsoft: https://www.nuget.org/packages/Microsoft.Graph

[frank4040]

Yeah... 2 years ago I already had to implement MS Graph support, which succeeded by the way. I was able to create a flow where I read mail messages in a stream and directly pass it to my good old parsing routine wich uses a MimeKit object for all content parsing (message = MimeKit.MimeMessage.Load(stream)). So luckily I could get away with just some fetching logic.

In this case, I at least already found a way to extract a simular format (.eml) using the Google API GmailService(), so in that regard I can hopefully blend it in smoothly, using this Google API (again) for just the fetching part.

The only wall I keep hitting is that even Google's own API does not seem to accept a service account, only a standard 'OAuth Client ID' which gives me a browser page for consent.

And although that works in my simple console testapp (I can fetch mails after the consent), this cannot be used in our production env because we use a Windows Service to fetch mails and store them in a database, all in the background, fully unattended, for numerous mailboxes. And this service (obviously) has no user interface, plus no one is logged into the server where this service runs. And even when someone would occasionally log in, it wil never be under the same username as the service. So no one will ever be able to see a consent screen (I think, didn't try. But is unacceptable anyway, I guess the service would even get stuck immediately and stops processing altogether, waiting for a response that never comes).

It's all very frustrating, since most Google docs seem out of date or broken. Links don't work like they should, most info and code is obsolete, it's a real mess.

In about a month time it becomes mandatory to use Oath when fetching mails from gmail, so I hope I will find a solution before then (fingers crossed).

But thanks for everything!

(When I find a solution, I will post it here for others)

[gdetra]

Hi I have the same problem! Simeone found solutions to that?