Automagically fetch Gmail certificates

[ildar]

There's the special type = Gmail and the documentation says: Specify the Gmail user name. This is the only mandatory parameter. But having configured so it complains: $ offlineimap

OfflineIMAP 6.5.6.1-devel Licensed under the GNU GPL v2+ (v2 or any later version) Account sync Z2Gmail: * Processing account Z2Gmail Establishing connection to imap.gmail.com:993 ERROR: No CA certificates and no server fingerprints configured. You must conf igure at least something, otherwise having SSL helps nothing. * Finished account 'Z2Gmail' in 0:00 ERROR: Exceptions occurred during the run! ERROR: No CA certificates and no server fingerprints configured. You must confi gure at least something, otherwise having SSL helps nothing.

Is it possible to fetch certificates automatically?

[yang-ling]

My configuration looks like this

[Repository Remote-gmail]

type = Gmail

remoteuser = user@gmail.com

remotepasseval = get_pass("my_user")

remotehost = imap.gmail.com
remoteport = 993

ssl = yes

sslcacertfile = /etc/ssl/certs/ca-certificates.crt

########## Advanced settings

maxconnections = 2

nametrans = trans_gmail_remote

get_pass and nametrans are methods in my pythonfile

I think you need at least configure sslcacertfile

I use Archlinux. If you also use it, you can just copy my config.

[nicolas33]

I think Google is taking the XOAUTH way. This latest authentication method supports fetching of tokens.

[danilobellini]

I had the same problem here with OfflineIMAP 7.0.8 on Arch Linux. Adding this line from the @yang-ling config was enough:

sslcacertfile = /etc/ssl/certs/ca-certificates.crt

There's no ssl, remotehost and remoteport in my config. As it seems, either the sslcacertfile or some other configuration still unknown for me is required, and I think the message "This is the only mandatory parameter" should be improved/fixed to include this information.

[nicolas33]

It appears that XOAUTH2 with refresh token looks incompatible with starttls.

[nicolas33]

It appears that XOAUTH2 with refresh token looks incompatible with starttls.

This is expected behaviour. The reason is that the Gmail type enables ssl so it must be configured correctly.

I think Google is taking the XOAUTH way. This latest authentication method supports fetching of tokens.

Also, XOAUTH2 does not prevent from having SSL working.

IOW, the original request to automatically download the certificate is valid.

[chris001]

@nicolas33 I think you'll be very interested in this page because it contains all of the troubleshooting steps for sending mail, in one place, including the full details on the best ways to get Gmail working and encryption. https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting Especially liking the sections on "XOAUTH2", "Encryption" and "Opportunistic TLS". Did you know, SSL on port 465 has been deprecated by the IETF since 1998! The only applications still trying to use it are Microsoft Exchange Server and Outlook! And they shouldn't! Only STARTTLS on port 587 is the right way!

[nicolas33]

Thanks! I did not add XOAUTH2 in offlineimap and I'm not much concerned by this protocol (for now, at least). I'm having hard times with this thing.

The port 993 (SSL) is hardcoded for the Gmail type. This was done @aroig (if I'm right) way before XOAUTH2 was provided by Google.

This might be why STARTTLS won't work. I didn't check what imaplib2 does and how we use it in the STARTTLS case but I wouldn't be suprised we are trying the wrong port.

[chris001]

Exactly, port 587 is for sending mail out thru SMTP such as how phpmailer does it. Port 993 is the right port for secure reading IMAPS with SSL or STARTTLS. Here's the official documentation for XOAUTH2, with TLS, and the official Python code sample: https://github.com/google/gmail-oauth2-tools/wiki/OAuth2DotPyRunThrough https://raw.githubusercontent.com/google/gmail-oauth2-tools/master/python/oauth2.py

[aroig]

I'm landing here only now due to @nicolas33 mention. I'm not sure what this issue is about anymore...

Regarding the certificate problem originally reported, yes, with ssl, we need to configure the list of Certificate Authority certificates (the sslcacertfile). I've just discovered that you can also use OS-DEFAULT as a value and offlineimap will try to find it on the common locations.

The SSL protocol fetches the certificate for the imap server, which is signed by a certification authority. Then fetches the certificate of the certification authority, which is signed by another certification authority, and keeps going up the tree until it finds a certification authority in the local list of trusted certification authorities you have provided.

This scheme protects you against someone in your network pretending to be gmail's imap server and spying on your email. You trust that every CA is legal, and only signs a certificate for which it has checked the identity, etc. A local spy on your network will have a hard time finding a CA to sign his fake gmail server. Without this chain of trust, you would instead be trusting a local DNS server giving you back the IP of gmail.

Getting gmail's certificate over the wire, or any of the CA's needed to validate it, from an untrusted server, is a bad idea. It kind of defeats the purpose of the chain of trust. If you want to use the chain of trust, there is no way around it, you need a local file with a handful of trusted CA's, obtained via a package manager with signed packages, etc. What offlineimap can do is try to find the local file in the usual places (which apparently it can do). Alternatively, we could hard-code the path at installation time. Then distros can package offlineimap with the cacerts file properly configured.

Regarding the XOAUTH discussion, I have not tried XOAUTH yet (it is somewhere on my list). I'm not sure I understood what is being discussed here. Is there a problem with gmail's backend hard-coding port 993? When I touched the gmail code, this port was already hard-coded, but of course it can be made configurable if necessary.

[nicolas33]

Regarding the XOAUTH discussion, I have not tried XOAUTH yet (it is somewhere on my list). I'm not sure I understood what is being discussed here. Is there a problem with gmail's backend hard-coding port 993? When I touched the gmail code, this port was already hard-coded, but of course it can be made configurable if necessary.

Ok, thanks. I was wrong, we are fine with port 993.

Getting gmail's certificate over the wire, or any of the CA's needed to validate it, from an untrusted server, is a bad idea.

Thank you. I'm not sure we want to support automagically fetching the certificate. Though, I would not be opposed to a configuration option like "automagically_fetch_certificate" as long as we explain this is less secured than certificate checks.

Now, about STARTTLS.

Port 993 is the right port for secure reading IMAPS with SSL or STARTTLS.

STARTTLS allows to enable SSL/TLS tunnels while starting in clear text mode. IOW, this elevates the security from clear text.

Here, I wonder XOAUTH2 and STARTTLS are exclusive or not in the first place. The reason I want to check this is because STARTTLS does NOT requires checking the certificate. IOW, this would be another way to "bypass" the certificate checks and not having the sslcacertfile configuration option set. I think both are NOT compatible because STARTTLS and XOAUTH2 are meant for very different purposes.

BUT in imapserver.py, we have:

        # Authentication routines, hash keyed by method name
        # with value that is a tuple with
        # - authentication function,
        # - tryTLS flag,
        # - check IMAP capability flag.
        auth_methods = {
          "GSSAPI": (self.__authn_gssapi, False, True),
          "XOAUTH2": (self.__authn_xoauth2, True, True),
          "CRAM-MD5": (self.__authn_cram_md5, True, True),
          "PLAIN": (self.__authn_plain, True, True),
          "LOGIN": (self.__authn_login, True, False),
        }

where tryTLS is about STARTTLS. I'm not sure trying STARTTLS with XOAUTH2 is correct in the first place.

Perhaps the XOAUTH2 authors can help, here. @cscorley @Frizlab @Roguelazer could you confirm that trying STARTTLS with XOAUTH2 is intended? Would this work?

[nicolas33]

I'm not sure trying STARTTLS with XOAUTH2 is correct in the first place.

Again, while this would be surprising (both are meant for different use cases) this limitation might not apply for all the IMAP servers out there. If Gmail (correctly, IMHO) does not allow STARTTLS with XOAUTH2 we might like to try STARTTLS with XOAUTH2 for other IMAP servers.

[chris001]

Here's the official gmail python code sample for XOAUTH2 IMAP4 login. Note how the login is 4 simple lines of code. There's no mention of STARTTLS for the IMAP login. https://github.com/google/gmail-oauth2-tools/blob/master/python/oauth2.py#L255 The SMTP sample however does use STARTTLS and port 587 smtp submission.

def TestImapAuthentication(user, auth_string):
  """Authenticates to IMAP with the given auth_string.
  Prints a debug trace of the attempted IMAP connection.
  Args:
    user: The Gmail username (full email address)
    auth_string: A valid OAuth2 string, as returned by GenerateOAuth2String.
        Must not be base64-encoded, since imaplib does its own base64-encoding.
  """
  print
  imap_conn = imaplib.IMAP4_SSL('imap.gmail.com')
  imap_conn.debug = 4
  imap_conn.authenticate('XOAUTH2', lambda x: auth_string)
  imap_conn.select('INBOX')

def TestSmtpAuthentication(user, auth_string):
  """Authenticates to SMTP with the given auth_string.
  Args:
    user: The Gmail username (full email address)
    auth_string: A valid OAuth2 string, not base64-encoded, as returned by
        GenerateOAuth2String.
  """
  print
  smtp_conn = smtplib.SMTP('smtp.gmail.com', 587)
  smtp_conn.set_debuglevel(True)
  smtp_conn.ehlo('test')
  smtp_conn.starttls()
  smtp_conn.docmd('AUTH', 'XOAUTH2 ' + base64.b64encode(auth_string))

We should consider adding this oauth2.py official gmail code, ship it with offlineimap as a library for offlineimap use, and rely on it for all XOAUTH2 auth and login, this could close any bug related to gmail secure oauth2 login.

[thekix]

Hi,

probably this patch from offlineimap (python 3) could help here:

https://github.com/OfflineIMAP/offlineimap3/pull/17

Regards, kix