search

OfflineIMAP / offlineimap

Read/sync your IMAP mailboxes (python2) [LEGACY: move to offlineimap3]
http://www.offlineimap.org
Other
1.78k stars 360 forks source link

Failed to do SSL handshake when using proxy #654

Open coiby opened 4 years ago

coiby commented 4 years ago

General informations

Configuration file offlineimaprc

[general]
ui = ttyui
accounts = Gmail
fsync = False

[Account Gmail]
localrepository = Gmail-Local
remoterepository = Gmail-Remote
maxage = 60
synclabels = yes
labelsheader = X-Keywords
proxy = HTTP:127.0.0.1:8123
authproxy = HTTP:127.0.0.1:8123
postsynchook = notmuch new

[Repository Gmail-Local]
type = GmailMaildir
localfolders = ~/Documents/Mail/Gmail
nametrans: lambda s: dict((value,key) for key,value in
                      {  '[Gmail]/Starred' : 'starred',
                         '[Gmail]/Trash'   : 'trash',
                         '[Gmail]/Spam'    : 'spam',
                         '[Gmail]/Sent Mail' : 'sent',
                         '[Gmail]/Important' : 'important',
                         '[Gmail]/Drafts'    : 'drafts',
                         '[Gmail]/All Mail'  : 'archive',
                         'INBOX'             : 'inbox',
                      }.iteritems()).get (s, s).decode ('utf8').encode ('imap4-utf-7')

[Repository Gmail-Remote]
type = Gmail
remoteuser = xxx
remotepasseval = get_pass()
realdelete = no
maxconnections = 3
ipv6 = False
nametrans: lambda s: {  '[Gmail]/Starred' : 'starred',
                        '[Gmail]/Trash'   : 'trash',
                        '[Gmail]/Spam'    : 'spam',
                        '[Gmail]/Sent Mail' : 'sent',
                        '[Gmail]/Important' : 'important',
                        '[Gmail]/Drafts'    : 'drafts',
                        '[Gmail]/All Mail'  : 'archive',
                        'INBOX'             : 'inbox',
                      }.get (s, s).decode ('imap4-utf-7').encode ('utf8')

folderfilter = lambda foldername: foldername in ['[Gmail]/Important', '[Gmail]/Sent Mail', '[Gmail]/Starred', 'INBOX']
# Necessary as of OfflineIMAP 6.5.4
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
# Necessary to work around https://github.com/OfflineIMAP/offlineimap/issues/573 (versions 7.0.12, 7.2.1)
ssl_version = tls1_2

pythonfile (if any)

Logs, error

OfflineIMAP 7.3.2
  Licensed under the GNU GPL v2 or any later version (with an OpenSSL exception)
imaplib2 v2.101 (bundled), Python v2.7.17, OpenSSL 1.1.1d  10 Sep 2019
Account sync Gmail:
 *** Processing account Gmail
 Establishing connection to imap.gmail.com:993 (Gmail-Remote)
 ERROR: While attempting to sync account 'Gmail'
  [Errno 0] Error
 *** Finished account 'Gmail' in 0:11
ERROR: Exceptions occurred during the run!
ERROR: While attempting to sync account 'Gmail'
  [Errno 0] Error

Traceback:
  File "/usr/lib/python2.7/site-packages/offlineimap/accounts.py", line 293, in syncrunner
    self.__sync()
  File "/usr/lib/python2.7/site-packages/offlineimap/accounts.py", line 369, in __sync
    remoterepos.getfolders()
  File "/usr/lib/python2.7/site-packages/offlineimap/repository/IMAP.py", line 452, in getfolders
    imapobj = self.imapserver.acquireconnection()
  File "/usr/lib/python2.7/site-packages/offlineimap/imapserver.py", line 572, in acquireconnection
    af=self.af,
  File "/usr/lib/python2.7/site-packages/offlineimap/imaplibutil.py", line 202, in __init__
    super(WrappedIMAP4_SSL, self).__init__(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/offlineimap/bundled_imaplib2.py", line 2191, in __init__
    IMAP4.__init__(self, host, port, debug, debug_file, identifier, timeout, debug_buf_lvl)
  File "/usr/lib/python2.7/site-packages/offlineimap/bundled_imaplib2.py", line 360, in __init__
    self.open(host, port)
  File "/usr/lib/python2.7/site-packages/offlineimap/imaplibutil.py", line 210, in open
    super(WrappedIMAP4_SSL, self).open(host, port)
  File "/usr/lib/python2.7/site-packages/offlineimap/bundled_imaplib2.py", line 2204, in open
    self.ssl_wrap_socket()
  File "/usr/lib/python2.7/site-packages/offlineimap/bundled_imaplib2.py", line 554, in ssl_wrap_socket
    self.sock = ctx.wrap_socket(self.sock, server_hostname=self.host)
  File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 599, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()

Steps to reproduce the error

chris001 commented 4 years ago

@coiby Question1 : Why are you using proxy? Are you trying to bypass the GFW in China? Question 2: Did you set these settings? A. proxy: https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf#L439 B. authproxy: https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf#L503 C. ipv6: https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf#L665

coiby commented 4 years ago

@chris001 Thanks for the reply!

Question 1: Yes, I'm trying to bypass the GFW. But as long as proxy is enabled, this issue will occur.

Question 2: I've set proxy/authproxy/ipv6 in the configuration file.

chris001 commented 4 years ago

Question 1: Yes, I'm trying to bypass the GFW. But as long as proxy is enabled, this issue will occur.

As test, are you able to connect to https://mail.google.com thru the machine's browser, using the same proxy config?

Question 2: I've set proxy/authproxy/ipv6 in the configuration file.

Are you pointing to a SOCKS or HTTP client on localhost (local machine), which is configured to connect thru your VPN subscription's server?

coiby commented 4 years ago

Question 1: Yes, I'm trying to bypass the GFW. But as long as proxy is enabled, this issue will occur.

As test, are you able to connect to https://mail.google.com thru the machine's browser, using the same proxy config?

Yes, I could access mail.google.com with the same proxy in the browser.

Question 2: I've set proxy/authproxy/ipv6 in the configuration file.

Are you pointing to a SOCKS or HTTP client on localhost (local machine), which is configured to connect thru your VPN subscription's server?

I've trid both SOCKS and HTTP proxy which are provided by the same VPN client. While composing this email, I give another VPN provider a try and find out it works. So this SSL handshake issue should be cause by previous VPN service although I don't know why brower with the same proxy config works.

chris001 commented 4 years ago

Question 1: Yes, I'm trying to bypass the GFW. But as long as proxy is > enabled, this issue will occur. As test, are you able to connect to https://mail.google.com thru the machine's browser, using the same proxy config? Yes, I could access mail.google.com with the same proxy in the browser.

Can you successfully compete the login to https://mail.google.com thru the browser, using the original (problematic) VPN service ? Because the gmail login process will redirect you to https://accounts.google.com which might fail...

Question 2: I've set proxy/authproxy/ipv6 in the configuration file. Are you pointing to a SOCKS or HTTP client on localhost (local machine), which is configured to connect thru your VPN subscription's server? I've trid both SOCKS and HTTP proxy which are provided by the same VPN client. While composing this email, I give another VPN provider a try and find out it works. So this SSL handshake issue should be cause by previous VPN service although I don't know why brower with the same proxy config works.

That's great news, you got offlineimap to work thru the second VPN service.

I suspect the reason for the fail on one VPN, and success with the other VPN, is because possibly GFW is an aggressive blocker, combined with the fact that we're still leaking DNS queries in plaintext, which I intend to fix ASAP, by adding private encrypted DNS lookups feature, enabled by default.

coiby commented 4 years ago

Question 1: Yes, I'm trying to bypass the GFW. But as long as proxy is > enabled, this issue will occur. As test, are you able to connect to https://mail.google.com thru the machine's browser, using the same proxy config? Yes, I could access mail.google.com with the same proxy in the browser.

Can you successfully compete the login to https://mail.google.com thru the browser, using the original (problematic) VPN service ? Because the gmail login process will redirect you to https://accounts.google.com which might fail...

Yes, I could access mail.google.com in private browsing mode.

Question 2: I've set proxy/authproxy/ipv6 in the configuration file. Are you pointing to a SOCKS or HTTP client on localhost (local machine), which is configured to connect thru your VPN subscription's server? I've trid both SOCKS and HTTP proxy which are provided by the same VPN client. While composing this email, I give another VPN provider a try and find out it works. So this SSL handshake issue should be cause by previous VPN service although I don't know why brower with the same proxy config works.

That's great news, you got offlineimap to work thru the second VPN service.

I suspect the reason for the fail on one VPN, and success with the other VPN, is because possibly GFW is an aggressive blocker, combined with the fact that we're still leaking DNS queries in plaintext, which I intend to fix ASAP, by adding private encrypted DNS lookups feature, enabled by default.

Thank you! I'm looking forward to this private encrypted DNS lookups feature.

chris001 commented 4 years ago

Thank you! I'm looking forward to this private encrypted DNS lookups feature.

It's preferred (but not required) to encrypt the DNS lookups at the operating system level. Some OS already have this feature built in and available. Which operating system are you running on?

coiby commented 4 years ago

Thank you! I'm looking forward to this private encrypted DNS lookups feature.

It's preferred (but not required) to encrypt the DNS lookups at the operating system level. Some OS already have this feature built in and available. Which operating system are you running on?

I'm using Linux. I tried dnscrypt-proxy but it produced the same SSL handshake error. I also tried putting the correct DNS record of (the one used by the working VPN) imap.gmail.com into /etc/hosts which failed too. By the way, I tried sending email using the not-working VPN in neomutt and neomutt gave an SMTP session error. So I guess there may be something wrong with the not-working VPN.

chris001 commented 4 years ago

@coiby Try this with the not-working VPN, and post back with the output, just to test if encrypted DNS would work on that VPN:

apt install -y git
yum install -y git
git clone https://github.com/stamparm/python-doh
cd python-doh
python
import client
print(client.query("mail.google.com"))
print(client.query("mail.google.com", "AAAA"))
print(client.query("accounts.google.com"))
print(client.query("accounts.google.com", "AAAA"))
quit()
coiby commented 4 years ago

Try this with the not-working VPN just to test if encrypted DNS would work with it:

apt install -y git
yum install -y git
git clone https://github.com/stamparm/python-doh
cd python-doh
python
import client
print(client.query("mail.google.com"))
print(client.query("mail.google.com", "AAAA"))
print(client.query("accounts.google.com"))
print(client.query("accounts.google.com", "AAAA"))
quit()

Thank you for the instructions! Here is the result,

$ proxychains python
[proxychains] config file found: /etc/proxychains.conf
[proxychains] preloading /usr/lib/libproxychains4.so
Python 3.8.1 (default, Jan 22 2020, 06:38:00)
[GCC 9.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import client
>>> print(client.query("mail.google.com"))
['216.58.200.37']
>>> print(client.query("mail.google.com", "AAAA"))
['googlemail.l.google.com.']
>>> print(client.query("accounts.google.com"))
['216.58.200.237']
>>> print(client.query("accounts.google.com", "AAAA"))
['2404:6800:4008:801::200d']
>>> quit()
chris001 commented 4 years ago

@coiby Your results:

print(client.query("mail.google.com", "AAAA")) ['googlemail.l.google.com.']

My results:

print(client.query("mail.google.com", "AAAA"))
['googlemail.l.google.com.', '2607:f8b0:4009:805::2005']

Besides the fact you're getting the google IPs which are geolocated in Taiwan, closest to you, in China, and I'm getting the IPs in Chicago US, geographically nearest to me, the only difference I see, is when I query for the IPb6 address of mail.google.com, I get an array containing the hostname, googlemail.l.google.com, and the closest geolocated IPv6 address of the hostname, 2607:f8b0:4009:805::2005, while you got only the hostname, googlemail.l.google.com, and no IP address for the hostname. This lack of IPv6 address, when your offlineimap client app is configured to use IPv6 only, might, or might not, cause a problem in the code at runtime. Needs more investigation.

urugang commented 2 years ago

it works with config ipv6=False 

# ** general
[general]
accounts = gmail
maxsyncaccounts = 5
pythonfile = ~/.offlineimap.py

# ** gmail
# *** account
[Account gmail]
localrepository = gmail-local
remoterepository = gmail-remote
proxy = SOCKS5:localhost:2080

# *** local
[Repository gmail-local]
type = Maildir
localfolders = ~/Mail/gmail

# *** remote
[Repository gmail-remote]
type = Gmail
remoteuser = xxx@gmail.com
remotepasseval = get_pass("gmail.com/xxx")
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
ipv6 = False
radfish commented 1 year ago

Note: ipv6 must be set to to yes or no; otherwise, if left unset, proxy setting will be silently ignored. See PR #162