Can't Start Servers

[w5pny]

Running "sudo ./emailproxy.py --no-gui" yields the attached log. I'm not sure why root can't open these ports.... log.txt

[w5pny]

By putting "local_address = 127.0.0.1" in each server stanza in the config file INSTEAD of "local_address = localhost" it works! But localhost is returned properly by DNS -- so I'm not sure why I need to do this.

[w5pny]

Now that emailproxy.py seems to be working, I can't get fetchmail to use a non-standard port for POP3 or IMAP. Have there been any successes with fetchmail?????

[w5pny]

I think I have a workaround for fetchmail on alternate ports, but the options I need may not be supported in the future. In any case, when I get the the authorization phase, I get errors from emailproxy.py:

Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/pystray/_base.py", line 324, in inner callback(self) File "/usr/local/lib/python3.8/dist-packages/pystray/_base.py", line 449, in call return self._action(icon, self) File "./emailproxy.py", line 2409, in authorise_account self.create_authorisation_window(request) File "./emailproxy.py", line 2431, in create_authorisation_window authorisation_window = webview.create_window(window_title, request['permission_url'], on_top=True) AttributeError: module 'webview' has no attribute 'create_window'

If I use --local-server-auth, the redirects are wrong (http://localhost).

[simonrob]

There are lots of things mentioned in these posts, but to focus on the main questions:

• sudo mode should work in the same way as normal usage. I don't know why you needed to manually resolve localhost, but given you wanted 127.0.0.1 I wonder whether it's anything to do with it trying IPv6 before IPv4? See #152 for potentially related discussion. I'd be interested in any further detail you can uncover here.
• I don't use fetchmail personally, but know that others do. This is obviously an issue with that client rather than the proxy. But if you're already using sudo you could just use the standard ports if that's easier.
• The webview error message makes me think you haven't installed the proxy's requirements. Have you done this? (python -m pip install -r requirements.txt). If so, what's the output of python -m pip list?
• What do you mean by "the redirects are wrong"? This is something you configure yourself via redirect_uri (and, if needed, redirect_listen_address). http://localhost is the default because this is the correct value in most cases. See the sample configuration file for more detail here.

[w5pny]

• sudo mode should work in the same way as normal usage. I don't know why you needed to manually resolve localhost, but given you wanted 127.0.0.1 wonder whether it's anything to do with it trying IPv6 before IPv4? I'd be interested in any further detail you can uncover here.

Here is what dig returns:

dig localhost

; <<>> DiG 9.16.1-Ubuntu <<>> localhost ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56673 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;localhost. IN A

;; ANSWER SECTION: localhost. 0 IN A 127.0.0.1

;; Query time: 0 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) ;; WHEN: Sat Apr 15 10:09:17 MDT 2023 ;; MSG SIZE rcvd: 54

• I don't use fetchmail personally, but know that others do. This is obviously an issue with that client rather than the proxy.

Yes, I was able to finally find a way to configure fetchmail to work. Fetchmail has made it less obvious since it is clear they don't want to support non-ssl/tls connections. The option to disable ssl/tls supposed won't be supported in the future AND even when it's disabled with the current option, one can change the default IMAP or POP ports easily. Only can only change the port on the command line when fetchmail is invoked now. It used to be one could do it in the config file as well.

Having gotten it to work brings me to the next problem -- getting emailproxy.py to successfully authorize the connection. I'll provide that info in a separate post, since now everything else seems to be working ok.

• The webview error message makes me think you haven't installed the proxy's requirements. Have you done this? (python -m pip install -r requirements.txt). If so, what's the output of python -m pip list?

Yes, I did (python -m pip install -r requirements.txt), but that wasn't enough, I had to do (`python -m pip install pywebview) to fix the problem.

Here is the output of python -m pip list:

Package Version

---

aeidon 1.7
appdirs 1.4.3
apt-xapian-index 0.49
apturl 0.5.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 0.15.5
astropy 4.0
atomicwrites 1.1.5
attrs 19.3.0
Automat 0.8.0
Babel 2.6.0
backcall 0.1.0
bashate 0.6.0
bcrypt 3.1.7
beautifulsoup4 4.11.1
bleach 3.1.1
blinker 1.4
block-tracing 1.0.1
bottle 0.12.25
Brlapi 0.7.0
Brotli 1.0.9
Cartopy 0.0.0
certifi 2019.11.28
cffi 1.15.0
chardet 3.0.4
charset-normalizer 2.1.1
checkbox-ng 0.23
checkbox-support 0.22
Cheetah3 3.2.4
chirp 0.3.0.dev0
chm2pdf 0.9
chrome-gnome-shell 0.0.0
Click 7.0
click-plugins 1.1.1
colorama 0.4.3
colorlog 4.1.0
colour 0.1.5
command-not-found 0.3
configobj 5.0.6
constantly 15.1.0
cryptography 2.8
cssselect2 0.4.1
cupshelpers 1.0
cycler 0.10.0
davis-weatherlink-scraper 0.1.0
dblatex 0.3.11py3
dbus-python 1.2.16
dbxfs 1.0.63
decorator 4.4.2
defer 1.0.6
defusedxml 0.6.0
dell-recovery 0.0.0
deluge 2.0.3
devscripts 2.20.2ubuntu2
diff-match-patch 20181111
distlib 0.3.0
distro 1.4.0
distro-info 0.23ubuntu1
dnspython 1.16.0
docopt 0.6.2
dropbox 11.35.0
duplicity 0.8.12.0
easygui 0.98.1
entrypoints 0.3
ephem 3.7.7.1
evdev 1.3.0
ExifRead 2.1.2
fasteners 0.14.1
feedparser 5.2.1
filelock 3.0.12
fonttools 4.28.2
fusepyng 1.0.7
future 0.18.2
gbp 0.9.19
GDAL 3.3.2
GeoIP 1.3.2
gpg 1.13.1-unknown
gphoto2 1.9.0
grabserial 1.9.8
gssapi 1.6.1
guacamole 0.9.2
html5lib 1.1
httplib2 0.14.0
hyperlink 19.0.0
hypothesis 4.36.2
idna 2.8
ifaddr 0.1.6
img2pdf 0.3.3
importlib-metadata 1.5.0
incremental 16.10.1
iniparse 0.4
input-remapper 1.5.0
ipykernel 5.2.0
ipython 7.13.0
ipython-genutils 0.2.0
ipywidgets 6.0.0
isodate 0.6.0
jaraco.classes 3.2.3
jedi 0.15.2
Jinja2 2.10.1
jsonschema 3.2.0
jupyter-client 6.1.2
jupyter-console 6.0.0
jupyter-core 4.6.3
kazam 1.4.5
keyring 18.0.1
keyrings.alt 4.2.0
kiwisolver 1.0.1
l20n 4.0.0a1
language-selector 0.1
launchpadlib 1.10.13
lazr.restfulclient 0.14.2
lazr.uri 1.0.3
lockfile 0.12.2
louis 3.12.0
lxml 4.5.0
lz4 3.0.2+dfsg
macaroonbakery 1.3.1
Mako 1.1.0
Markdown 3.1.1
MarkupSafe 1.1.0
matplotlib 3.1.2
meson 0.53.2
mistune 0.8.4
monotonic 1.5
more-itertools 4.2.0
mpmath 1.1.0
mysqlclient 1.4.4
MythTV 31.0.-1
nbconvert 5.6.1
nbformat 5.0.4
netifaces 0.10.4
networkx 2.4
nose 1.3.7
notebook 6.0.3
notify2 0.3
numpy 1.17.4
oauthlib 3.1.0
ocrmypdf 9.6.0+dfsg
olefile 0.46
onboard 1.4.1
packaging 20.3
padme 1.1.1
pandocfilters 1.4.2
paramiko 2.6.0
parso 0.5.2
pbr 5.4.5
pdfkit 0.6.1
pdfminer.six 20191020
pexpect 4.6.0
phply 1.2.5
pickleshare 0.7.5
pikepdf 1.10.3+dfsg
Pillow 7.0.0
pip 20.0.2
pipenv 11.9.0
plainbox 0.25
pluggy 0.13.0
ply 3.11
privy 6.0.0
progressbar 2.5
prometheus-client 0.7.1
prompt-toolkit 2.0.10
protobuf 3.6.1
proxy-tools 0.1.0
psutil 5.5.1
py 1.8.1
py-cpuinfo 5.0.0
py3dns 3.2.1
pyasn1 0.4.2
pyasn1-modules 0.2.1
pycairo 1.16.2
pychm 0.8.6
pycountry 19.8.18
pycparser 2.21
pycrypto 2.6.1
pycups 1.9.73
pycurl 7.43.0.2
pydantic 1.2
pydbus 0.6.0
pydyf 0.1.2
pyenchant 2.0.0
pygame 1.9.6
Pygments 2.3.1
PyGObject 3.36.0
PyHamcrest 1.9.0
pyheif 0.7.0
PyJSONViewer 1.6.0
PyJWT 1.7.1
pykerberos 1.1.14
pyliblo 0.10.0
pymacaroons 0.13.0
pymediainfo 4.1
PyNaCl 1.3.0
PyOpenGL 3.1.0
pyOpenSSL 19.0.0
pyparsing 2.4.6
pyphen 0.11.0
PyPrind 2.11.3
PyQt5 5.14.1
pyqtgraph 0.11.0rc0
PyQtWebEngine 5.14.0
pyRFC3339 1.1
pyrsistent 0.15.5
pyserial 3.4
pyshp 2.1.0
PySide2 5.15.2.1
PySimpleSOAP 1.16.2
pystray 0.19.4
pytest 4.6.9
pytest-arraydiff 0.3
pytest-astropy 0.7.0
pytest-astropy-header 0.1.2
pytest-doctestplus 0.5.0
pytest-openfiles 0.4.0
pytest-remotedata 0.3.2
python-apt 2.0.1+ubuntu0.20.4.1 python-dateutil 2.7.3
python-debian 0.1.36ubuntu1
python-debianbts 3.0.2
python-Levenshtein 0.12.0
python-libtorrent 1.1.13
python-lzo 1.12
python-magic 0.4.16
python-xapp 1.8.1
python-xlib 0.33
pytz 2019.3
pyusb 1.0.2
pywebview 4.0.2
pyxdg 0.26
PyYAML 5.3.1
pyzmq 18.1.1
QtPy 2.3.1
rapid-photo-downloader 0.9.33
rdflib 4.2.2
redis 2.10.5
rencode 1.0.6
reportbug 7.6.0ubuntu1
reportlab 3.5.34
requests 2.28.1
requests-cache 0.4.13
requests-unixsocket 0.2.0
ruamel.yaml 0.15.89
scipy 1.3.3
scour 0.37
screen-resolution-extra 0.0.0
screenkey 0.9
SecretStorage 2.3.1
selenium 4.0.0a1
Send2Trash 1.5.0
sentry-sdk 1.9.10
service-identity 18.1.0
setproctitle 1.1.10
setuptools 45.2.0
Shapely 1.7.1
shiboken2 5.15.2.1
show-in-file-manager 1.1.4
simplejson 3.16.0
sip 4.19.21
six 1.14.0
sortedcontainers 2.1.0
sos 4.4
soupsieve 1.9.5
SPARQLWrapper 1.8.5
ssh-import-id 5.10
stone 3.3.1
sympy 1.5.1
system-service 0.3
systemd-python 234
tenacity 6.0.0
terminado 0.8.2
testpath 0.4.4
timeago 1.0.16
tinycss2 1.1.1
tornado 5.1.1
tqdm 4.64.1
traitlets 4.3.3
translate-toolkit 2.5.0
Twisted 18.9.0
txt2tags 3.4
ubuntu-advantage-tools 8001
ubuntu-drivers-common 0.0.0
ufw 0.36
unattended-upgrades 0.1
unidiff 0.5.5
unity-scope-calculator 0.1
unity-scope-devhelp 0.1
unity-scope-gdrive 0.7
unity-scope-manpages 0.1
unity-scope-tomboy 0.1
unity-scope-zotero 0.1
uritools 3.0.0
urllib3 1.26.12
usb-creator 0.3.7
userspacefs 2.0.5
vboxapi 1.0
vinetto 0.8.0
virtualenv 20.0.17
virtualenv-clone 0.3.0
vobject 0.9.6.1
wadllib 1.3.3
wcwidth 0.1.8
weasyprint 53.4
WeatherLink 1.1.10
webencodings 0.5.1
webview 0.1.5
wheel 0.34.2
widgetsnbextension 2.0.0
wxPython 4.0.7
XDG-Prefs 0.2
xdiagnose 3.8.9
xkit 0.0.0
XlsxWriter 1.1.2
xpra 3.0.6
youtube-dl 2021.12.17
zeroconf 0.24.4
zipp 1.0.0
zope.interface 4.7.1
zopfli 0.1.9

[w5pny]

What do you mean by "the redirects are wrong"? This is something you configure yourself via redirect_uri (and, if needed, redirect_listen_address). http://localhost is the default because this is the correct value in most cases. See the sample configuration file for more detail here.

This is related to my problem with fetchmail after I finally got it working with emailproxy.py ...

When I first tried to authorize the connection, I did get a webview Xwindow browser from Microsoft telling me to select the account. That window ended up going to GoDaddy immediately without me having to click on anything. The normaly Microsoft authentication does this for vendor accounts like GoDaddy and Godaddy does the authorization and returns to Microsoft. My e-mail is provided by GoDaddy, but GoDaddy outsourced all their e-mail service to Microsoft/Outlook but the configuration is done on GoDaddy. I'm not sure how the communication works between Godaddy and Microsoft.

Needless to say when I authorized on the Godaddy page with the webview Xwindow browser window and clicked on the OK button the webview browser complained that I had an adblocker and refused to submit the response.

Repeating this procedure from now on just yields a webview browser window that is a blank page.

So -- I next tried '--external-auth' and '--local-server-auth'.

When with the redirect I get my local web server and the URL passed to it is very long with the "code=..." in it and I get a form from emailproxy.sh to fill in with the code from that URL. I tried that with everything that was in the URL, and with everything from "code=" on, and everything from "code=" on but stopping before the next '&' in the URL. None of those attempts worked, they all gave be parse errors on what I pasted into that from from emailproxy.sh. So I'm not sure what I should be pasting into that form.

[w5pny]

Attached is a screen-shot of the blank page I get mentioned above. Also attached are the messages I get in the emailproxy.py console window where it is being run. authorize.txt

[simonrob]

they don't want to support non-ssl/tls connections.

If this is the main issue then you could use the proxy's support for encrypted local connections. See the sample configuration file for documentation here.

Yes, I did (python -m pip install -r requirements.txt), but that wasn't enough, I had to (python -m pip install pywebview)

This is odd because those two commands should be equivalent in this case. What OS are you using?

[simonrob]

Attached is a screen-shot of the blank page I get mentioned above. Also attached are the messages I get in the emailproxy.py console window where it is being run.

The error messages here are web page rendering issues rather than anything to do with the proxy. I don't quite follow why --external-auth isn't working though - are you pasting the full unedited URL from the external browser window? If so, what output does the proxy give when this is rejected?

[w5pny]

My OS is Ubuntu 20.04 LTS.

I paste the FULL unedited URL from the external browser into Vivaldi, my desktop browser, and it comes back with the default apache page since I haven't set up wep page on this machine. That default says my local web server is working ok and the URL passed to get that resulted in that page is: http://localhost/?code=0.AUYAHqgspUwRVEisknxpI6yzJ3gVEuZf5M9GqMG6QtYv8iKAAPs.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wUA9P_NOrRp3dW0A9tVjzlULZ71S6rBDfETEhQsjZEZ_9O5S8JaVJ1Ix6btGydmXfEBefeY1vj4NA0EAFWnBEVVwktq0f59XvK8k1WsjqH5bHbnJj6iqDOO7qyOMSky-wOTk8VTWWenJ3CeJbeFN15jziyyWV2m38cX08WZtI5WMiWhaEnazDGKnoKhtNXtAi46ALzv_I2M8u1SVWiZmfNwycI_L47OCcfU5TNwPd7xIujeGqRPlpCoGTz91DYBbuxgIs8x6uywSqE79hZSlf-4_w6rDYwgR8oAQ92xPble9Yi4z3DxEklPxvP9lgOEwIjGi-GaLOvtvNsKBuGWkLEPqHvo49xt3sQpI3sUrQPqS75zbsf8sP0OoFzamgjx5ShJvNatGMGVqkra-5IYI4zOrOIhJwl_hyqK1WhQ-uq9C2-kI-xbZIA66Po86dlzt8wkDJT4kMFDXt0764vCXH1eSliGLtlj3r_K2bFo2CI2moVskyascqJZdsR8J79LShHyQ-qSAUWRsE96nPpMBgwK_55-4VnBajf3oIiZYS-TxCjdIMrgjQGZg4nb5hwQC4enIYiYwJh_GlES1hIpbUfrVplGDmeAfS7YAcPzDkXcS-KVQa5Q1P50tnnxjVrd1qrwB91bXbYO3LsNvGdBJAXboLPFMes_20cda59XApserjiAOQkDB8HhrnPLrTodlXCn2bFxONO5hP7Ef245O1zr0Y67thtYlSe1aFxvqbKhM2lNALBR1o57Sq4QZ0RybF7wseD1q0kAEeEvDXUBASNQES_OPNse9_PvZFg4tgilL514EPUgjUYo9NukIfM0grIuGFH98-PFTfpJMKQzgyr1sLCc5oDHlnPkEtr60htpVz2JK657gLEu9JAiAdKJd1MFtlNZb70pMf6LPWfQ-oFNvHLnDjsIt-AJpUXpmt0JbTzBwu9eAPY7gLt4XRL9T-lPUAP4QB4BBPlIerkpAaPJ7OUcimHZEAs5TCGKr5l9EqmOU1Vc

and the emailproxy.py console message is:

2023-04-15 16:13:15: Authorisation request received for w5pny@w5pny.com (external auth mode) 2023-04-15 16:16:03: Returning authorisation request result for w5pny@w5pny.com 2023-04-15 16:16:03: Returning authorisation request result for w5pny@w5pny.com 2023-04-15 16:16:03: Caught exception while requesting OAuth 2.0 credentials for w5pny@w5pny.com: {'error': 'invalid_client', 'error_description': "AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'.\r\nTrace ID: 3b4f894a-2a18-4e85-9fce-147fd6840c00\r\nCorrelation ID: cafccd9e-ff9d-4e31-b318-9f9f152caa1e\r\nTimestamp: 2023-04-15 22:16:03Z", 'error_codes': [7000218], 'timestamp': '2023-04-15 22:16:03Z', 'trace_id': '3b4f894a-2a18-4e85-9fce-147fd6840c00', 'correlation_id': 'cafccd9e-ff9d-4e31-b318-9f9f152caa1e', 'error_uri': 'https://login.microsoftonline.com/error?code=7000218'} 2023-04-15 16:16:03: Warning: IMAP server at 127.0.0.1:1993 (unsecured) proxying outlook.office365.com:993 (SSL/TLS) unable to remove orphan client connection <main.IMAPOAuth2ClientConnection 127.0.0.1:32790 at 0x7f413fe85880> 2023-04-15 16:16:03: Caught exception while requesting OAuth 2.0 credentials for w5pny@w5pny.com: {'error': 'invalid_client', 'error_description': "AADSTS7000218: The request body must contain the following parameter: 'client_assertion' or 'client_secret'.\r\nTrace ID: 551616ae-dc47-465e-bebe-597a05e10c00\r\nCorrelation ID: 283743f5-43f8-49ac-8a42-1c8aaa03e519\r\nTimestamp: 2023-04-15 22:16:03Z", 'error_codes': [7000218], 'timestamp': '2023-04-15 22:16:03Z', 'trace_id': '551616ae-dc47-465e-bebe-597a05e10c00', 'correlation_id': '283743f5-43f8-49ac-8a42-1c8aaa03e519', 'error_uri': 'https://login.microsoftonline.com/error?code=7000218'}

[w5pny]

From what I read in the config documentation, emailproxy,py only supports local server encryption for SMTP, and not for POP or IMAP. I've been tackling POP and IMAP first, so I had assumed fetchmail had to NOT do ssl/tls, is this not true?

[simonrob]

Ah - this is a configuration issue with your AAD client registration. This can be tricky to get set up correctly via Microsoft's immensely confusing interface. Search for the error code that is being returned (AADSTS7000218) to find a resolution here - you may have forgotten to add the client ID to the proxy's config file, or might need to add a tenant ID to your permission/token URLs (or something else).

To get started with the proxy, I'd always recommend first getting things working with the simplest possible setup, ideally with a reused client ID/secret, and only then trying the advanced options. This is especially recommended if you're using the CCG or ROPCG flows.

From what I read in the config documentation, emailproxy,py only supports local server encryption for SMTP, and not for POP or IMAP. I've been tackling POP and IMAP first, so I had assumed fetchmail had to NOT do ssl/tls, is this not true?

The proxy can encrypt all local connections, regardless of protocol. The only SMTP-specific restriction is to do with an O365-specific advanced setup option that does not support SMTP, but this has nothing to do with local encryption.

[w5pny]

As nearly as I can tell in the fetchmail documentation, the only way to do unencrypted sessions is by turning off ssl/tls in the fetchmail config. So to do encrypted sessions I have to turn on ssl/tls in fetchmail for POP and IMAP, but the emailproxy.py documentation says the local POP/IMAP servers can't do ssl/tls.

[simonrob]

That's incorrect - the proxy can encrypt any local connection if you provide your own certificate (via, e.g., mkcert).

What you are referring to is STARTTLS, which is a way of establishing a secure connection with an email server from an initially insecure connection. The proxy handles this for you.

[w5pny]

Chasing down that AADSTS7000218 error on Google led me to the changes in the registration that finally worked. I had to turn on public access and add a token type... Thanks for your help! !