search

Foundry376 / Mailspring-Sync

GNU General Public License v3.0
124 stars 33 forks source link

ProtonMail Bridge IMAP integration #3

Closed bl00mber closed 2 years ago

bl00mber commented 3 years ago

resolve Foundry376/Mailspring#429 https://community.getmailspring.com/t/protonmail-bridge-imap/485 Foundry376/Mailspring#2323

CodeMouse92 commented 3 years ago

WHAT!? Awesoooooooooome! Hey @bengotow, check this out. (I haven't reviewed it yet)

bengotow commented 3 years ago

Hey @bl00mber thanks for submitting this! It looks great - was this really the only blocker to Protonmail support? I didn't realize it was hung up on the folder hierarchy + delimiters! I looked into this back when Protonmail Bridge launched and was seeing some strange IMAP command parsing errors - at the time it seemed like their bridge was sending slightly nonstandard IMAP responses.

Will take a more detailed look at this tomorrow! 🙌

arch-btw commented 3 years ago

Thank you so much @bl00mber ! Looks great.

CodeMouse92 commented 3 years ago

@bl00mber Do you know if this will resolve either of these related bugs?

https://community.getmailspring.com/t/protonmail-new-emails-move-automatically-between-archive-and-inbox/210

https://community.getmailspring.com/t/protonmail-all-emails-disappear-from-inbox-and-reappear-then-disappear-again/309

bl00mber commented 3 years ago

@CodeMouse92 I checked these today, seems they are needed to be debugged separately so I'm converting those PRs to drafts while I'm trying to do that.

bl00mber commented 3 years ago

I have fixed this: https://community.getmailspring.com/t/protonmail-all-emails-disappear-from-inbox-and-reappear-then-disappear-again/309 As for the second one, wasn't able to reproduce. May be this also fixing it.

gkaskonas commented 3 years ago

Can this be merged?

K-MeLeOn commented 3 years ago

All my protonmail mail disappear like this : https://community.getmailspring.com/t/protonmail-all-emails-disappear-from-inbox-and-reappear-then-disappear-again/309/13

This correction has not been merged yet ?

K-MeLeOn commented 3 years ago

@bengotow Can you ping anyone who are able to review this update ?

armaccloud commented 3 years ago

Hi @bengotow, Is there something I can do to move this forward? I am not a programmer, but I have a protonmail account, so could test this version if I get instructions on how to install mailspring with this fix included (Ubuntu 20.10)

K-MeLeOn commented 3 years ago

Hello, unfortunately, the mails disappear after more time, but disappear anyway. (1.9.1-30ef802f)

bl00mber commented 3 years ago

@K-MeLeOn stop spamming, don't you see that the PR is not merged? If you do not like the fact that the project owner do not want to review this PR for now and seemingly do not even notified of comments here - you can try to convince him to review it, email is placed in his profile.

hallh commented 3 years ago

I spent quite some time building and testing this today, combined with the related PR on the Mailspring repo. I am only able to have the Snoozed directory created on Protonmail, but the email won't sync (see attached log below).

Also, while running the patched versions, I found that the Gmail OAuth connections become unstable as well. In this specific case, I had those set up before switching branches and replacing the patched libs, and the connection became unstable after launching the patched version. I'm unsure whether there's a direct connection to this patch or not, but it's worth mentioning.

Log mailspring-error.txt

bl00mber commented 3 years ago

@hallh I do not have such error. Have you deleted your local database before testing? I did not but you can try to.

hallh commented 3 years ago

@bl00mber I haven't tried to delete the database, but I can attempt this in the weekend and see if it helps.

Not sure if this belongs here, or whether I should raise a new issue, but since PM support was recently released as of 1.9.2-6e14dad1, I did the update and have been using it daily since.

The folders are not showing up for the PM inbox on the left-side menu, though mail is coming through. However, the contents of the PM inbox seems to appear and disappear sporadically.

I can force a sync and everything will show up, but if I switch between inboxes, the contents of the inbox may disappear again.

This only affects the PM inbox, Gmail and others are unaffected.

The logs are extensive, any suggestions for what I might be looking for to explain/fix this?

mike-lloyd03 commented 3 years ago

@bengotow Respectfully, can we get this PR merged? Mailspring is by far the best Linux email client but the disappearing/reappearing Proton Mail issue is the thing keeping me from using it full time.

JonathanTreffler commented 2 years ago

Can we PLEASE get this merged?

bl00mber commented 2 years ago

Here is a Linux build of patched version. @armaccloud @mike-lloyd03 @alexcanepa @jamesgecko please confirm that it works. You need to replace Mailspring.app/Contents/Resources/app.asar.unpacked/mailsync 2_Dec_mailsync.zip

bengotow commented 2 years ago

Hey folks! Ahh my gosh, I thought that this had merged alongside the other one but I guess not. Let's ship this right now 👍

mike-lloyd03 commented 2 years ago

Thanks so much!

gkaskonas commented 2 years ago

Finally!

bengotow commented 2 years ago

Hey folks! Quick update here - I merged this and made one small change, moving the "if folder is All Mail, skip" check up to the parent function. Gmail also has an "All Mail" folder, so we need to scope the if statement a bit more to avoid breaking anything I think. (5d9bbd42a77961ed3205284da7476f587cd23065)

I merged a few other open PRs and ran a 1.9.3 build - want to give this a spin and make sure it looks good? Also going to test that the "no container folder settings" case and sync with other providers is unchanged.

If it's all good I'll publish this update tomorrow. So sorry for the hassle here guys, I had forgotten this PR was split across both repos.

https://mailspring-builds.s3.amazonaws.com/client/ac79e444/osx/Mailspring.zip https://mailspring-builds.s3.amazonaws.com/client/ac79e444/win-ia32/MailspringSetup.exe https://mailspring-builds.s3.amazonaws.com/client/ac79e444/linux/mailspring-1.9.3-0.1.x86_64.rpm https://mailspring-builds.s3.amazonaws.com/client/ac79e444/linux/mailspring-1.9.3-amd64.deb

K-MeLeOn commented 2 years ago

@bengotow Access denied on darwin and win32 :)

bengotow commented 2 years ago

Ahh thanks @K-MeLeOn - fixed them above!

K-MeLeOn commented 2 years ago

@bengotow top! Thanks, I'm using CanaryMail while waiting for a stable connectivity with Proton. I will try your new version :)

EDIT: for the moment I only found one issue, my protonmail boxes are on scanning everytime (see image bellow)

scanning
ipbody commented 2 years ago

I have the exact same problem with the snap version (lastest edge) on ubuntu 20.04 2021-11-24_13-14 How can i get logs of this?

armaccloud commented 2 years ago

For me the same, I did a fresh installation of Pop!_OS 21.04 & above Mailspring deb-file, using the Protonmail Bridge. Mailspring was using high CPU for a while syncing the different folders, now it only remains for the "All Mail" that keeps scanning.

image

bl00mber commented 2 years ago

See comment I dont think it is actually scanning, just showing status.

armaccloud commented 2 years ago

@bl00mber I am curious whether this is the case, as I get a continuous 40%-65% CPU usage from Mailspring while it doesn't seem to do anything else.

image

When I open the process, it shows the following. The Waiting Channel changes and cycles between

image

ADDITION: To test, I have removed the protonmail account from Mailspring, then added another (non-protonmail account): the CPU issue is not occurring. Added back the protonmail account and the CPU issue is back.

Interesting note: The CPU only occurs when Mailspring is shown on-screen. Not when it is minimized.

bl00mber commented 2 years ago

@armaccloud here is a Linux x86-64 build with change from this comment. Currently I do not have PM subscription. Check this out, is All Mail folder still scanning on this version?

bengotow commented 2 years ago

Hey folks! I think that the "All Mail" folder is showing as "scanning" because we're intentionally skipping that folder during sync (since it contains a copy of the messages in all the other folders and syncing it causes them to flicker back and forth in mailspring), but we're not updating the status.

I think maybe we could hide that folder entirely? It might be that sync status < 100% is also causing the sync process to think that it needs to keep syncing as fast as it can rather than falling into an idle state. I'll take a look and see.

armaccloud commented 2 years ago

@armaccloud here is a Linux x86-64 build with change from this comment. Currently I do not have PM subscription. Check this out, is All Mail folder still scanning on this version?

@bl00mber Can you give me directions on how to install this? I am on Linux, the version I now have installed is the .deb file from here. Your .zip contains a file called mailsync. What should I do with this file?

bengotow commented 2 years ago

Hey folks - could you give this build a spin and see if it addresses the All Mail "syncing" issue and also the heavy CPU use? I think that this should correctly mark the folder as "sync finished", which will also let Mailsync switch to it's idle mode.

One thing I'm curious about - the All mail folder is still visible and moving mail to All Mail might do weird things, since we're not actually syncing the folder. Would be curious to know if it'd be better to just hide the folder entirely, or if this works pretty well?

https://mailspring-builds.s3.amazonaws.com/client/9ee21cf2/osx/Mailspring.zip https://mailspring-builds.s3.amazonaws.com/client/9ee21cf2/linux/mailspring-1.9.3-0.1.x86_64.rpm https://mailspring-builds.s3.amazonaws.com/client/9ee21cf2/linux/mailspring-1.9.3-amd64.deb

ipbody commented 2 years ago

Hey folks - could you give this build a spin and see if it addresses the All Mail "syncing" issue and also the heavy CPU use? I think that this should correctly mark the folder as "sync finished", which will also let Mailsync switch to it's idle mode.

One thing I'm curious about - the All mail folder is still visible and moving mail to All Mail might do weird things, since we're not actually syncing the folder. Would be curious to know if it'd be better to just hide the folder entirely, or if this works pretty well?

https://mailspring-builds.s3.amazonaws.com/client/9ee21cf2/osx/Mailspring.zip https://mailspring-builds.s3.amazonaws.com/client/9ee21cf2/linux/mailspring-1.9.3-0.1.x86_64.rpm https://mailspring-builds.s3.amazonaws.com/client/9ee21cf2/linux/mailspring-1.9.3-amd64.deb

Using the .deb file i have the same problem, the All Mail folder is still scanning for ever, no progress (%) is showed

armaccloud commented 2 years ago

Hi @bengotow The same result for me as for @ipbody. I followed the steps.

@bl00mber I just found the folder you were referring to in your post to replace the mailsync file. Will test.

armaccloud commented 2 years ago

Here is a Linux build of patched version. @armaccloud @mike-lloyd03 @alexcanepa @jamesgecko please confirm that it works. You need to replace Mailspring.app/Contents/Resources/app.asar.unpacked/mailsync The size difference is because it bundles libs with it, or because some another script that I do not found is used to build prod version. 26_Nov_mailsync.zip

I just tested the mailsync file. The same issues are occuring for me (high CPU & All Mail Scanning = visible). What I did was

image

bl00mber commented 2 years ago

@armaccloud see updated comments, I reproduced it without an account and seemingly fixed. You can test updated version 27_Nov from the comment above.

armaccloud commented 2 years ago

@armaccloud see updated comments, I reproduced it without an account and seemingly fixed.

Ah, good news & congrats! Let me know if you need me to test anything.

armaccloud commented 2 years ago

@armaccloud see updated comments, I reproduced it without an account and seemingly fixed. You can test updated version 27_Nov from the comment above.

I have tested the configuration. The scanning issue is solved, including the high cpu!

When trying to send a message, I get an authentication error on SMTP, even though I am using the provided password from the bridge (double-checked) and the settings are good.

server: 127.0.0.1 port: 1025 security: STARTTLS allow insecure ssl: ticked username: as provided in the bridge settings password: as provided in the bridge settings

I see in the logs that it defaults to plain text for authentication (which is ok I guess, since it happens locally?) When I untick allow insecure ssl, I get a (self-signed) certificate error

What could be the issue here? IMAP works fine (same uname & passwd)

Error log for SMTP

----------SMTP---------- connect 127.0.0.1 1025 220 127.0.0.1 ESMTP Service Ready init EHLO pop-os 250-Hello pop-os 250-PIPELINING 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-CHUNKING 250-STARTTLS 250 AUTH LOGIN PLAIN start TLS STARTTLS 220 2.0.0 Ready to start TLS done init after starttls EHLO pop-os 250-Hello pop-os 250-PIPELINING 250-8BITMIME 250-ENHANCEDSTATUSCODES 250-CHUNKING 250 AUTH PLAIN LOGIN

SASL_PATH:

SMTP Last Response Code: 250 SMTP Last Response: Hello pop-os PIPELINING 8BITMIME ENHANCEDSTATUSCODES CHUNKING AUTH PLAIN LOGIN

mailsmtp Last Error Code: 296`

bl00mber commented 2 years ago

@armaccloud may be related to this Is SMTP works on account from other mail providers? (In my case works) Is SMTP works on version that Ben attached here? (with scanning issue) What is the log file you are copying from?

ipbody commented 2 years ago

@armaccloud see updated comments, I reproduced it without an account and seemingly fixed. You can test updated version 27_Nov from the comment above.

Is just copying the mailsync file replacing the old one? when i do that, it shows me a database error that i can get rid of. Do i have to use the stable mailspring version or the beta one from up there

armaccloud commented 2 years ago

@ipbody The version .deb. you refer to is the one I used as well. These are the steps I took to make it work.

armaccloud commented 2 years ago

@armaccloud may be related to this Is SMTP works on account from other mail providers? (In my case works) Is SMTP works on version that Ben attached here? (with scanning issue) What is the log file you are copying from?

Hi @bl00mber These are the steps I took to answer your questions.

Steps to Reproduce

To replace the mailsync file, took the following steps

So the issue specifically occurs after replacing the mailsync file.

To retrieve the logfile, I go to

Screenshots

image image image

bl00mber commented 2 years ago

@armaccloud thanks for explanation! Attached above 2_Dec exec build from latest commit. It is 17kb larger, maybe it helps. But I don't think it will solve the issue. It still smaller than a Ben version by 54kb... this may be something build-related, I need to investigate. For some reason SMTP works on my system.

armaccloud commented 2 years ago

@armaccloud thanks for explanation! Attached above 2_Dec exec build from latest commit. It is 17kb larger, maybe it helps. But I don't think it will solve the issue. It still smaller than a Ben version by 54kb... this may be something build-related, I need to investigate. For some reason SMTP works on my system.

I have just tried out this new 2 Dec build, but the SMTP issue is still occurring. Let me know if there are troubleshooting steps you want me to try.

armaccloud commented 2 years ago

Hi @bl00mber, I am wondering whether there is anything I can do to help solve this issue? It is occurring on a fresh install of both the operating system and Mailspring and the Protonmail Bridge, so I suspect I will not be the only one experiencing this. Are there any logs I can send you?

bl00mber commented 2 years ago

@bengotow can you build this change on your setup?

bl00mber commented 2 years ago

@armaccloud try to build it on your own machine.

git clone -b bl00mber-patch-1 git@github.com:bl00mber/Mailspring-Sync.git
cd <folder>
./build.sh

You may get some errors, try to debug them with web first.

fuzzylogicltd commented 2 years ago

I've just installed 1.10.0 and I'm having the "emails dissapear and reappear issue" as descibed here. Did anyone figure out a solution for this?

oktayacikalin commented 2 years ago

Please, guys, try to fix this. This disappear and reappear issue really annoys me. Thanks!