search

mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.97k stars 1.18k forks source link

Android Gmail App - ActiveSync Mails disappearing #4270

Closed AzonInc closed 2 years ago

AzonInc commented 3 years ago

Prior to placing the issue, please check following: (fill out each checkbox with an X once done)

Summary

Whenever I setup a Mailaccount on the Gmail Android App via Exchange ActiveSync it's only showing a few of the last Mails in the inbox. Everything else starts to disappear. But when I setup the Account as IMAP everything works fine. However, when I search for mails, they are showing up in the results. So I'm not sure whether this is a bug or not.

I set the sync period to "All" Mails but it didn't fix the Problem. In fact I tried several Settings Changes of Gmail itself but no noticable change.

Mails that disappeard within one day: image

Logs

13.09.2021, 14:04:57 notice [58]: [ERROR] <0x0x562bea822000[WOHttpTransaction]> client disconnected during delivery of response for <WORequest[0x0x562bea9ee710]: method=POST uri=/SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=**********&DeviceId=androidc167&DeviceType=Android app=SOGo rqKey=Microsoft-Server-ActiveSync rqPath=(null)> (len=13): the socket was shutdown 13.09.2021, 14:04:57 notice [62]: 188.xxx.xxx.xxx "POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=**********&DeviceId=androidc167&DeviceType=Android HTTP/1.0" 200 13/225 45.289 - - 0 - 13 13.09.2021, 14:04:57 notice [62]: [ERROR] <0x0x562bea888390[WOHttpTransaction]> client disconnected during delivery of response for <WORequest[0x0x562bea8233a0]: method=POST uri=/SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=**********&DeviceId=androidc167&DeviceType=Android app=SOGo rqKey=Microsoft-Server-ActiveSync rqPath=(null)> (len=13): the socket was shutdown 13.09.2021, 14:04:47 notice [70]: 188.xxx.xxx.xxx "POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=**********&DeviceId=androidc167&DeviceType=Android HTTP/1.0" 200 13/225 90.552 - - 0 - 13 13.09.2021, 14:04:47 notice [70]: [ERROR] <0x0x562be9c3bf00[WOHttpTransaction]> client disconnected during delivery of response for <WORequest[0x0x562beaa36b50]: method=POST uri=/SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=**********&DeviceId=androidc167&DeviceType=Android app=SOGo rqKey=Microsoft-Server-ActiveSync rqPath=(null)> (len=13): the socket was shutdown 13.09.2021, 14:04:18 notice [65]: <0x0x562bea5317f0[SOGoActiveSyncDispatcher]> Sleeping 45 seconds while detecting changes for user ********* in Ping... 13.09.2021, 14:04:17 notice [65]: 188.xxx.xxx.xxx "POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Search&User=**********&DeviceId=androidc167&DeviceType=Android HTTP/1.0" 200 1064/77 0.387 - - 112K - 13

Reproduction

  1. Setup a new Exchange ActiveSync Account on Gmail App for Android
  2. Mails Disappear after some time

System information

Question Answer
My operating system Ubuntu 20.04
Is Apparmor, SELinux or similar active? No
Virtualization technlogy (KVM, VMware, Xen, etc - LXC and OpenVZ are not supported None
Server/VM specifications (Memory, CPU Cores) 32GB RAM, 4 Cores
Docker Version (docker version) 20.10.8, build 3967b7d
Docker-Compose Version (docker-compose version) 1.29.2, build 5becea4c
Reverse proxy (custom solution) None
jangatzke commented 3 years ago

I do not know if it is the same problem, but I am facing missing mails in the gmail App, too. Just added a new account to my phone. Inbox was empty in gmail, but filled with mails in SOGo. Sent items however display all mails correctly. New mails, that arrived after the setup of the account are displayed correctly. I have set the period to all, too.

romprod commented 3 years ago

Tbf I've had this issue for quite a while with the official Outlook android app but started using the BlueMail android app which appeared to resolve it. Now it has come back again and neither app shows the full list.

I can connect both apps to an o365 Exchange account and they show all emails as expected.

dellarte commented 3 years ago

I discovered mailcow-dockerized last week and decided to install it on my server. I'm facing the same problem. I reinstalled 2 times mailcow-dockerized from scratch.

I can see all the emails in SoGo. IMAP sync is working on BlueMail and Outlook for Android But when i try to setup my account as an Exchange Account (on the same apps), only 3 of my emails are showing up and then start disepearing until there are no more emails. All folders/sub folders are empty, FYI.

I also tested to set the var 'useEASforOutlook' to 'yes' and make Desktop Outlook "works" with ActiveSync but it seemed to load endlessly...

BTW, both Exchange ActiveSync and Outlook Connectivity on https://testconnectivity.microsoft.com/tests/exchange fails

Edit: I'm syncing email from France, the server is located in a DC at Helsinki and the output of the timedatectl status command is the following :

               Local time: Wed 2021-09-22 17:42:38 CEST
           Universal time: Wed 2021-09-22 15:42:38 UTC
                 RTC time: Wed 2021-09-22 15:42:39
                Time zone: Europe/Paris (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

Did I configured TZ wrongly ?

EDIT 2:

I would add some details. I tested to set up a mailcow account with EAS within 4 different android apps and I always get the same behavior. So I don't think it's a Gmail related problem. Firstly, I configure the account in the app without any problem and few emails appear in the inbox (not all). Next I define the sync period from 1 or 2 weeks to unlimited. Suddenly all emails appear. Even in folders. Then, few seconds later, everything is gone...

plantroon commented 3 years ago

This started happening for me too. I noticed it today/yesterday. I even reset the gmail app and resynced everything but now I have missing mails in another folder. Originally I couldn't sync "Sent" folder, now I cannot sync Inbox. I think this has to do with Gmail app on Android. SOGO webmail and IMAP clients are unaffected. I know that it worked fine till recently, as I would have noticed the missing mails (most likely). I blame the recent updates to the Gmail app, they may have changed something other than the fancy Material you theming. And I wouldn't put it past Google if this was broken on purpose.

Other notes:

Update: I tried another phone with Android 11 and Gmail app and the same behavior, with the same mails missing. I then moved these mails to another folder from SOGO webui and then moved them back and then they appeared in Gmail app. I have no idea what changed though.

It would be great if someone with Exchange account that is not on mailcow could confirm a similar issue or any of the previous findings. This started happening only recently.

plantroon commented 3 years ago

Further observations: I tried a very old phone with Android 6 and its stock Gmail app and I am getting the same behavior. I am not sure how to debug SOGo, I tried comparing debug logs of working (Samsung Mail EAS) and non-working (Gmail) apps and I noticed no difference in the logs output. In the Gmail app messages since 17th of August till 20th of September are missing. Both phones with the gmail app have the same problem, with exactly the same messages missing. I know that these messages were there 10 days ago.

milkmaker commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

dellarte commented 2 years ago

still not resolved

natoleet commented 2 years ago

I get the same problem on my galaxy s20 ultra. Tried both Gmail and outlook.

MrKrisKrisu commented 2 years ago

Have you set the synchronization to "All" in the settings? By default, the Gmail app only syncs the last week on ActiveSync.

image image

AzonInc commented 2 years ago

Yeah already tried that but no success.

milkmaker commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

phil-bot commented 2 years ago

I have the same problem with a Pixel 4a.

With the native Gmail app and the Outlook app.

ElMischa commented 2 years ago

I have the same problem with a Samsung S21 and A52s using EAS With Gmail and Outlook app mails in inbox disappeared as soon as the first subfolder got synched. With Samsung Mail mails I deleted on an other device, did not disappear, even though they are deleted in SOGo.

I now use IMAP in Samsung Mail (which supports imap idle) and DavX5 for syncing calendar and contacts. That works fine, but SOGo and mobile EAS clients is horrible .. Outlook on Windows works fine btw.

plantroon commented 2 years ago

One important question to people experiencing this: do you use any proxy? Nginx, traeffic, haproxy? As that seems to be the culprit.

I am not sure about fixing the other two, but here's what seems to fix it for Nginx (I applied this only last week after running into #4537 after upgrade, so I am not sure if it works long-term):

server {
...
        proxy_request_buffering off;
        proxy_buffering off;
        proxy_buffer_size 128k;
        proxy_buffers 64 512k;
        proxy_busy_buffers_size 512k;
        large_client_header_buffers 2 50k;
        client_max_body_size 0;
...

Ideally put this in server block, not all of the directives work per location context. Whatever your preferred system is. Also read up on the directives in Nginx manual, to not break something else in your config, but it should be fine.

Probably EAS isn't that robust and won't report any problems on either server or client side when something gets "stripped" from its responses? For now my mails are working through EAS. I was using IMAP until a week ago as a workaround.

Mailcow already sets up nginx proxy to Sogo as can be seen in config files in: /data/conf/nginx - so any further proxy in front of it should be configured similarly. It's not the best setup but I guess this was not how Mailcow as a standalone AIO solution was intended to be used.

plantroon commented 2 years ago

The above was tested on Samsung Mail app's EAS implementation. Everything including mails, calendars, contacts continues to work properly to this day (at least I have not noticed any errors). The app is fine but I do not like it as much as Gmail.

However, Gmail continues to be broken. Here among other things, calendar syncs only last 2 weeks of events (not precisely, just an estimate). I do not understand how this can be happening.

To completely eliminate that it is my proxy causing this, I exposed my mailcow installation directly to the internet and nothing has changed.

Before every testing step, I cleared EAS cache in Sogo and removed + readded the EAS account. I tested this on Android 6 (up to date Gmail app) and Android 12 (Gmail and Samsung Mail apps). Behavior is the same.

I want to use EAS but this is unreliable as hell xD the thing is, that the e-mails or calendar events missing cannot be noticed on purpose. There are no errors reported anywhere, the mails/events are just missing on the EAS client. I only accidentally notice 2-3 weeks later that I missed 2-3 important mails or that I can't see old calendar events when I am looking at Sogo webUI or IMAP/caldav connected client :(

Is there anyone who could test this with me or who has alternative EAS implementations available (from work or similar)? Sogo is simply broken for me.

ElMischa commented 2 years ago

In my setup I use apache as reverse proxy according to the docs. I have the same problems (disappearing mails, incomplete list of contacts). I fixed that by using the Samsung Mail App (which provides IMAP idle, what GMail does not) and DavX5 for calendar and addressbook, what results in NOT using EAS. I'd rather use EAS if it would work, so .. did anyone successfully test the complete setup I'm using and has an idea how to fix that behaviour?

plantroon commented 2 years ago

In my setup I use apache as reverse proxy according to the docs. I have the same problems (disappearing mails, incomplete list of contacts). I fixed that by using the Samsung Mail App (which provides IMAP idle, what GMail does not) and DavX5 for calendar and addressbook, what results in NOT using EAS. I'd rather use EAS if it would work, so .. did anyone successfully test the complete setup I'm using and has an idea how to fix that behaviour?

Well, as I said above, try without your proxy or a properly configured Nginx proxy (as is in the docs or above) and use Samsung Mail (it supports EAS and works very well) - for me this combination works perfectly. Only with Gmail all is still broken - mails go missing, calendar only syncs up to last month.

I asked a friend who uses my very same mailcow instance with iOS and macOS and reports no problems with EAS. He has a company's MS 1st party EAS synced up as well. So it really is down to that Gmail app, but there must be some way to make it work from server-side imo as other services aren't broken. Even going as far as to make per-client exceptions based on user agent, I don't know who to blame as the client implementations seem really poor with the issues I read about MS Outlook and Android apps.

ElMischa commented 2 years ago

Thanks for you reply .. I forgot to mention, that with Samsung Mail EAS works only partially in my setup. Email and calender were OK, but none of my contacts got synced to my smartphone using Samsung Mail - that was the reason, I switched to DavX5. MS Outlook (M365) on Win10 is almost working fine - I only have to force syncing all folders using "Send / Receive all folders" once after startup.

milkmaker commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

plantroon commented 2 years ago

I cannot replicate these issues with current versions of SOGo in mailcow. If this happens again, I'll report it directly to SOGo. It would be nice if anyone else confirmed but in the meantime everyone probably gave up on EAS.

huncksas commented 2 years ago

Hey plantroon, i just recently installed the latest version of MailCow behind a nginx proxy (setup was done as described in the docs) and experienced the same problems as all others. One of my last tries was to add the above mentioned nginx - parameters into both server instances for port 80 and 443. I tried GMail and BlueMail and both did not work out.Before the changes

So now i tried it with the Samsung Mail and for now it looks good - at least i can not experience any probs within the last 30 minutes.

plantroon commented 2 years ago

Hey plantroon, i just recently installed the latest version of MailCow behind a nginx proxy (setup was done as described in the docs) and experienced the same problems as all others. One of my last tries was to add the above mentioned nginx - parameters into both server instances for port 80 and 443. I tried GMail and BlueMail and both did not work out.Before the changes

So now i tried it with the Samsung Mail and for now it looks good - at least i can not experience any probs within the last 30 minutes.

Just wait, Samsung Mail went bad for me later too. I stopped using EAS, it's broken and unreliable, the problems just keep coming out of nowhere. A lot of it can be due to proxy but even the default mailcow setup is broken. :( Latest EAS problem I experienced was that it was breaking my attachments. It never ends.

luitzifa commented 1 year ago

I installed mailcow in february without a reverse proxy and i'm experiencing this problem too with android 13 and gmail app.

romprod commented 1 year ago

FWIW Im now using the cloud service via servercow.de and the problem stil exists.

As soon as I can find an alternate service ill be moving away. Nobody seems to be bothered to fix it

plantroon commented 1 year ago

I literally didn't have this happen in the past 3 months lol. But yea, that could be luck.

All I really want is some email over http, same way commercial solutions like zoho and gmail do it. This is quite useful because a lot of public wifis block imap and smtp submission. I don't know why but they do.

I don't care about proprietary EAS with quirks that are apparently hard to replicate in FOSS products.

CalDAV works, CardDAV works, maybe we need MailDAV? :D

biosmatrix123 commented 1 year ago

Also experiencing this issue without using a proxy!

plantroon commented 1 year ago

I am curious, do they at least show up in search?

Also try looking through your Inbox manually too, to see if the messages could simply be in the wrong order.

JoshMout commented 1 year ago

I'm having this issue too but they appear if I search for them.