Old mails deleted on other clients still show up as unread in Mail, when opened "Not found"

[jancborchardt]

Steps to reproduce

1. Use Mail a bunch as your main Mail app
2. Also use another mail client e.g. something on mobile
3. At some point see outdated mails in Mail, even indicated as unread

Expected behavior

No non-existent mails should show up in Mail, especially not indicated as unread. If the preview line can not be fetched, maybe it’s better to not show it in the message list to begin with? (It could still be loaded in later if the mail is retrieved.)

Actual behavior

The 3 mails of upgrade, activity, and invitation are actually deleted. They show up in the list without preview text line, and indicated as unread. When opening, they show "Not found":

Mail app version

Current on cloud.nc.com

Mailserver or service

No response

Operating system

No response

PHP engine version

None

Web server

None

Database

None

Additional info

No response

[cnaumer]

For the memcache.local I use APCu and Redis for the rest.

[yankees5963]

Nextcloud docs recommended APCu for local and Redis for the rest. If the mail app doesn't work with that then the mail app needs to be fixed. Hard to believe when background tasks work for everything else it's not a mail issue.

[rakekniven]

Nextcloud docs recommended APCu for local and Redis for the rest. If the mail app doesn't work with that then the mail app needs to be fixed. Hard to believe when background tasks work for everything else it's not a mail issue.

Please fix your tone and help fixing mail app instead of complaining.

[powerpaul17]

I have Redis for distributed and locking and APCu for local caching. I changed all to Redis to see if the problem persists.

[Githopp192]

That's my config:

'memcache.distributed' => '\OC\Memcache\Redis', 'memcache.locking' => '\OC\Memcache\Redis', 'memcache.local' => '\OC\Memcache\APCu', 'filelocking.enabled' => true, 'redis' => array ( 'host' => '/run/redis/redis.sock', 'port' => xxx, 'dbindex' => xxx, 'timeout' => xxx, 'password' => 'xxx', ),

[powerpaul17]

Unfortunately I must report that even using Redis the problem still persists.

[AlexanderZuehr]

We are also using nextcloud 27.1.1 & mail app version 3.4.0 and redis caching and having this problem.

[nicksmith53]

This appears fixed with mail app 3.4.1 (nextcloud 27.1.1).

To clear old messages that were already stuck, I had to enable debug and hit the clear cache button. And now messages correctly disappear from the web app when they are deleted or moved from outlook.

[meichthys]

This doesn't seem to be fixed for me in 3.4.1 (NC 27.1.1). Using Redis for all caching (Nextcloud VM)

[dcgsteve]

Aha - for me I think the latest version works fine ... although I can't 100% tell as I deleted the app and re-installed, as opposed to upgraded. Either way, I can now delete an email using a different client, and I can see Mail sync'ing correctly again, which it didn't do before.

So hopefully it is fixed for others as well :)

(Thanks Mail team!!!)

[ChristophWurst]

Thanks for the reports everyone.

I have to refine my findings. The Mail app uses a distributed cache for IMAP. Only for instances with no distributed cache configured Nextcloud will use the local one.

In other words for instances with APCu for local and Redis for distributed cache, Redis will still be used.

From the feedback above I take that none of you only run APCu. The AIO container and the VM use Redis.

[meichthys]

I deleted the app and re-installed

@dcgsteve could you clarify this? Did you remove the so via the web interface then reinstall, or did you delete the app directory from the terminal? Also do you have debugging turned on?

[dcgsteve]

@meichthys - everything was done through the GUI, without debug being turned on - hope that helps!

[meichthys]

@dcgsteve Thanks. Unfortunately my instance is still displaying the issue after disabling debug and removing/re-adding the Mail app. I'm curious if it's fixed for @nicksmith53 @AlexanderZuehr @powerpaul17 and @Githopp192 ?

[AlexanderZuehr]

I was able to solve the problem multiple times with multiple users by using the debug and clear cache method. (mentioned above) This evening I am going to update mail to stable 3.4.1 and tomorrow I'll test if the problem still occurs by using different mail clients.

[meichthys]

@AlexanderZuehr Thanks for the feedback. I'm also able to workaround the issue with the debug + clear cache method.

[AlexanderZuehr]

Hey all, I've updated nextcloud to 27.1.2 and mail-app to 3.4.1 stable. After the update I used two other mail clients to delete mails. The other clients both recognised the deleted mails and updated accordingly but nextcloud showed the old deleted mails with an error.

Some of the mails fixed themself and are deleted accordingly in nextcloud, but 1 email is stuck and showing that it is not found. Furthermore a second deleted mail should not exist anymore but I am able to read and interact with it. Normally this mail would also show up with an error that it is not found.

All other mail clients don't have this problem.

I think this problem still exists.

I will add a screenshot and a log.

Screenshot:

Log: [mail] Fehler: OCA\Mail\Exception\ServiceException: Could not load message at <>

0. /var/www/html/apps/mail/lib/Controller/MessagesController.php line 220 OCA\Mail\Service\MailManager->getImapMessage()
1. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 230 OCA\Mail\Controller\MessagesController->getBody()
2. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 137 OC\AppFramework\Http\Dispatcher->executeController()
3. /var/www/html/lib/private/AppFramework/App.php line 183 OC\AppFramework\Http\Dispatcher->dispatch()
4. /var/www/html/lib/private/Route/Router.php line 315 OC\AppFramework\App::main()
5. /var/www/html/lib/base.php line 1068 OC\Route\Router->match()
6. /var/www/html/index.php line 36 OC::handleRequest()

Caused by:

OCP\AppFramework\Db\DoesNotExistException: Message does not exist at <>

0. /var/www/html/apps/mail/lib/Service/MailManager.php line 192 OCA\Mail\IMAP\MessageMapper->find()
1. /var/www/html/apps/mail/lib/Controller/MessagesController.php line 220 OCA\Mail\Service\MailManager->getImapMessage()
2. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 230 OCA\Mail\Controller\MessagesController->getBody()
3. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 137 OC\AppFramework\Http\Dispatcher->executeController()
4. /var/www/html/lib/private/AppFramework/App.php line 183 OC\AppFramework\Http\Dispatcher->dispatch()
5. /var/www/html/lib/private/Route/Router.php line 315 OC\AppFramework\App::main()
6. /var/www/html/lib/base.php line 1068 OC\Route\Router->match()
7. /var/www/html/index.php line 36 OC::handleRequest()

GET /apps/mail/api/messages/1519771/body from 172.17.51.100 by alz at 2023-10-11T17:28:53+00:00

[dcgsteve]

Ok, with no changes, it's suddenly stopped sync'ing again after 2 (?) days with odd "Not Found" emails now coming up again .... bah :(

[meichthys]

Just updating to clarify that the issue is still present on the latest v3.4.2

[dcgsteve]

Just updating to clarify that the issue is still present on the latest v3.4.2

Yes, confirmed here too - I updated to v3.4.2 before I posted above - I should have said :)

[ChristophWurst]

None of the patch release had fixed for this bug. The situation is expected to be identical to v3.4.0

[dcgsteve]

None of the patch release had fixed for this bug. The situation is expected to be identical to v3.4.0

Understood @ChristophWurst - thanks for confirmation.

Unfortunately though, for me anyway (as I need a working web mail system quickly and I don't have the skills to dive in to code on this), I'll have to migrate off to another product. I hope you manage to find out what is happening here and it's an easy fix - good luck all!

[Githopp192]

updated, too - to v.3.4.2 - bug seems to be still there (i did not clear the cache manually)

[nicksmith53]

Guess I spoke too soon. Everything was behaving for like a week, after cache clear and update. But now I once again have deleted messages still showing in the message list and "Not found" when trying to open them.

[pneumokoniose]

I don't know if it helps:

If I delete emails via Thunderbird, I can often still access them via Nextcloud Mail. Without an error message.

I have root access to my email server (dovecot). I looked on the server to see what was actually in my mailbox: They were exactly the emails that Nextcloud showed me.

The emails deleted via Thunderbird have the file extension “.STa”. The others ".Sa". My guess: If an email is deleted via IMAP, it will first be marked as deleted on the server. But it still remains in the mailbox. It will probably be deleted later by a cron job on the server.

The question is: Why does Nextcloud Mail keep showing these emails marked as deleted? This is not the case in Outlook, Thunderbird and K-9 Mail.

If I manually delete one of these emails marked as deleted from the server, it also disappears from Nextcloud Mail.

I suspect that these are exactly the emails where I will later get the “Not found” message. But that's just a guess. I haven't been able to reproduce that yet.

[hunterdrayman]

I'm trying to follow the debug/clear cache method mentioned previously but I'm unclear as to where the clear cache button is. Is it in the mail app itself, the nextcloud settings or is it the usual browser clear cache button? Do I need to log on as the nextcloud admin account to find this? Thanks.

[meichthys]

@hunterdrayman You must enable debug mode first by setting 'debug' => true in your config.php Then you'll see the clear cache option in the ... menu of your inbox:

[hunterdrayman]

Thanks, that seems to have fixed it.

[meichthys]

Yes, it's a temporary workaround for an issue that still needs to be fixed. Also, remember that Debug mode should not be enabled long-term in Production environments as per the Nextcloud Docs, so use at your own risk: https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html#debug-mode

[ChristophWurst]

This should be fixed with https://github.com/nextcloud/mail/pull/8968. Testing appreciated.

[meichthys]

I could be testing things wrong, but the changes don't seem to be working for me. Here's the steps I took to test:

1. Navigated to /var/www/nextcloud/apps/lib/Service/Sync
2. Ran wget https://raw.githubusercontent.com/nextcloud/mail/346560fb616963e1fdc0180c796d037e87306eb8/lib/Service/Sync/ImapToDbSynchronizer.php
3. Ran chmod --reference=ImapToDbSynchronizer.php ImapToDbSynchronizer.php.1 to copy permissions to new file
4. Removed the old file: rm ImapToDbSynchronizer.php
5. Moved the new file into place: mv ImapToDbSynchronizer.php.1 ImapToDbSynchronizer.php
6. Rebooted nextcloud server
7. Sent test email to my imap account
8. Confirmed test email showed up in nextcloud mail webui and on other client
9. Archived test email on other client
10. Waited 30 seconds
11. Clicked refresh icon in Nextcloud mail webui (email stays in inbox)
12. Refreshed nextcloud mail browser tab (email stays in inbox)
13. Force refreshed nextcloud mail browser tab (email stays in inbox)

When selecting the email in Nextcloud mail it displays like this:

Is there something I missed? I think I remember once someone saying I may need to wait for a cron job to run? Is that relevant in this case?

Nextcloud logs show this which seems like it could be relevant?

If there's any more information I can provide, please let me know.

[AlexanderZuehr]

I do also have no positive feedback for #8968 I use Nextcloud 27.1.3 and Mail-App 3.4.4 and patched your fix manually but the problem still reoccured.

Also it is not solving my other problem #9048

Error Log: [mail] Fehler: OCA\Mail\Exception\ServiceException: Could not load message at <>

0. /var/www/html/apps/mail/lib/Controller/MessagesController.php line 220 OCA\Mail\Service\MailManager->getImapMessage()
1. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 230 OCA\Mail\Controller\MessagesController->getBody()
2. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 137 OC\AppFramework\Http\Dispatcher->executeController()
3. /var/www/html/lib/private/AppFramework/App.php line 183 OC\AppFramework\Http\Dispatcher->dispatch()
4. /var/www/html/lib/private/Route/Router.php line 315 OC\AppFramework\App::main()
5. /var/www/html/lib/base.php line 1068 OC\Route\Router->match()
6. /var/www/html/index.php line 36 OC::handleRequest()

Caused by:

OCP\AppFramework\Db\DoesNotExistException: Message does not exist at <>

0. /var/www/html/apps/mail/lib/Service/MailManager.php line 192 OCA\Mail\IMAP\MessageMapper->find()
1. /var/www/html/apps/mail/lib/Controller/MessagesController.php line 220 OCA\Mail\Service\MailManager->getImapMessage()
2. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 230 OCA\Mail\Controller\MessagesController->getBody()
3. /var/www/html/lib/private/AppFramework/Http/Dispatcher.php line 137 OC\AppFramework\Http\Dispatcher->executeController()
4. /var/www/html/lib/private/AppFramework/App.php line 183 OC\AppFramework\Http\Dispatcher->dispatch()
5. /var/www/html/lib/private/Route/Router.php line 315 OC\AppFramework\App::main()
6. /var/www/html/lib/base.php line 1068 OC\Route\Router->match()
7. /var/www/html/index.php line 36 OC::handleRequest()

GET /apps/mail/api/messages/1963063/body from 172.17.51.100 by alz at 2023-11-29T08:03:26+00:00

[ChristophWurst]

I don't think the patch is able to fix old data. It will probably only help for messages deleted now.

[cnaumer]

It also does not work for me. Manually applied the patch. Wrote an email to myself. Email is there in Thunderbird and Nextcloud. Logged out of nextcloud. Deleted the email in Thunderbird. Logged in again in nextcloud and email is still listed but cannot be opened/found because it was deleted.

[hunterdrayman]

I installed Nextcloud 27.1.4, restarted and then deleted emails from my inbox and from a folder using the Nextcloud Snappymail app. The changes very quickly showed up in the native Nextcloud Mail app. Thanks for fixing.

[meichthys]

I've also updated to NC 27.1.4 @hunterdrayman was this previously broken with snappymail? I'm still seeing this issue with other clients, so I'm not sure if snappymail might somehow share a mailbox with nextcloud or something?

[hunterdrayman]

Yes. It was previously broken. Both clients set to use the same IMAP mailboxes and had to use the Debug/Clear cache workaround until now. Can't vouch for things like Tbird as I don't use but can't think why it wouldn't work if IMAP settings correct and consistent.

On Wed, 29 Nov 2023, 14:47 MeIchthys, @.***> wrote:

I've also updated to NC 27.1.4 @hunterdrayman https://github.com/hunterdrayman was this previously broken with snappymail? I'm still seeing this issue with other clients, so I'm not sure if snappymail might somehow share a mailbox with nextcloud or something?

— Reply to this email directly, view it on GitHub https://github.com/nextcloud/mail/issues/8223#issuecomment-1832036841, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARIIIZ2ZQZ2KWFCD5CEX4FTYG5DIDAVCNFSM6AAAAAAV2FWIKKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZSGAZTMOBUGE . You are receiving this because you were mentioned.Message ID: @.***>

[CashAMatic]

The fix works for my setup. Nextcloud 27.1.4 with Cyrus IMAP server and various clients deleting (move to trash) including Apple Mail and AquaMail for Android.

Just replaced ImapToDbSynchronizer.php and restarted web server (Apache).

[meichthys]

@AlexanderZuehr Did this eventually start working for you? If not, would you mind providing your email provider? I can confirm that i see this as being fixed for some mail accounts and not others. Perhaps some mail accounts don't report the QRESYNC command like others do.

[AlexanderZuehr]

@meichthys Hey, no it is not working for me right now. I applied the patch 2 weeks ago and for me and the other users this problem still exists. In the meantime I updated to nextcloud 27.1.4 and I am planing to update the mail-app to 3.4.5 maybe that will help. We aren't using a mail provider as we are self hosting our mailserver with standard linux tools (postfix, dovecot) etc.

[meichthys]

Ok, Thanks. You may need to enable the QRESYNC IMAP Extension, but good luck! Nice to hear some folks are self-hosting email!

[Githopp192]

updated to nc 27.1.4 - newest app stack.

same issue; ok - i did clear the cache over 'debug' - let's see, if the problem re-occurs -

btw - is there any option into the app to let the users clear the cache, without, that the admin needs to enable the debug=true mode ?

[cnaumer]

I tested again as some said it works for them. I am using Cyrus-Imapd on the same host as nextcloud. This reports QRESYNC in its capabilities but it still does not work for me.

when I refresh the INBOX I get this:

[mail] Fehlersuche: partial sync 4:INBOX - get vanished messages via Horde took 0s. 5/5MB memory used [mail] Fehlersuche: partial sync 4:INBOX - delete vanished messages took 0s. 5/5MB memory used ... [mail] Fehlersuche: Forcing full sync due to QRESYNC

If I can help debug this let me know.

[cnaumer]

As you can see the app knows there are only 3 messages but it still displays the 4th in the list.

[ChristophWurst]

the app knows

It's what IMAP reports on the mailbox STATUS

[powerpaul17]

Unfortunately the problem still persists on my server. I patched the file manually and cleared the cache for the folders. After about 2 days it again shows old/deleted messages. :slightly_frowning_face: QRESYNC is enabled on the IMAP Server..

[bovender]

For me it sometimes works, sometimes it doesn't. I have two IMAP accounts, one is a personal Dovecot server (with QRESYNC), the other one is an Exchange IMAP server at a large university (without QRESYNC). Some deleted mails disappear from Nextcloud mail, others don't. It's unpredictable.

This is really too bad given that other solutions have absolutely no problem in handling this. Roundcube, for example. Speaking of which... https://nextcloud.com/de/blog/open-source-email-pioneer-roundcube-comes-aboard-nextcloud/ Evidently there are no plans to incorporate Roundcube into Nextcloud (in terms of web UI), however I wish they would, just like they did with Collabora Office. 2 cents.

[Chartman123]

@st3iny I'm also still facing this issue after updating to the latest Mail release... :/

[jakobroehrl]

Still the same error when the mail was deleted by my mail app on the smartphone

[Chartman123]

@jakobroehrl Are you using K-9? I think that my problems started last week when they implemented the MOVE command for IMAP operations instead of Copy and Delete.

https://github.com/thunderbird/thunderbird-android/pull/7369