search

deltachat / deltachat-android

Email-based instant messaging for Android.
GNU General Public License v3.0
1.14k stars 147 forks source link

MUA replies left behind in INBOX #307

Closed gerroon closed 5 years ago

gerroon commented 6 years ago

Hi

Thanks for this app. It works well when sending messages meaning that all messages are sent to the chat folder on my server. But the replies to my messages are delivered to Inbox. The friend who replies does not use DC, using regular email client . Is that how it works? If so, seems convoluted.

I am using it Android 0.17.1 Lollipop Server Exim, Dovecot on Debian Wheezy.

thanks

r10s commented 6 years ago

Is that how it works? If so, seems convoluted.

No, the idea is that replies should go to the "Chats" folder, too. However, this only works if Delta Chat is online and got the mail before. Can you check, if this happened? Is the reply visible in Delta Chat?

testbird commented 6 years ago

Here, I see a qr related message (according to the time sent) that stayed in the inbox.

testbird commented 6 years ago

After the qr test last night, and now receiving an unrelated non-encrypted message from a former deltachat user (correctly in a chat): The old qr message is now in the chats folder, but the new unencrypted message stayed in the server's INBOX.

gerroon commented 6 years ago

@r10s

DC is running in the background constantly, I see it is in the status bar of my phone. I am sure DC gets the message first because the chat (ones with Re:Chat:) messages (in the Inbox) in the email client (Claws mail) on my desktop shows that the message is already read.

The replies are notified by DC to me on my phone and I read the replies in DC first.

gerroon commented 6 years ago

So can you guys guide me on this? This is driving me nuts here :( The whole reason I use DC was to keep my inbox clean now it is flooded.

the weird thing is that I remember earlier versions did not have this issue, I was still using the same email and email server.

Is this a bug?

testbird commented 6 years ago

It looks I had wrongly thought the message from a qr contact verification was later moved to the chats folder. (Due to different time formats (hours/minutes hours/minutes/seconds) in the MUA.) The messages have not been moved later.

The situation is rather that:

1) As gerroon, I see a plain-text email message that appears in the chat but is not moved to the chats folder.

In my case the message received seems to be a reply to a message not sent with deltachat.

2) I see two deltachat messages staying in the inbox from using the experimental qr-code based contact verification process that can be enabled through the advanced options. The messages seem to be the messages send by the qr-code scanning party (received by the qr-code displaying party) during two scans with about ten minutes in between. (In the inbox of the scanning sender, strangely, only the first one is still present, but marked for deletion. May that be due to having the folder open and attempting to view the message with the MUA?)

testbird commented 6 years ago

Here is an idea to solve the problem of flooding the inbox with plain-text chat messages that chat contacts may send not in-reply-to a chat message (which was true in my case):

Gerroon, could you check if the messages you get have a proper in-reply-to header?

gerroon commented 6 years ago

@testbird,

Thanks for looking into this.

I do not have DC to DC chat use case. My friends are all using their own emails/clients.

here is the portion of the header


Subject: Re: Chat: interesting that ...
In-Reply-To: <20180517154221.GF31946@XXXXXXXXXX.com>
References: <Rf.eNoz9vcdCQ8@mr.thread>
 <Mr.6IUJy7bmq7K.SoMYTU9QQF5@XXXXXXXXXX.com>
 <4368603544a128ca1f6eb920c2a44a7e@FRIEND.com>
 <20180517154221.GF31946@XXXXXXXXXX.com>
X-Priority: 1 (Highest)
Message-ID: <21e421e4b7c3d142411874ed41952468@FRIEND.com>
X-Sender: FRIEND@FRIEND.com
User-Agent: Roundcube Webmail/1.2.7
X-Originating-IP: xx.xx.xx.xx
testbird commented 6 years ago

From my naive view it looks as if References: even contains a deltachat reference (mr.thread), so is it a bug @r10s?

testbird commented 6 years ago

Is 20180517154221.GF31946@XXXXXXXXXX.com the Message ID of the deltachat message you've sent?

gerroon commented 6 years ago

Yess all XXXX are my domain

testbird commented 6 years ago

OK, thanks, could you please edit the title to something like "MUA replies left behind in INBOX" for later recognition?

gerroon commented 6 years ago

So is this a bug?

testbird commented 6 years ago

I guess so. Hopefully, @r10s will find it.

gerroon commented 6 years ago

I uninstalled and reinstalled Deltachat (Fdroid 0.7.3) and I added my account from scratch. It is still broken with message replies staying in Inbox.

violoncelloCH commented 6 years ago

0.7.3 or 0.17.3?

gerroon commented 6 years ago

Sorry It is is actually "v0.17.3"

gerroon commented 6 years ago

Is this going to get fixed or any ideas?

I am thinking to stop using this app just because of this. If this issue is not concern to others, that is fine. I am not a dev, I can't do much about it. It is just that it is quite a bit of hassle for me to keep cleaning my inbox, not worth the trouble really. And I do not know how to make it work.

thanks

webratte commented 6 years ago

@DC-deveteam

Maybe there is a way everytime DC connect to the server it could run a scan for DS message replies in the inbox.

And if it found a DC message or a a reply of a DC message (readed or not readed) it could move it into the chat folder. Maybe like a Sieve filter (I hope I use the correct word - my bad english ;-))

I don't know how you mark a chat message to seperate it from normal mails (by subject I suppose).

I hope you understand what I mean.

gerroon commented 6 years ago

I thought that is what it ws doing but appearantly it is not the case. It seems like this works fine for the most people but not for me :(

Ji-eF commented 6 years ago

Maybe like a Sieve filter

It's planned for a future release. In the meantime, you can create a filter that move messages in a Chat folder on your Inbox. I created these rules on my Seamonkey Mail client (Tools > Messages Filters) for sorting out my DC messages : Match all of the following :

gerroon commented 6 years ago

@Ji-eF

Are you saying that replies goes to Inbox is by design? I mean sounds like you have this issue as well?

I personally would like this issue to be resolved and I am not running an email clinet on my computer so that some messgaes are sorted propery, in reality DC should have been taking care of this properly.

Also it is a bit embarrasing when I recommend this app to freiends, and they also get confused by their inboxes are littered with "Chat:" messages. Am I supposed to tutorial them about how to set up emails filters on theor desktops?

Do not get me wrong, but as @r10s indicated in his reply to my post, this sounds like a bug and to me an important bug, because it just destroys the experience and I stopped recommending to my friends :(

webratte commented 6 years ago

I have now created a filter in my mail account and it works.

But I think @gerroon is right. This job should be done by DC.

As I said before. If DC is runnig it should move the mail immediately after receiving a new mail.

If DC is off-line it should check the inbox for new mails if it comes online an move it into the chat folder. Of course in the meanwhile the mail stay in the inbox until DC comes online.

violoncelloCH commented 6 years ago

for me delta chat moves all sent messages as well as replies to them to the Chats folder as soon as it is active...

gerroon commented 6 years ago

@violoncelloch

This is quite a dilemma :( What email server are you using?

violoncelloCH commented 6 years ago

@gerroon not quite sure, but I think my hoster uses exim

Ji-eF commented 6 years ago

Guys :) Don't forget there's no stable release yet. The 1.0 Milestone should give you a hint on what to expect from Delta Chat when it's released to the general public. Meanwhile, kindly explain to your friends that you found a new cool Chat software to test with them, but also tell them that it still is under development, so they should expect that few things go wrong sometimes. ( Also take a look at #108 )

gerroon commented 6 years ago

@Ji-eF

Thanks for the link, it seems like it was broken for a while but I do not understand why this is broken for this long? Is it like a mysterious edge case (seems like many users experience it)? is it that the devs cant replicate it? Is it going to be a very hard to fix?

To me delta chat has two serious issues.

1 - This bug already

2 - It is not poling messages fast enough, sometimes I have to kill the app and restart it to make it force get it. Maybe it should offer press and pull down in the message window for a refresh like K-9 does

These two makes this app more like a buggy email client than a chat app to be honest. Sorry for the negative tone in my reply, but sometimes hard truths have to come out hard to be useful.

webratte commented 6 years ago

Meanwhile, kindly explain to your friends that you found a new cool Chat software to test with them, but also tell them that it still is under development, so they should expect that few things go wrong sometimes.

I will do that if there is a Beta version in Google Playstore ;-)

csb0730 commented 6 years ago

DC v0.17.1

I can confirm a similar behaviour in the moment. Messages which should be moved to Chats folder cannot be moved because connection to IMAP server is lost after each try to move. Please see or compare core 170 issue. I don't saw this with v0.16.0 release as far as I'm able to estimate this. But this is only an assumption.

What I see too, sometimes DC seems not to be able to send any message. Only a restart by application manager helps to get it working again. But then adhoc it works properly again.

I suspect, that there are some (two) issues in the air!

Please check imap/smtp logging for errors (settings->Info->Info). Maybe there are some hints?!

gerroon commented 6 years ago

@csb0730

I do not see anything beside DC constantly checking some empty or non existing folders on my server with "0 mails read from _ORDER/.@WORK.Something " and some others in that folder :(

For instance I have "_ORDER" (which I created back then) folder but it is empty in my server. Somehow DC looking for _ORDER/.@WORK.Something (Something is not te real name there)> The _ORDER folder is empty according to my desktop email client (after subscribing to everything in the folder).

Beside those I am wondering why Dc is parsing all the folders, sshould not be just looking at Inbox and Chat folders only in the first place?

Maybe you should give us an option to choose where to parse the new emails as well, that way DC does not keep the email server busy constantly..

gerroon commented 6 years ago

So is there any rough estimation about fixing this? Are we going to wait for the release to get this working properly?

t-oster commented 6 years ago

@gerroon for estimating and fixing we have to reproduce the issue fist. In my case it works as expected on DomainFactory. So what provider are you using? Can you provide a test-account? Does this happen to all replys or just randomly?

Ji-eF commented 6 years ago

@t-oster @r10s : I can provide you with an e-mail address from my host if you want to fiddle with.

t-oster commented 6 years ago

yes please. You can send the details mail at thomas-oster.de

Ji-eF commented 6 years ago

Sent (Hope I got your mail correctly :p )

gerroon commented 6 years ago

yay progress!

t-oster commented 6 years ago

didn't receive it yet ;(

Ji-eF commented 6 years ago

Did my parser failed me ? mail<at>thomas-oster.de Also, check your Junk mail folder I guess...

t-oster commented 6 years ago

Well. I can reproduce this issue. IMHO this belongs to core instead of android. But I am still searching for the code which is responsible for moving the message. Here is a sample log:

t-oster commented 6 years ago

@r10s the message is generated here https://github.com/deltachat/deltachat-core/blob/master/src/mrmailbox_receive_imf.c#L1334 can you point me to the code, where the message should be moved afterwards? Or should it already be moved then? I think it should be moved in the markseen function, but I only see this call (https://github.com/deltachat/deltachat-core/blob/master/src/mrmailbox_receive_imf.c#L1447) which is only for MDNs if I see correctly?

r10s commented 6 years ago

@t-oster the log line is a bit misleading, when Message has .. parts and is moved to chat #..., the message is already "moved" there by the INSERT INTO msgs command some lines above. maybe we should write is assigned to chat #....

r10s commented 6 years ago

this call (https://github.com/deltachat/deltachat-core/blob/master/src/mrmailbox_receive_imf.c#L1447) which is only for MDNs if I see correctly?

yes, for received MDNs, this adds the job MRJ_MARKSEEN_MDN_ON_IMAP to the database which results in a call to mrmailbox_markseen_mdn_on_imap() { .. mrimap_markseen_msg(.. MR_MS_ALSO_MOVE ..) .. }

however, it might really be that the overall logic is not always working ... great that you have a look on this :)

t-oster commented 6 years ago

so where should the E-mail be (imap)-moved to the DeltaChat folder in this case? Or be marked as seen?

r10s commented 6 years ago

so where should the E-mail be (imap)-moved to the DeltaChat folder in this case? Or be marked as seen?

i do not understand the question, what does the "where" refer to? the "real" moving on the imap server is done in this line https://github.com/deltachat/deltachat-core/blob/master/src/mrimap.c#L1709 however, i am not sure if the problem is the "real" moving - or if the mess of conditions whether to add the move job or not is just wrong :) (i will have a closer look on this and trying to understand :)

t-oster commented 6 years ago

Is it correct that the messages are only moved AFTER they are read in deltachat? Maybe we should change this so they are moved on receive.

gerroon commented 6 years ago

I say they should be moved on receibe if not moved by some other server/client side filter.

r10s commented 6 years ago

Is it correct that the messages are only moved AFTER they are read in deltachat?

i'm only half here currently, however, yes, iirc, this is true.

Maybe we should change this so they are moved on receive.

iirc, then reason for this is multi-client (only the INBOX is watched via push and if it is moved away by a device currently not in use (eg. the computer), the device in use (eg. the handy) may get the message delayed)

t-oster commented 6 years ago

So we should monitor inbox and DeltaChats folder via push.we can still use the Seen flag for tracking if the msg is read by another client

r10s commented 6 years ago

So we should monitor inbox and DeltaChats folder via push.

yes, this could be a solution. unfortunately, it's not easily possible to monitor multiple folders, see here: https://github.com/deltachat/deltachat-core/issues/180#issuecomment-394720059