Replies to DM thread don't appear in thread

[IBBoard]

I don't use DMs very often, but if I have an existing thread and reply to the thread then the reply sends, but the message becomes its own conversation in the DMs list and doesn't show up in the DM thread. However, if I click on the reply-pretending-to-be-its-own-thread then I just get a blank page. Existing DMs are correctly threaded.

Closing and re-opening Corebird puts my new DMs back in the thread, but the reply-pretending-to-be-its-own-thread remains.

Oddly, I can't recreate it between my two test accounts, but it happens fairly reliably with other users.

[Edit] And now, after a restart, it is cooperating when DMing someone else who it was breaking for just five minutes ago.

[baedert]

Can't reproduce that, the messages all end up in the proper thread here.

[IBBoard]

It's definitely intermittent. I'll try to do some more debugging and get a good, repeatable case. The "my last reply shows as its own thread in the threads list" part of it should (hopefully) be the easier bit to recreate/diagnose.

[Vistaus]

Can't reproduce this either.

[IBBoard]

I've just checked and I'm still seeing a "my reply is its own thread" DM, and the DM was from a couple of days ago, so hopefully it'll still be there and I'll be able to debug tomorrow.

[IBBoard]

The problem with the self-reply thread appears to be because something made an entry in the "dm_threads" table for my own user account. I've not worked out what made it do that yet.

Still trying to recreate the other issue (after it was so persistent when DMing a friend)

[IBBoard]

Ahah! I think I found how to recreate this (using Git Master from just now).

1) Go to DM tab 2) Go to conversation 3) Send a DM 4) Click Home tab 5) Click DM tab 6) Open conversation from step 2

Expected results:

• No thread from self at step 5
• At step 6, conversation looks the same as after step 3

Actual results:

• (empty) thread to self appeared in thread list
• Conversation doesn't contain latest DM from self

[Vistaus]

Still can't reproduce. Seems like it's an issue with your account or the Twitter server.

[IBBoard]

I've added a bit of debugging (see my dm-debug branch). When sending a DM from my IBBoard account to my other account then I got:

(corebird:4363): corebird-DEBUG: DMManager.vala:157: Inserting message - sender == account
(corebird:4363): corebird-DEBUG: DMManager.vala:215: Updating dm_thread for IBBoard

So it knows that I was the sender, but then it creates a dm_thread entry. I'm suspicious of these lines in the DM Manager, because there's an IF with identical code on each branch, and a commented out function call that's different.

I'll try to debug more when I get a chance.

For reference, I'm testing between two accounts that are configured in Corebird (although I tend to send replies from the web, rather than swapping accounts). However, this bug happened with a colleague as well, so I only had one of the accounts from the conversation configured then.

[ValdikSS]

I face the same problem and it's been for me for months like this.

[IBBoard]

I'm glad someone else is having the issue!

Possibly related: I was on my secondary computer today, and while I could see all of my messages, I couldn't see the replies! I don't know whether it is a caching vs fetching issue or what.

Also, I've now actually pushed some debug to the "dm-debug" branch, rather than having just pushed a branch that's identical to Master!

[ValdikSS]

I'm having missing replies and avatars in DM. Also, not sure if related, but new DM notification "dot" on DM button is also almost never shown.

[IBBoard]

Okay, the "empty self-reply" thread is because this block is doing its job properly and it is trying to load a thread of DMs I sent to myself. Only I don't have any, because the thread is bogus.

Still debugging a) why the thread appears anyway and b) why the messages sometimes aren't shown in the thread.

[ValdikSS]

Also when I press DM button and the threads are shown, I see only messages from a person I have conversation with, not my replies, even if I replied last.

[IBBoard]

It seems like there's some kind of lag in adding DMs to the dms table.

Using my dm-debug branch, I went to the DMs section, opened the thread to my test account, sent a DM with the text "Test6" and after I pressed "Send" then the following appeared:

(corebird:16949): corebird-DEBUG: SelectStatement.vala:90: SQL: SELECT `id` FROM `dms` WHERE `to_id`='[[acct1-id]]' ORDER BY id DESC LIMIT 1
(corebird:16949): corebird-DEBUG: SelectStatement.vala:90: SQL: SELECT `id` FROM `dms` WHERE `from_id`='[[acct1-id]]' ORDER BY id DESC LIMIT 1
(corebird:16949): corebird-DEBUG: DMManager.vala:149: DM: Saving message for sent message
(corebird:16949): corebird-DEBUG: DMManager.vala:272: DM: Inserting DM from IBBoard to IBBTwtr
(corebird:16949): corebird-DEBUG: InsertStatement.vala:50: SQL: INSERT INTO `dms` (`id`, `from_id`, `to_id`, `from_screen_name`, `to_screen_name`, `from_name`, `to_name`, `timestamp`, `text`) VALUES (?,?,?,?,?,?,?,?,?);
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     0 = [[id]]
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     1 = [[acct1-id]]
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     2 = [[acct2-id]]
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     3 = IBBoard
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     4 = IBBTwtr
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     5 = IBBoard
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     6 = Test Account
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     7 = 1494084538
(corebird:16949): corebird-DEBUG: InsertStatement.vala:54: SQL:     8 = Test5
(corebird:16949): corebird-DEBUG: DMPage.vala:210: DM: Joined for [[acct2-id]]
(corebird:16949): corebird-DEBUG: SelectStatement.vala:90: SQL: SELECT `from_id`, `to_id`, `text`, `from_name`, `from_screen_name`, `timestamp`, `id` FROM `dms` WHERE `from_id`='[[acct2-id]]' OR `to_id`='[[acct2-id]]' ORDER BY timestamp DESC LIMIT 35
(corebird:16949): corebird-DEBUG: SelectStatement.vala:90: SQL: SELECT `id`, `key`, `value` FROM `snippets` ORDER BY id
(corebird:16949): corebird-DEBUG: DMPage.vala:297: DM: Sending DM to [[acct2-id]]: Test6
(corebird:16949): corebird-DEBUG: DMThreadsPage.vala:150: DM: DMThreadsPage received a DM - inserting
(corebird:16949): corebird-DEBUG: DMManager.vala:163: DM: Inserting message - sender == account
(corebird:16949): corebird-DEBUG: DMManager.vala:178: DM: Updating thread for [[acct1-id]] with message [[msg-id]]: Test6
(corebird:16949): corebird-DEBUG: DMManager.vala:222: DM: Updating dm_thread for IBBoard
(corebird:16949): corebird-DEBUG: DMThreadsModel.vala:84: DM: Updating last message for [[acct1-id]] with message Test6
(corebird:16949): corebird-DEBUG: DMThreadsModel.vala:89: DM: Found thread - updating…

While typing this message, the following was added:

(corebird:16949): corebird-DEBUG: UpdateStatement.vala:40: SQL: UPDATE `dm_threads` SET `last_message` = ?, `last_message_id` = ? WHERE `user_id`='[[acct1-id]]';
(corebird:16949): corebird-DEBUG: UpdateStatement.vala:43: SQL:     0 = Test6
(corebird:16949): corebird-DEBUG: UpdateStatement.vala:43: SQL:     1 = [[id]]
(corebird:16949): corebird-DEBUG: DMPage.vala:74: DM: Received streamed DM to account [[acct1-id]]

Something seems odd there. I sent the "Test5" message five minutes ago, and it was already showing in the list. I don't understand why it would get inserted after I sent another test message.

Also, the extra debug that came out a few minutes later is the bit that causes the invalid thread, because there shouldn't be a dm_threads entry for acct1-id (the current account) when I haven't sent anything to myself, and sending to another account shouldn't update it even if it did exist!

[IBBoard]

I've not used it enough to be certain of that fix, but what I just pushed a) seems to fix things and b) approximately matches checks made elsewhere (e.g. the on_dm_result function).

Any thoughts? Did I miss something?

[ValdikSS]

Applied your patch, nothing changed visually. I still see my last reply as a new thread on the very top, other threads shows last correspondent message even if I replied to it and my reply is the latest message in that thread.

[IBBoard]

This won't fix any existing "self-thread" situations (because those entries are in the database and you can't trivially differentiate between "false self-reply thread" and "real self-reply thread"), but it should prevent any new ones. And I think that it is expected behaviour to show the latest reply from the other user, because that's what I see with master as well.

To delete old self-reply threads then you need sqlite and you need to run the following:

sqlite3 ~/.config/corebird/accounts/<your-acct-number>.db

then:

DELETE FROM dm_threads WHERE screen_name='<your-handle-without-@>';

Only use this if you've never sent yourself a DM!

[ValdikSS]

OK, I deleted configuration files entirely and added my account. Self-thread disappeared but other threads shows last correspondent message.

[IBBoard]

In the main thread view? I think that's expected behaviour, but @baedert would have to confirm.

Do the threads always contain all DMs when you click in to them, even if you've clicked away to the Home tab and clicked back? That's the main issue I was having (other than the incorrect self-reply thread appearing)

[ValdikSS]

Do the threads always contain all DMs when you click in to them, even if you've clicked away to the Home tab and clicked back?

Yes, I see no problem with that.

[IBBoard]

That's my main bug - new (this session) DMs vanishing from threads. That's what I'm trying to fix. And I think my patch is fixing it. But I don't know if there was a reason why it wasn't like that in the first place and I've just not noticed yet.

[IBBoard]

Okay, I found a problem with my patch - it seems that if you start a new thread by making your first ever DM to someone then it doesn't show until they reply. Which is a bit awkward.

I really need to try carefully walking through the logic on this at some point.

[axfelix]

I'm getting this behaviour for the first time right now.

[bitemyapp]

I'm getting this problem, all my replies look like they were sent to myself. They don't thread in the proper DM conversation.

[mwotton]

I'm getting this too - not every reply, but occasionally it will self-reply instead.

[axfelix]

FYI, this doesn't seem to be fixed in the most recent commits -- was bugging me, so I built from git, removed all the DMs that were marked as sent to myself from both my twitter account and the local Corebird sqlite DB, sent a new DM, and I can still reproduce the issue. You probably know that already! Just wanted to make sure.

[baedert]

You probably know that already!

No, I was waiting for feedback after that last commit mentioning this issue.

So to reproduce you removed all the dms you sent to yourself both on twitter.com and in the sqlite db and then sent a dm to yourself? From corebird or twitter.com? And it ended up where?

[axfelix]

Nope -- I removed them all, then I sent a DM to someone else via Corebird, and it's still showing up as being sent to myself in Corebird on occasion.

[baedert]

Can't reproduce the mentioned problem anymore after that commit (although I was unable to persistently delete all conversations from the twitter server; the interface tells me there are none but they still appear on the rest api...)

[axfelix]

Yeah, I was curious about that -- Corebird seems to be still be pulling something down from when I sent the erroneous DMs in the first place.

[IBBoard]

I've just done a quick test with 1.5.1 and it appears to be doing the right thing. I've only tested it with a couple of DMs back and forwards, though.

[baedert]

Eh, 1.5.1 does not contain any fix for this issue, they are just in master.

[IBBoard]

Sorry, bad assumption on my part. It is my master build. I thought I'd compared Master to the 1.5 branch and not noticed any major commit differences.

[Vistaus]

Ah, it's taken some months but now I can finally reproduce this issue. Not sure why I didn't experience it before, but as of now, it does.

[IBBoard]

Which version is that happening with? I'm running (almost latest) master and it seems to be fixed for me.

[Vistaus]

@IBBoard I'm running latest git master as well.

[baedert]

Ah, it's taken some months but now I can finally reproduce this issue. Not sure why I didn't experience it before, but as of now, it does.

Which one, exactly? There are several (un)related problems mentioned in this issue.

[baedert]

Ping