search

Mincka / DMArchiver

A tool to archive the direct messages, images and videos from your private conversations on Twitter
GNU General Public License v3.0
223 stars 25 forks source link

Only 50 latest conversations are downloaded in "All conversations" mode #8

Closed Mincka closed 7 years ago

Mincka commented 7 years ago

From https://github.com/Mincka/DMArchiver/issues/7:

I think I may have an idea why all the threads have not been downloaded the first time. I've counted all the threads in your previous message and found exactly 50 conversations. Currently, to find "all" the conversation IDs, the script loads the conversations available on the "first" "Messages" page but do not simulate scrolling to load more. I though that all the conversations were listed directly.

My guess is that when you scroll down through all the conversations, at the bottom, Twitter loads the next 50 conversations. I did not identify this case because I have a lot less than 50 conversations on Twitter! But it's an interesting case and I'm going to open a new ticket to improve the "all threads" mode which is in fact a "latest 50 conversations" mode it seems.

Mincka commented 7 years ago

I need help for this one. I don't know which HTTP request is done by Twitter to load the next 50 private conversations in the "Messages" popup.

Mincka commented 7 years ago

Ok, so the flow is: 1) https://twitter.com/messages (min_entry_id=XX1 & has_more=True) 2) https://twitter.com/messages?max_entry_id=XX1 (min_entry_id=XX2 & has_more=True) 3) https://twitter.com/messages?max_entry_id=XX2 (min_entry_id=XX3 & has_more=True) ... 4) https://twitter.com/messages?max_entry_id=XXY (missing min_entry_id & has_more=False) 5) End