search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.36k stars 925 forks source link

[Twitter] Expand conversations/threads in timeline and few improvements #2665

Open nisehime opened 2 years ago

nisehime commented 2 years ago

Hello.

Twitter truncates long conversations in users timelines (by long I mean longer than 3 tweets), which causes missed tweets from /with_replies timeline and even in threads from normal timeline. In other words if a long thread/conversation appears on timeline, only first one and last two tweets from it would be downloaded. So these conversations should be expanded. For example, when there's a tweet which is a reply you open it directly and then download rest of the thread. Maybe there's another way to do it with twitter's web API.

Secondly, I asked you to improve the replies setting and add option to grab tweets where user id matches the id of target (from URL) user id. Currently, if you set it to self it doesn't download target's tweet where he replies to other user, and if you set it to true than it downloads tweets from non-target users which are probably unneeded in most cases. Actually, I think you can just make self to behave like this instead of doing 4rd option. It would be nice if target's user id or better all their metadata will be always accessible in keyword dictionary for some filtering purposes.

After that you can improve the default algorithm

  1. Set the default replies option to self
  2. Use /with_replies timeline for regular user links (twitter.com/user) instead of /tweets when retweets=true
  3. text-tweets=true should also switch the timeline to /with_replies instead of media like retweets. This also needs to remove media filter from search.
aleksusklim commented 2 years ago

If there will be a reliable method to stop downloading older content, then there is no need in implicitly changing endpoints (which is bad, when the user knows better what he wants) in trying to guess "which is better".

One such method I explained in the bottom of https://github.com/mikf/gallery-dl/issues/2624#issuecomment-1140021564 I still did not experiment further, but I will.