Thread listing/downloading limitation

beareogaming commented 1 month ago

Version

2.43.3

Flavor

CLI (Command-Line Interface)

Platform

win11

Export format

HTML, TXT

Steps to reproduce

When exporting threads via the --include-threads command or by exporting a list of threads using the channel command, if a server has a very large amount of threads, not all threads will download or be listed, For example i have a server with multiple 300+ thread channels. CLI will only collect a third of it roughly.

i believe this may possibly be a discord issue aswell as when viewing those channels in gallary view via the discord client the full list of channels wont load unless you switch to list view, then they will all load. perhaps related? (to reproduce the discord client issue you must restart your client to clear cache, once all channels are loaded they will be visible in both gallary and list view but exclusively load only in list view to its entirety)

Details

Expecting to export full list of roughly 1000 threads, actually results are less than expected value but consistant across multiple attempts. if it only exports 100 of the 300 threads it will do that the second attempt aswell.

Attempted with both DiscordChatExporter.Cli.exe channels -g 123456789 --include-threads all and exportguild -g123456789 --include-threads all

Checklist

[X] I have looked through existing issues to make sure that this bug has not been reported before
[X] I have provided a descriptive title for this issue
[X] I have made sure that this bug is reproducible on the latest version of the application
[X] I have provided all the information needed to reproduce this bug as efficiently as possible
[ ] I have sponsored this project
[ ] I have not read any of the above and just checked all the boxes to submit the issue

Tyrrrz commented 1 month ago

Are you using a bot token or a user token? If possible, try switching from one to another, as they use different ways to fetch threads. Curious if that has an effect.

Also, are the missing threads archived or mixed archived/active?

beareogaming commented 1 month ago

Tested with both bot and user tokens, i typically use a bot for it.

Both archived and active, its like it only displays roughly 100 of the most recent threads. regardless of archived or active. i havent compared this list to the ones you can view in gallary mode without switching to list view to see if they are the same.

CharithBhat commented 1 month ago

I'm having the same issue with a large server with over 1k threads per channel. Only around 200 threads are downloaded. With a small server, everything works fine.

Noticed even some channels that the bot had permission for were not downloading in the large server. Don't know if this is some sort of rate limiting. I had parallelization at 1.

beareogaming commented 1 month ago

I've contacted discord about it as well and they are currently "looking into" I was told. Havn't heard back in a week or two now. (About their janky loading system that is)

Yorkemartin commented 1 month ago

i get this issue as well, if anyone finds a solution please post!

Tyrrrz / DiscordChatExporter