search

victornpb / undiscord

Undiscord - Delete all messages in a Discord server / channel or DM (Easy and fast) Bulk delete
https://victornpb.github.io/undiscord
MIT License
5.24k stars 716 forks source link

"Ended because API returned an empty page" with 10K messages left #471

Open CSIS-James opened 1 year ago

CSIS-James commented 1 year ago

Undiscord Version

v5.2.0

Browser

111.0.5563.65 (Official Build) (64-bit)

Extension

TamperMonkey

Reproduction steps

  1. I initially had around 76K messages in a DM to delete
  2. I followed proper instructions, using TamperMonkey, and installed the script
  3. I ran the script, and bulk deleted around 66K messages in the span of 2-3 days
  4. This morning, I ran the script again with 10K messages remaining

Actual results

The actual result was that the window returned:

Fetching messages...
Grand total: 10565  (Messages in current page: 0    To be deleted: 0    Skipped: 0) offset: 87
Ended because API returned an empty page.
[End condition A].  {"running":true,"delCount":0,"failCount":0,"grandTotal":10565,"offset":87,"iterations":5,"_seachResponse":{"total_results":10565,"messages":[],"analytics_id":"2c2bbd338e4da352bd5b5fce114bd10f"},"_messagesToDelete":[],"_skippedMessages":[]}

Expected results

I had expected the continuation of deletion

Bug affects

Additional information & file uploads

None further

kvere commented 1 year ago

getting the same issue, undiscord currently broken

CSIS-James commented 1 year ago

getting the same issue, undiscord currently broken

Ah damn. I thought it was an issue on just my end. Glad to know it's not just me

victornpb commented 1 year ago

It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.

sorry and thanks for your understanding

CSIS-James commented 1 year ago

It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.

sorry and thanks for your understanding

Hey, no worries! Thanks for everything you've done; it's been really convenient. Take your time, rest up, and make sure it isn't anything bad like a fever.

Take care!

sorryzzss commented 1 year ago

It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.

sorry and thanks for your understanding

Is there a discord server for this script?

olstice commented 1 year ago

It looks like the issue might be due to a recent Discord update. I haven't had a chance to investigate it yet, and unfortunately, I'm currently feeling sick, so I can't fix it right away. However, I'll do my best to find a solution over the weekend and let you guys know when it's resolved.

sorry and thanks for your understanding

How will we be informed that its been resolved?

victornpb commented 1 year ago

expect updates on this thread

this issue will get closed, when I have a solution, and a new version will be pushed to everyone

olstice commented 1 year ago

expect updates on this thread

this issue will get closed, when I have a solution, and a new version will be pushed to everyone

im sorry for making a duplicate! i really didnt know there was one already that you actually responded to

PetrBlue commented 1 year ago

Tried to use it after @hannahgrayson19 commented. Deleted 150+ messages, still going. Normally it would delete either 0 or 25 messages.

victornpb commented 1 year ago

Update

The search results are no longer up-to-date, this is a optimization on discord side, as it makes sense to let the database slowly reindex deleted messages instead of make it instantly reindex, it's known that removing stuff on historical databases is a resource expensive operation. Unfortunately this makes things a bit harder on Undiscord side. The good news is that I already had a plan to allow Undiscord to prefetch messages. It's a bit of a happy coincidence but I still have to finish it, so this will be a more long term solution https://github.com/victornpb/undiscord/compare/prefetch. This will be in v5.3.0.

For now, I just released v5.2.1 (Make sure to click the update button, or just click the GreasyFork button again on the home page.)

I increased the defaultSearch delay to 30s between pages. Keep in mind this is a temporary workaround, and is not a complete fix, it still stops sometimes.

tawtis commented 1 year ago

Error searching messages, API responded with status 401! {"message":"401: Unauthorized","code":0} CoreException {}

tawtis commented 1 year ago

With the authorisation token filled:


Error searching messages, API responded with status 400!
    {"code":50035,"errors":{"max_id":{"_errors":[{"code":"NUMBER_TYPE_COERCE","message":"Value \"insertemailhere@mail.com\" is not snowflake."}]}},"message":"Invalid Form Body"}
CoreException   {}
Hummel009 commented 1 year ago

Same problem

Error searching messages, API responded with status 400!
    {"code":50035,"errors":{"max_id":{"_errors":[{"code":"NUMBER_TYPE_COERCE","message":"Значение «myemail@gmail.com» не является snowflake."}]}},"message":"Invalid Form Body"}
CoreException   {}
SorianoMarmol commented 1 year ago

Same here.

I can get a 200 removing &max_id= param

hope it helps...

thanks.

image

GGG-KILLER commented 1 year ago

The search results are no longer up-to-date, this is a optimization on discord side, as it makes sense to let the database slowly reindex deleted messages instead of make it instantly reindex, it's known that removing stuff on historical databases is a resource expensive operation. Unfortunately this makes things a bit harder on Undiscord side.

@victornpb It'd be nice if we could use the data dump, that way it'd be able to avoid relying on the search endpoint entirely if it's available.

VoidPr1me commented 1 year ago

HI guys, just summing up to see if my understanding is correct on this issue:

1st - v5.20 has issues with the the deleted messages lingering on for ~1min and therefore not being able to progress due to ghost "blank page" which previously held the deleted messages.

2nd - v5.21 tries to solve that by implementing a quick-fix by increasing the page search to 30s (this most likely allows the re-indexing of discords' DB)

3rd - v5.21 has authentication issue: 401, etc., downgrading to v5.20 is the only solution to keep minimal functionality.

4th - v5.30 will mostly work-out these issues but it's more or less unclear when it will be released.

is there a current "quick fix" for the issues that can allow me to progress with deleting over 50k messages on a Discord server ?

@SorianoMarmol - is this a temporary fix ? if yes, how could I (with minimal technical knowledge) implement your proposed change ? _(I can get a 200 removing &maxid= param)

SorianoMarmol commented 1 year ago

I don't know how this works nor have I collaborated, I just know that I did a test removing that parameter and the response was 200. But as you can see in the screenshot, it can't even be interpreted, so I don't know if it's the expected response.

Regarding your question, and stressing that I don't know how this script works, maybe you can try removing the parameter from the call, I think it is this line:

https://github.com/victornpb/undiscord/blob/a9ce537bae6a40f343e09329848bd1caa394ccf0/deleteDiscordMessages.user.js#L658

Alternatively (outside of undiscord) you can always take the JSON that includes the messages (and their IDs) from the interface, parse the json to get the ids, and call delete in a loop with each of the ids. With Python it should be easy. For the base you simply copy the requests (and the json) from the chrome console (for example). Last night I tried to make call by hand changing the ID of the copied request and it worked. But doing it by hand is not feasible for 50k ;)

Regarding the part of 200 removing the param, it seems that it is irrelevant, today it already responds but it continues giving an error.

SorianoMarmol commented 1 year ago

BTW: right now using undiscord with the token filled, the response is that indicated in the issue:

Ended because API returned an empty page.

Fetching messages... Grand total: 0 (Messages in current page: 0 To be deleted: 0 Skipped: 0) offset: 0 Delete delay: 1000ms, Search delay: 30000ms Last Ping: 236ms, Average Ping: 276ms Rate Limited: 0 times. Total time throttled: 0h 0m 0s. Estimated time remaining: 0h 0m 0s Ended because API returned an empty page. [End state] {"running":true,"delCount":0,"failCount":0,"grandTotal":0,"offset":0,"iterations":1,"_seachResponse":{"total_results":0,"messages":[],"analytics_id":"bdec65dd137eb5b2ee4756babcbb59a4"},"_messagesToDelete":[],"_skippedMessages":[]} Waiting 30.00s before next page... Ended at 30/3/2023, 19:38:10! Total time: 0h 0m 7s

VoidPr1me commented 1 year ago

{"fileName":"eval","lineNumber":1160,"columnNumber":16,"message":"JSON.parse: unexpected character at line 1 column 1 of the JSON data"} Could not automatically detect Authorization Token! Please make sure Undiscord is up to date Alternatively, you can try entering a Token manually in the "Advanced Settings" section.

It doesnt detect my auth token, not even when I manually add it :( (running 5.2.1)

VoidPr1me commented 1 year ago

Update! I tweaked around the delays in v5.20 and currently for me the following settings work:

search delay: 5000ms delete delay: 3000ms

I probably could go lower than 3000ms but the current settings give Discord enough time to re-index the search results. (From what I've seen it takes roughly 30 sec.)

glassglue commented 1 year ago

It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance.

I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here.

It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again.

Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!

exile2023 commented 1 year ago

very very cool! worked.. and i found its working too with! 5000 2600

image

seems that you need to get here values where on one slide not more than 20-24 msgs get deleted

EDIT: stopped working but

5000 2800 is working perfectly for many K messages right now and still running

exile2023 commented 1 year ago

It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance.

I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here.

It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again.

Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!

see my post, just easy working with many K messages with setting 5000 2800 slow but working proper right now

githubUser01946 commented 1 year ago

It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance.

I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here.

It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again.

Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!

Not sure how to use your fork with tampermonkey

ttuleyb commented 1 year ago

Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found.

https://pastebin.com/9Yq3r9Rw

Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.

JayBoom commented 1 year ago

Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found.

https://pastebin.com/9Yq3r9Rw

Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.

@victornpb Can you confirm if this works and push it into the next update?

sapphonie commented 1 year ago

Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found.

https://pastebin.com/9Yq3r9Rw

Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.

...could you open a PR?

glassglue commented 1 year ago

It seems there's some unknown rate-limit happening on occasion when searching for old messages -- whatever patch was implemented unfortunately isn't covering all the bases, as I found there to be relatively high degrees of variance. I fixed this issue by editing the core to not break on the condition that an empty page is returned. My solution is available in my fork and viewable here. It isn't a perfect solution (really just a quick hotfix so I can leave this running without error) but it does allow for autonomous deletion once again. Edit: Successfully was able to delete ~2800 messages with this fix autonomously. There was one error I didn't handle (faulty messages continuously causing this error to propagate) that I have since fixed. Good workaround for anyone needing a quick fix!

Not sure how to use your fork with tampermonkey

You can find the compiled script here. The source code is also provided.

Hello! I've added a fix that bypasses this issue by skipping any empty pages. I have not checked this thoroughly for any bugs but it seems to be deleting messages just fine, the only change I made was to skip a maximum of 3 empty pages every time an empty page is found. https://pastebin.com/9Yq3r9Rw Make sure to Diffcheck it with the original before loading the script so you can see and approve my changes.

...could you open a PR?

Do note that this page-blank fix opens the door to a myriad of issues. I knocked out quite a few of them in my fork, but there is one remaining bug that I did not test for accuracy. I would not recommend PRing just a page-blank fix as it will (probably) create more problems than it's worth.

I can review potential bugs a bit later and potentially PR that when I am reasonably confident no further bugs will propagate.

etozheDT commented 1 year ago

Bug still reproducing image