search

ChinemeremChigbo / discord-mutual-friends-and-servers

Provides lists of Discord server members that are your friends, members you have mutual friends with, and members you have mutual servers with.
19 stars 0 forks source link

Scanning speed. #4

Closed LFL38 closed 1 month ago

LFL38 commented 3 months ago

Awesome tool, love it! However it scans incredibly slowly.. could I change some stuff in the code to make it faster, or does that completely break the bot? Thanks in advance!

Nozlightning commented 3 months ago

Also very interested in this question!

ChinemeremChigbo commented 3 months ago

Yes, if you change the --sleep_time flag to be shorter you can speed up the script. The default sleep time is 3 seconds, meaning that the code waits 3 seconds between each member request. This means that a server with 100 members would take at least 300 seconds to scan. The reason the sleep time is so high is because lower values sometimes hit discord's rate limit. I have never experienced this, but I think if the rate limit is hit excessively, Discord might temporarily, or permanently ban your account. You can experiment with a lower sleep time e.g. 1 second, but your results may vary. You can specify the sleep_time like so: python3 main.py --sleep_time=1. Please let me know if you have any other questions, otherwise I will close the issue.

LFL38 commented 3 months ago

Yes, if you change the --sleep_time flag to be shorter you can speed up the script. The default sleep time is 3 seconds, meaning that the code waits 3 seconds between each member request. This means that a server with 100 members would take at least 300 seconds to scan. The reason the sleep time is so high is because lower values sometimes hit discord's rate limit. I have never experienced this, but I think if the rate limit is hit excessively, Discord might temporarily, or permanently ban your account. You can experiment with a lower sleep time e.g. 1 second, but your results may vary. You can specify the sleep_time like so: python3 main.py --sleep_time=1. Please let me know if you have any other questions, otherwise I will close the issue.

Oh that's awesome... also, when the script crashes with a weird error (I'll run it now and send a screenshot when it happens) does that mean the discord API is rate limiting me?

LFL38 commented 3 months ago

Yes, if you change the --sleep_time flag to be shorter you can speed up the script. The default sleep time is 3 seconds, meaning that the code waits 3 seconds between each member request. This means that a server with 100 members would take at least 300 seconds to scan. The reason the sleep time is so high is because lower values sometimes hit discord's rate limit. I have never experienced this, but I think if the rate limit is hit excessively, Discord might temporarily, or permanently ban your account. You can experiment with a lower sleep time e.g. 1 second, but your results may vary. You can specify the sleep_time like so: python3 main.py --sleep_time=1. Please let me know if you have any other questions, otherwise I will close the issue.

Oh that's awesome... also, when the script crashes with a weird error (I'll run it now and send a screenshot when it happens) does that mean the discord API is rate limiting me?

[2024-07-06 17:59:17] [ERROR   ] discord.client: Ignoring exception in on_ready
Traceback (most recent call last):
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\client.py", line 637, in _run_event
    await coro(*args, **kwargs)
  File "C:\Users\Floppa\Desktop\discord-mutual-friends-and-servers-main\main.py", line 48, in on_ready
    server_info = await self.get_server_info(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\Desktop\discord-mutual-friends-and-servers-main\main.py", line 310, in get_server_info
    member_profile = await server.fetch_member_profile(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\guild.py", line 2643, in fetch_member_profile
    data = await state.http.get_user_profile(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\http.py", line 928, in request
    raise NotFound(response, data)
discord.errors.NotFound: 404 Not Found (error code: 10013): Unknown User
ERROR:discord.client:Ignoring exception in on_ready

running with --sleep_time = 10

artypoop commented 2 months ago

Yes, if you change the --sleep_time flag to be shorter you can speed up the script. The default sleep time is 3 seconds, meaning that the code waits 3 seconds between each member request. This means that a server with 100 members would take at least 300 seconds to scan. The reason the sleep time is so high is because lower values sometimes hit discord's rate limit. I have never experienced this, but I think if the rate limit is hit excessively, Discord might temporarily, or permanently ban your account. You can experiment with a lower sleep time e.g. 1 second, but your results may vary. You can specify the sleep_time like so: python3 main.py --sleep_time=1. Please let me know if you have any other questions, otherwise I will close the issue.

Oh that's awesome... also, when the script crashes with a weird error (I'll run it now and send a screenshot when it happens) does that mean the discord API is rate limiting me?

[2024-07-06 17:59:17] [ERROR   ] discord.client: Ignoring exception in on_ready
Traceback (most recent call last):
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\client.py", line 637, in _run_event
    await coro(*args, **kwargs)
  File "C:\Users\Floppa\Desktop\discord-mutual-friends-and-servers-main\main.py", line 48, in on_ready
    server_info = await self.get_server_info(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\Desktop\discord-mutual-friends-and-servers-main\main.py", line 310, in get_server_info
    member_profile = await server.fetch_member_profile(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\guild.py", line 2643, in fetch_member_profile
    data = await state.http.get_user_profile(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\http.py", line 928, in request
    raise NotFound(response, data)
discord.errors.NotFound: 404 Not Found (error code: 10013): Unknown User
ERROR:discord.client:Ignoring exception in on_ready

running with --sleep_time = 10

I'm also getting similar error :(

[2024-07-17 17:31:09] [ERROR   ] discord.client: Ignoring exception in on_ready
Traceback (most recent call last):
  File "C:\Users\artem\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\discord\client.py", line 637, in _run_event
    await coro(*args, **kwargs)
  File "C:\Users\artem\main.py", line 48, in on_ready
    server_info = await self.get_server_info(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\artem\main.py", line 310, in get_server_info
    member_profile = await server.fetch_member_profile(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\artem\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\discord\guild.py", line 2655, in fetch_member_profile
    mutual_friends = await state.http.get_mutual_friends(member_id)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\artem\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\discord\http.py", line 930, in request
    raise DiscordServerError(response, data)
discord.errors.DiscordServerError: 503 Service Unavailable (error code: 0): upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: immediate connect error: No such file or directory
ERROR:discord.client:Ignoring exception in on_ready
Traceback (most recent call last):
  File "C:\Users\artem\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\discord\client.py", line 637, in _run_event
    await coro(*args, **kwargs)
  File "C:\Users\artem\main.py", line 48, in on_ready
    server_info = await self.get_server_info(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\artem\main.py", line 310, in get_server_info
    member_profile = await server.fetch_member_profile(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\artem\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\discord\guild.py", line 2655, in fetch_member_profile
    mutual_friends = await state.http.get_mutual_friends(member_id)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\artem\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12_qbz5n2kfra8p0\LocalCache\local-packages\Python312\site-packages\discord\http.py", line 930, in request
    raise DiscordServerError(response, data)
discord.errors.DiscordServerError: 503 Service Unavailable (error code: 0): upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: immediate connect error: No such file or directory
ChinemeremChigbo commented 2 months ago

Yes, if you change the --sleep_time flag to be shorter you can speed up the script. The default sleep time is 3 seconds, meaning that the code waits 3 seconds between each member request. This means that a server with 100 members would take at least 300 seconds to scan. The reason the sleep time is so high is because lower values sometimes hit discord's rate limit. I have never experienced this, but I think if the rate limit is hit excessively, Discord might temporarily, or permanently ban your account. You can experiment with a lower sleep time e.g. 1 second, but your results may vary. You can specify the sleep_time like so: python3 main.py --sleep_time=1. Please let me know if you have any other questions, otherwise I will close the issue.

Oh that's awesome... also, when the script crashes with a weird error (I'll run it now and send a screenshot when it happens) does that mean the discord API is rate limiting me?

[2024-07-06 17:59:17] [ERROR   ] discord.client: Ignoring exception in on_ready
Traceback (most recent call last):
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\client.py", line 637, in _run_event
    await coro(*args, **kwargs)
  File "C:\Users\Floppa\Desktop\discord-mutual-friends-and-servers-main\main.py", line 48, in on_ready
    server_info = await self.get_server_info(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\Desktop\discord-mutual-friends-and-servers-main\main.py", line 310, in get_server_info
    member_profile = await server.fetch_member_profile(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\guild.py", line 2643, in fetch_member_profile
    data = await state.http.get_user_profile(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.12\LocalCache\local-packages\Python312\site-packages\discord\http.py", line 928, in request
    raise NotFound(response, data)
discord.errors.NotFound: 404 Not Found (error code: 10013): Unknown User
ERROR:discord.client:Ignoring exception in on_ready

running with --sleep_time = 10

Does this error only happen when running with sleep_time = 10? Or always?

LFL38 commented 2 months ago

Does this error only happen when running with sleep_time = 10? Or always?

I'm not sure... I'll let the tool run overnight in a few minutes! I'll set the sleep time to 30, hoping that fixes it.

LFL38 commented 2 months ago

Does this error only happen when running with sleep_time = 10? Or always?

I'm not sure... I'll let the tool run overnight in a few minutes! I'll set the sleep time to 30, hoping that fixes it.

This happened after a few minutes: progress = 1/23 servers 3/1032 members [2024-07-23 01:04:04] [ERROR ] discord.client: Attempting a reconnect in 0.70s Traceback (most recent call last): File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\client.py", line 907, in connect await self.ws.poll_event() File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 623, in poll_event await self.received_message(msg.data) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 528, in received_message await self.send_as_json(beat) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 659, in send_as_json await self.send(utils._to_json(data)) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 655, in send await self.socket.send_str(data) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\client_ws.py", line 165, in send_str await self._writer.send(data, binary=False, compress=compress) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\http_websocket.py", line 729, in send await self._send_frame(message, WSMsgType.TEXT, compress) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\http_websocket.py", line 675, in _send_frame self._write(header + mask + message) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\http_websocket.py", line 702, in _write raise ConnectionResetError("Cannot write to closing transport") ConnectionResetError: Cannot write to closing transport ERROR:discord.client:Attempting a reconnect in 0.70s Traceback (most recent call last): File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\client.py", line 907, in connect await self.ws.poll_event() File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 623, in poll_event await self.received_message(msg.data) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 528, in received_message await self.send_as_json(beat) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 659, in send_as_json await self.send(utils._to_json(data)) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\discord\gateway.py", line 655, in send await self.socket.send_str(data) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\client_ws.py", line 165, in send_str await self._writer.send(data, binary=False, compress=compress) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\http_websocket.py", line 729, in send await self._send_frame(message, WSMsgType.TEXT, compress) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\http_websocket.py", line 675, in _send_frame self._write(header + mask + message) File "C:\Users\Floppa\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\aiohttp\http_websocket.py", line 702, in _write raise ConnectionResetError("Cannot write to closing transport") ConnectionResetError: Cannot write to closing transport

EDIT IT RESUMED!!!

LFL38 commented 1 month ago

Updated code myself, fixing any issues posted in this thread. Check the newest post in "issues" named "Updated Code".