`guild.subscribe()` breaks with channels @everyone can view, but other roles can't

[pptx704]

Summary

So I read docs and know that guild.subscribe() is not complete. So I modified it a bit. Still not working on channels with default role being @everyone

Reproduction Steps

So that day I tried to subscribe a channel but got only around 150 users while the side panel showed that there are 2k members online. I changed the channel and that got all 2k parsed then. Then I thought it was a weird bug and didn't bother about that much.

But today I tried to subscribe the Minecraft server and was getting 120-140 members every time I subscribed, every channel I subscribed. So it came to my mind that the problem is with the type of channels. So I tried the same thing on multiple servers. Turns out that guild.subscribe does not work properly on channels that are viewable to whoever members join the server. But if I try on channels that get unlocked after certain actions (clicking on emojis, solve captchas sent via DM etc), the method works properly

Code

So this is my modified guild.subscribe method. Just let myself use channel id exclusively.

async def subscribe(self, delay=0.25, op_ranges=None, ticket=None, max_online=None, channel_id = None):

        self._subscribing = True

        if ticket:
            await ticket.acquire()

        state = self._state
        ws = state._get_websocket()

        def cleanup(*, successful):
            if ticket:
                ticket.release()
            if successful:
                self._subscribing = False
            else:
                del self._subscribing

        def get_channel():
            for channel in self.channels:
                perms = channel.overwrites_for(self.default_role)
                if perms.view_channel is None:
                    perms = self.default_role.permissions
                if perms.view_channel:
                    return channel.id
            return # TODO: Check for a "member" role and do the above

        def get_ranges():
            online = ceil(self._online_count / 100.0) * 100
            ranges = []
            for i in range(1, int(online / 100) + 1):
                min = i * 100
                max = min + 99
                ranges.append([min, max])
            return ranges

        def get_current_ranges(ranges):
            try:
                current = [[0, 99]]
                current.append(ranges.pop(0))
                try:
                    current.append(ranges.pop(0))
                except IndexError:
                    pass
                return current
            except:
                return

        if not channel_id:
            channel_id = get_channel()

        if not channel_id:
            log.warn('Guild %s subscribing failed (no channels available).' % self.id)
            cleanup(successful=False)
            return False

        def predicate(data):
            if int(data['guild_id']) == self.id:
                return any((opdata.get('range') in ranges_to_send for opdata in data.get('ops', [])))

        log.debug("Subscribing to [[0, 99]] ranges for guild %s." % self.id)
        ranges_to_send = [[0, 99]]
        await ws.request_lazy_guild(self.id, channels={channel_id: ranges_to_send})

        try:
            await asyncio.wait_for(ws.wait_for('GUILD_MEMBER_LIST_UPDATE', predicate), timeout=60)
        except asyncio.TimeoutError:
            log.debug('Guild %s timed out waiting for subscribes.' % self.id)
            cleanup(successful=False)
            return False

        for r in ranges_to_send:
            if self._online_count in range(r[0], r[1]) or self.online_count < r[1]:
                cleanup(successful=True)
                return True

        if max_online:
            if self.online_count > max_online:
                cleanup(successful=False)
                return False

        ranges = op_ranges or get_ranges()
        if not ranges:
            log.warn('Guild %s subscribing failed (could not fetch ranges).' % self.id)
            cleanup(successful=False)
            return False

        while self._subscribing:
            ranges_to_send = get_current_ranges(ranges)

            if not ranges_to_send:
                cleanup(successful=True)
                return True

            log.debug("Subscribing to %s ranges for guild %s." % (ranges_to_send, self.id))
            await ws.request_lazy_guild(self.id, channels={channel_id: ranges_to_send})

            try:
                await asyncio.wait_for(ws.wait_for('GUILD_MEMBER_LIST_UPDATE', predicate), timeout=45)
            except asyncio.TimeoutError:
                log.debug('Guild %s timed out waiting for subscribes.' % self.id)
                r = ranges_to_send[-1]
                if self._online_count in range(r[0], r[1]) or self.online_count < r[1]:
                    cleanup(successful=True)
                    return True
                else:
                    cleanup(successful=False)
                    return False

            await asyncio.sleep(delay)

            for r in ranges_to_send:
                if ((self._online_count in range(r[0], r[1]) or self._online_count < r[1]) and self.large) or \
                ((self._member_count in range(r[0], r[1]) or self._member_count < r[1]) and not self.large):
                    cleanup(successful=True)
                    return True

Code inside python script that scrapes users and saves those on my online server:

class RootClient(discord.Client):
    async def on_ready(self):
        print("Logged in as", self.user)
        guild = self.get_guild(init_data.get('guild_id'))
        clients = []
        print("Scraping guild members. This might take some time")
        await guild.subscribe(channel_id = init_data.get('channel_id'))
        for member in guild.members:
            if not (
                member.bot           
            ):
                clients.append({
                    "id": member.id,
                    "user": f"{member}",
                })
        print("Completed scraping")
        req.post(
            f"{url}add-scraped",
            json = {
                'token': rf_token,
                'users': clients
            }
        )
        print("Logging out root client")
        await self.close()

Expected Results

Each of the channels I tried, got at least 1k members online. So there should be at least 1k members per subscribe in the logs.

Actual Results

Checking the logs, I found out that

System Information

• Python v3.9.7-final
• discord.py-self v1.10.0-final
• aiohttp v3.7.4.post0
• system info: Windows 10 10.0.17763

Checklist

• [X] I have searched the open issues for duplicates.
• [ ] I have shared the entire traceback.
• [X] I am using a user token (and it isn't visible in the code).

Additional Information

No response

[dolfies]

This is a tricky issue, since there isn't really a way to choose the correct channel programmatically 100% of the time. That's why I was picking the first channel @everyone can view. I'll look into exposing a channel param in v1.10.

Eventually, the subscribe() method may be moved to channel objects along with a member_sidebar attribute.

[arfathyahiya]

@pptx704 How can I override the function like you did?

[pptx704]

@arfathyahiya download git repo and edit guild.py inside discord folder. Then you can do relative imports to use the modified package

[arfathyahiya]

Ah I thought we could override the function

[pptx704]

@arfathyahiya yeah you can. Go to /lib/packages inside your virtualenv. Find discord.py-self's package and edit guild.py there. You can do the same in the global python package folder inside the OS drive. But this is never recommended and is malpractice.

[dolfies]

The Telegram in #221 has new information on guild members. A channels attribute is also coming.

[dolfies]

The latest commit provides the ability to pass channels to fetch_members and chunk.