timotejroiko / discord.js-light

All the power of discord.js, zero caching. This library modifies discord.js's internal classes and functions in order to give you full control over its caching behaviour.
Apache License 2.0
292 stars 29 forks source link

Fetching members from a guild returns partial results #35

Closed WOZARDLOZARD closed 3 years ago

WOZARDLOZARD commented 3 years ago

Hello, When using <Guild>.members.fetch(), the Member and User structures returned are sometimes partial for everyone except the bot itself. Not sure if this is the intended behavior, since it is my understanding that fetching should return non-partial results.

Example:

guild.members.fetch({ withPresences: true }).then(members => {
    console.log(members.map(x => x.displayName)); // sometimes returns ["bot name", null, null, null, etc...]
    console.log(members.map(x => x.user)); // sometimes returns User structures with nothing except for the ID
});

discord.js-light version: 3.5.3 Client options:

const client = new Client({
    partials: ["MESSAGE", "CHANNEL", "REACTION", "USER", "GUILD_MEMBER"],
    ws: { intents: ["DIRECT_MESSAGES", "DIRECT_MESSAGE_REACTIONS", "GUILDS", "GUILD_BANS", "GUILD_EMOJIS", "GUILD_INVITES", "GUILD_MEMBERS", "GUILD_PRESENCES", "GUILD_MESSAGES", "GUILD_MESSAGE_REACTIONS", "GUILD_VOICE_STATES"] },
    cacheChannels: true,
    cacheEmojis: true,
    cacheOverwrites: true,
    cacheGuilds: true,
    cacheRoles: true,
    cachePresences: true,
});
timotejroiko commented 3 years ago

Are you observing this in a barebones setup? I just tested your exact client options with the following code and did not find any null username

for(let g of client.guilds.cache.values()) {
    if(g.available) {
        let members = await g.members.fetch({ withPresences: true });
        console.log(members.map(x => x.displayName).filter(x => !x))
    }
}

The only way this would happen is if somehow a partial version of members are being cached by accident, in which case you would need to fetch with the force option.

btw, the partials client option is not needed in djsl.

WOZARDLOZARD commented 3 years ago

That's strange... the error is still occurring for me. I tried using the code below and got nonzero values for a large portion of my bot's servers.

for(let g of client.guilds.cache.values()) {
    if(g.available) {
        let members = await g.members.fetch({ withPresences: true });
        console.log(members.filter(x => x.displayName == null).size);
    }
}

In addition, I tried using the force option when fetching as well and that did not resolve the issue. I've never run into this issue when using the main discord.js library either 🤔.

timotejroiko commented 3 years ago

Are you using the latest npm version (3.5.3)? or the master version?

WOZARDLOZARD commented 3 years ago

I tried using both the latest npm version and the master version and encountered this issue on both.

timotejroiko commented 3 years ago

what does console.log(guild.members.cache.size, client.users.cache.size) show before and after fetching members?

WOZARDLOZARD commented 3 years ago

Using the latest npm version: Before fetching: 1, 1 After fetching: >1, 1

However, logging guild.members.cache.filter(x => x.displayName != null).size returns 1, 1 after fetching.

timotejroiko commented 3 years ago

can you test this?

client.on("guildMembersChunk", (_, data) => console.log(data))
client.on("message", async m => {
    if(m.content === "!!!test") {
        const guild = m.guild;
        console.log(guild.members.cache.size, client.users.cache.size, guild.memberCount);
        const promise = guild.members.fetch({ withPresences:true });
        console.log(client._events.guildMembersChunk.toString());
        console.log((await promise).map(x => [x.user.username,x.presence.status]))
        console.log(guild.members.cache.size, client.users.cache.size, guild.memberCount);
    }
})
WOZARDLOZARD commented 3 years ago

Just ran the test a few times in a server with 18 members. The bot was unable to fetch any users except for itself, and the displayName property was null even though the members cache size was 18. The statuses were always fetched without problem, but the members partial and the users were unable to be fetched at all.

timotejroiko commented 3 years ago

could you provide all the logging from the above code? can be in a pastebin or similar. Or if you prefer DM me on discord and we could continue over there

WOZARDLOZARD commented 3 years ago

Here's the link to the paste: https://pastebin.com/rkfVJLE4

timotejroiko commented 3 years ago

I still haven't been able to reproduce this behavior under normal conditions... can you show this part console.log(client._events.guildMembersChunk.toString()); from the code above? I want to confirm if the function's code is correct.

The only way i could reproduce this behavior was by manually uncaching users in between fetching and mapping, are you perhaps attempting anything of the sort?

WOZARDLOZARD commented 3 years ago

Here is the code from console.log(client._events.guildMembersChunk.toString()):

(c, d) => {
    console.log(d);
}, (_, data) => {
    if (data.nonce !== nonce) { return; }
    timeout.refresh();
    i++;
    if (data.not_found) { failed += data.not_found.length; }
    for (const member of data.members) {
        fetched.set(member.user.id, this.add(member, options.cache || this.client.users.cache.has(member.user.id)));
    }
    if (presences && data.presences) {
        for (const presence of data.presences) {
            if (this.client.options.cachePresences || this.client.users.cache.has(presence.user.id)) {
                this.guild.presences.add(Object.assign(presence, { guild: this.guild }));
            }
        }
    }
    if (
        fetched.size >= this.guild.memberCount ||
        (limit && fetched.size >= limit) ||
        (typeof user_ids === "string" && fetched.size + failed === 1) ||
        (Array.isArray(user_ids) && user_ids.length === fetched.size + failed) ||
        i === data.chunk_count
    ) {
        this.client.clearTimeout(timeout);
        this.client.removeListener(Discord.Constants.Events.GUILD_MEMBERS_CHUNK, handler);
        this.client.decrementMaxListeners();
        if (typeof user_ids === "string") {
            const result = fetched.first();
            if (result) {
                r(result);
            } else {
                j(new Discord.DiscordAPIError("GUILD_MEMBERS_CHUNK", { message: "Unknown User" }, "Gateway"));
            }
        } else {
            if (!options.limit && !this.guild.memberCount) { this.guild.memberCount = fetched.size; }
            r(fetched);
        }
    }
}

I encountered this behavior by manually uncaching all users (except for the bot itself) before fetching members. In addition, I encountered this behavior regardless of whether I manually cleared user caches between fetching and mapping (I tested both scenarios and got the same results).

timotejroiko commented 3 years ago

did you use client.sweepUsers() or your own functions to clear users?

WOZARDLOZARD commented 3 years ago

I use client.users.cache.sweep(x => x.id != "bot ID")

timotejroiko commented 3 years ago

I am still not able to reproduce.

The following code returns 0 for all guilds in both "fetch" and "refetch" logs, with "after sweep" being the only one showing non-zero numbers, which is expected.

Discord = require("discord.js-light");
client = new Discord.Client({
    partials: ["MESSAGE", "CHANNEL", "REACTION", "USER", "GUILD_MEMBER"],
    ws: { intents: ["DIRECT_MESSAGES", "DIRECT_MESSAGE_REACTIONS", "GUILDS", "GUILD_BANS", "GUILD_EMOJIS", "GUILD_INVITES", "GUILD_MEMBERS", "GUILD_PRESENCES", "GUILD_MESSAGES", "GUILD_MESSAGE_REACTIONS", "GUILD_VOICE_STATES"] },
    cacheChannels: true,
    cacheEmojis: true,
    cacheOverwrites: true,
    cacheGuilds: true,
    cacheRoles: true,
    cachePresences: true,
});
client.login(token).catch(console.log);

client.on("ready", async () => {
    for(let g of client.guilds.cache.values()) {
        if(g.available) {
            let members = await g.members.fetch({ withPresences: true });
            console.log("after fetch", members.filter(x => x.displayName == null).size);
            client.users.cache.sweep(x => x.id !== client.user.id);
            console.log("after sweep", members.filter(x => x.displayName == null).size);
            members = await g.members.fetch({ withPresences: true });
            console.log("after refetch", members.filter(x => x.displayName == null).size);
        }
    }
})

If you're still getting different results with this exact code, try uninstalling both discord.js and discord.js-light, and reinstalling only discord.js-light, without discord.js

WOZARDLOZARD commented 3 years ago

On a test bot (in 2 servers) I was unable to reproduce this issue. Unfortunately, I got nonzero values for some fetch/refetch values on the main bot (in 150+ servers) using the exact code you provided, as shown in the logs here: https://pastebin.com/6qE4vnPC

try uninstalling both discord.js and discord.js-light, and reinstalling only discord.js-light, without discord.js

I used npm uninstall discord.js then npm install discord.js-light and still received nonzero values for some fetch/refetch values.

timotejroiko commented 3 years ago

well, im still not able to reproduce unfortunately, so i have no idea whats going on on your side...

try editing node_modules/discord.js-light/classes.js and add these console.logs around lines 721-731, then repeat the above test and send the logs.

            if(data.not_found) { failed += data.not_found.length; }
            console.log("before members", this.client.users.cache.size, this.cache.size);
            for(const member of data.members) {
                fetched.set(member.user.id, this.add(member, options.cache || this.client.users.cache.has(member.user.id)));
            }
            console.log("after members", this.client.users.cache.size, this.cache.size);
            if(presences && data.presences) {
                for(const presence of data.presences) {
                    if(this.client.options.cachePresences || this.client.users.cache.has(presence.user.id)) {
                        this.guild.presences.add(Object.assign(presence, { guild: this.guild }));
                    }
                }
            }
            console.log("after presences", this.client.users.cache.size, this.cache.size);
            console.log(options, fetched.size, failed);
WOZARDLOZARD commented 3 years ago

Here are the logs: https://pastebin.com/8ARbpNpj

timotejroiko commented 3 years ago

The logs look fine, the function is caching as expected. Your users are somehow disappearing afterwards. Try logging caches again before and after r(fetched); on line 755

Ortovoxx commented 3 years ago

Are any guilds unavailable? This may be related to a known discord bug

GoogleSites commented 3 years ago

You need the Server Members Intent to be enabled on your bot application in Discord in order to fetch large quantities of users

timotejroiko commented 3 years ago

i'll go ahead and close this since its very old and likely not relevant anymore