1.1+ - Roadmap (bug fixes and more)

[realcoloride]

[Roadmap] Sum up of update goals: (from issue #14, merged here)

• [x] Switch to big int and convert to bigger ints (Added Parser class for this dedicated use) #14
• [x] Update some stuff in README.md
• [x] Fix some function arguments that required strings instead of ints #14
• [x] Fix node issues in #15
• [x] Fix fetch not defined issue (if possible)
• [ ] Fix custom.util operations not working for some people #17
• [x] Fix net::ERR_ABORTED at https://beta.character.ai/chat/streaming/ error when parsing with puppeteer (priority)
• [x] Add new user agents to go around the new limitations in #16
• [x] Fix cloudflare limiting guest authentification in #13
• [x] Update package.json
• [x] Add ascending/descending mode to deleting in bulk (from up/down)
• [x] Added delete messages in array function [ids]
• [x] Converted message ids to strings instead of integers for big integers conversion in both classes and in functions
• [x] Make errors throw when an error occurs (message could not be sent for a reason such as nsfw, guest message limit)

This will be updated overtime.

[realcoloride]

Quoting Naozumi520

@realcoloride After contacting w/ user K3YOMI in #1, it turns out that he fixed with adding the user agent. He sent me a copy with the patched package and it was working, but today I checked for it and it returns nothing. Here's his modified version and I have granted his permission. I hope this helps you to confirm what's going on in here.

node_characterai.zip

According to his instructions, there are two step to make it work:

1. fill in the the token (not access token) in chat.js, by replacing (TOKEN HERE) to the token (Keep the TOKEN word) The token can be found in the network tab of dev tool, by searching recent:

2. Only authenticateWithToken will work (with access token), cannot login as guest

I have confirmed that it was working yesterday night, but somehow it stopped working in today. See if it works for you.

This will be my priority as for now, because it seems like it seems to be a root of other issues right now... Thank you for the help though, I will keep you updated

[realcoloride]

Hello!

I located some issues regarding logging in:

• Continuing the chat returns 403, so no chat instance is returned because of a cloudflare error
• Trying to log in with a guest account will cause a cloudflare error
• Trying to log in with an account will fail because for some reason the tokens were reset

So far, I really need a way to go around the cloudflare issues at the moment. EDIT: I'm considering using puppeteer to go around cloudflare.

[realcoloride]

Hello, I've made a puppeteer implentation and it turns out that even with plugins such as the stealth plugin, the request seem to work only when not headless. (so chrome needs to appear, this may be annoying and more resource eating on lower end machines)

Headless Broswer:

Non headless Browser:

I will check out selenium as recommended by some people and friends of mine. In case of last scenario, I might try hosting a proxy that uses puppeteer to intercept requests but that might be bloated and if the server goes offline, many uses can go down too.

I will keep you updated!

[realcoloride]

Alright for now, I will use a temporary fix that uses headful chrome. Until any solution is found, I will port the requests to use pupeteer. This might make requests slower & consume more resources but I don't have a choice at the moment.

[realcoloride]

For 1.1.1, the temporary solution will be to use the pupeteer instance or a temporary proxy I will try to make in order to forward requests. Hopefully if after 1.1.1 a way is found to be headless & performant, 1.1.2 could be pushed.

I will keep you updated!

[moon91210]

What about linux users? like myself. i only have access to a terminal.

[realcoloride]

Yes I had thought of this part too, I work on Windows and don't know how to use pupeteer on Linux. For now, I might add proxy support and pupeteer support until i find a viable solution.

[marcosathanasoulis]

Getting undefined when logging in with a token (guest login did not work).

const response = await chat.sendAndAwaitResponse('Hi', true) TypeError: Cannot read properties of undefined (reading 'sendAndAwaitResponse')

Happy to help testing, I have a proxy service I can try and also have some experience with puppeteer. Thanks for the helpful project!

[feelinSleepy]

Hey! Sorry if this is a redundant question but I just wanted to ask, will 1.1.1 have a fix for the Status 403 error(At least I think that's the error from reading through the old issues, its the same error as the comment above) on Linux? Its what I run my server on, so hopefully you can figure something out haha. Also, thanks for picking up this project!

[realcoloride]

I'm aware of the issues. To answer it all: cloudflare.

Cloudflare blocks the requests at the moment, see up and other issues. Which breaks the lib completely.

At the moment, I'm doing my best to find the best way to go around cloudflare protection.

For such, the best at the moment is a chromium browser using puppeteer.

Le mer. 15 mars 2023 à 05:02, feelin__sleepy @.***> a écrit :

Hey! Sorry if this is a redundant question but I just wanted to ask, will 1.1.1 have a fix for the Status 403 error(At least I think that's the error from reading through the old issues, its the same error as the comment above) on Linux? Its what I run my server on, so hopefully you can figure something out haha. Also, thanks for picking up this project!

— Reply to this email directly, view it on GitHub https://github.com/realcoloride/node_characterai/issues/17#issuecomment-1469293094, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ4WO5LNDLLA75V3HIXPVKDW4E5NBANCNFSM6AAAAAAV2UBEL4 . You are receiving this because you were mentioned.Message ID: @.***>

[Naozumi520]

@realcoloride The way to fix the issue is simple:

const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);

[Naozumi520]

[Naozumi520]

It's now working with headless mode.

[feelinSleepy]

@realcoloride The way to fix the issue is simple:

const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);

Decided to try this real quick before I went to work, no luck, just says "page" in "page.setUserAgent(userAgent);" is undefined. Could definitely be an issue on my end though with my code being wonky.

[Naozumi520]

@realcoloride The way to fix the issue is simple:

const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);

Decided to try this real quick before I went to work, no luck, just says "page" in "page.setUserAgent(userAgent);" is undefined. Could definitely be an issue on my end though with my code being wonky.

This isn't mean for the original package. OP used the puppeteer tricks to bypass the Cloudflare service. And I'm currently using my own code to get the response with puppeteer. So you cannot just add it to the package.

[feelinSleepy]

@realcoloride The way to fix the issue is simple:

const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
 page.setUserAgent(userAgent);

Decided to try this real quick before I went to work, no luck, just says "page" in "page.setUserAgent(userAgent);" is undefined. Could definitely be an issue on my end though with my code being wonky.

This isn't mean for the original package. OP used the puppeteer tricks to bypass the Cloudflare service. And I'm currently using my own code to get the response with puppeteer. So you cannot just add it to the package.

Ohhh. Gotcha, sorry about that

[Naozumi520]

But as soon as OP updated the code with my trick, it should be just working fine!

[realcoloride]

Ok after fiddling around a little more, I've found out how to get around it succesfully.

Thank you for the help @Naozumi520!

[realcoloride]

So far i've been doing great converting all requests:

Except for sending messages which listen for streaming. I need to check if there is a workaround around this.

[Naozumi520]

This is how I'm currently doing:

page.on('response', async (response) => {
                try {
                    if (response.url() == "https://beta.character.ai/chat/streaming/") {
                        const text = await response.text()
                        // the last choke, which is "lastchoke": true
                        let raw = text.trim().split('"replies": [{"text":')[text.trim().split('"replies": [{"text":').length - 1];
                        raw = raw.split(', "id":')[0].trim()
                        const emojiRegex = /\\u([\dA-Fa-f]{4})/g;
                        const reply = raw.substring(1, raw.length - 1).replaceAll('↵', ', ').replaceAll('\\n', '').replaceAll('\\"', '"')
                        const decodedText = reply.replaceAll(emojiRegex, (match, grp) => String.fromCharCode(parseInt(grp, 16)));
                        console.log(decodedText)
                    }
                } catch (e) {
                    console.log(e)
                }
            });

It's very dirty indeed 😓

[realcoloride]

@Naozumi520 I might use your code and modify some stuff until I can find a viable better solution. Then probably push an experimental update after im done doing a few stuff. Thank you again for your help & contribution.

[realcoloride]

@Naozumi520 Hello, Here is my current sample code, could you be more precise please? This code is currently experimental though, but hopefully if it works, I might push it as a temporary update until the cloudflare issue is fixed. (this code does not work for /streaming/)

async request(url, options) {
        const page = this.page;

        const method = 
        (options.method == 'POST' || options.method == undefined || options.method == null
         ? 'POST' : 'GET');

        const body = (method == 'GET' ? {} : options.body);
        const headers = options.headers;

        page.once('request', interceptedRequest => {
            var data = {
                'method': method,
                'postData': body,
                'headers': headers
            };

            try {
                interceptedRequest.continue(data);
            } catch (error) {
                console.log("[node_characterai] Puppeteer - Non fatal error: " + error);
            }
        });
        page.on('response', async (response) => {
            // thank you Naozumi520 for this piece of code
            try {
                if (response.url() == "https://beta.character.ai/chat/streaming/") {
                    const text = await response.text()
                    // the last choke, which is "lastchoke": true
                    let raw = text.trim().split('"replies": [{"text":')[text.trim().split('"replies": [{"text":').length - 1];
                    raw = raw.split(', "id":')[0].trim()
                    const emojiRegex = /\\u([\dA-Fa-f]{4})/g;
                    const reply = raw.substring(1, raw.length - 1).replaceAll('↵', ', ').replaceAll('\\n', '').replaceAll('\\"', '"')
                    const decodedText = reply.replaceAll(emojiRegex, (match, grp) => String.fromCharCode(parseInt(grp, 16)));
                    console.log(decodedText)
                }
            } catch (e) {
                console.log(e)
            }
        });

        let response

        try {
            response = await page.goto(url, { waitUntil: 'networkidle0' });
        } catch (error) {
            console.log("[node_characterai] Puppeteer - " + error)
        }
        return response;
    }

[Naozumi520]

Sure I will check for it! But I have lessons soon, I will report the result later.

[Naozumi520]

Hi! Would you mind sending me your currently testing code, so I can make a test with it?

Edit: My discord is "Naozumi#0233"

[realcoloride]

Hello,

I've added you via discord. I will send it when I'm home. However, please know that the code I've sent you above does not work properly. What I meant is how do I embed your piece of code in mine so it does not crash upon sending messages via /streaming/.

If the issue is fixed, I will push an experimental update.

Le jeu. 16 mars 2023 à 01:07, Naozumi @.***> a écrit :

Hi! Would you mind sending me your currently testing code, so I can make a test with it?

— Reply to this email directly, view it on GitHub https://github.com/realcoloride/node_characterai/issues/17#issuecomment-1471008611, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ4WO5LNFNG7M2ONLRQ45S3W4JKVTANCNFSM6AAAAAAV2UBEL4 . You are receiving this because you were mentioned.Message ID: @.***>

[ThantHtetAung101]

I'm facing a problem [util.inspect.custom](depth, opts) { return this.text; } This line is having an error "Cannot read properties of undefined (reading 'custom')"

I install your package through npm and use your demo code. `const handleAnswer2 = async (text) => { const characterAI = new node_characterai();

    (async () => {
        await characterAI.authenticateAsGuest();

        const characterId = "4ip3o1aigesIvZ95Q1FTFMiC15UVt_oAmUslc3y_D2M" // Discord moderator

        const chat = await characterAI.createOrContinueChat(characterId);
        const response = await chat.sendAndAwaitResponse(text, true)

        console.log(response);
    })();
}`

I really need a fix for this. 

[realcoloride]

What is your node version?

Le jeu. 16 mars 2023 à 11:53, Thant Htet Aung @.***> a écrit :

I'm facing a problem [util.inspect.custom](depth, opts) { return this.text; } This line is having an error "Cannot read properties of undefined (reading 'custom')"

I install your package through npm and use your demo code. `const handleAnswer2 = async (text) => { const characterAI = new node_characterai();

(async () => {
    await characterAI.authenticateAsGuest();

    const characterId = "4ip3o1aigesIvZ95Q1FTFMiC15UVt_oAmUslc3y_D2M" // Discord moderator

    const chat = await characterAI.createOrContinueChat(characterId);
    const response = await chat.sendAndAwaitResponse(text, true)

    console.log(response);
})();

}`

I really need a fix for this.

— Reply to this email directly, view it on GitHub https://github.com/realcoloride/node_characterai/issues/17#issuecomment-1471725269, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ4WO5P5LDNYFOMMA7PXYFDW4LWLPANCNFSM6AAAAAAV2UBEL4 . You are receiving this because you were mentioned.Message ID: @.***>

[ThantHtetAung101]

v 16.16.0

[realcoloride]

Upgrade to node v19 and once done, wait for me to publish the new experimental update (around today)

Le jeu. 16 mars 2023 à 12:02, Thant Htet Aung @.***> a écrit :

v 16.60.0

— Reply to this email directly, view it on GitHub https://github.com/realcoloride/node_characterai/issues/17#issuecomment-1471738862, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ4WO5MT7L6N5JUG7D5HW4LW4LXODANCNFSM6AAAAAAV2UBEL4 . You are receiving this because you were mentioned.Message ID: @.***>

[moon91210]

I see you're still working on the fetch issue? This can be easily fixed by importing the node-fetch library at the top of your files, it's already in the package.json, so users need to do npm install as well. But change it to version 2.6.1 because the newer ones are ESM only so it might not be compatible with other packages and/or node versions. If you do this, users will be able to use older NodeJS versions.

[SycamorexStreet]

I'm facing a problem [util.inspect.custom](depth, opts) { return this.text; } This line is having an error "Cannot read properties of undefined (reading 'custom')"

I install your package through npm and use your demo code. `const handleAnswer2 = async (text) => { const characterAI = new node_characterai();

    (async () => {
        await characterAI.authenticateAsGuest();

        const characterId = "4ip3o1aigesIvZ95Q1FTFMiC15UVt_oAmUslc3y_D2M" // Discord moderator

        const chat = await characterAI.createOrContinueChat(characterId);
        const response = await chat.sendAndAwaitResponse(text, true)

        console.log(response);
    })();
}`

I really need a fix for this. 

I am having the exact same issue, except with sendAndAwaitResponse. I am updated to node v19 and my node_characterai is at v1.1.0. What can I do?

[realcoloride]

Hello, try to install the 'util' package

[realcoloride]

Update: I will use in-page evaluation for /streaming/ or any other functions that need it.

[SycamorexStreet]

Hello, try to install the 'util' package

After installing, the same error is occurring.

[realcoloride]

whats your environment? (node, package versions, os, etc)

[SycamorexStreet]

node: v19.8.1, characterai: v1.1.0, npm: v9.5.1 using windows 11. i'm still new to all of this so let me know if you need more detail, or if i'm missing any info. For clarification, I am not using custom.util, just trying to do a basic chat.

[realcoloride]

Hello, turns out that evaulating is harder than it seemed and does not seem to work well. However, in the meantime, I cant find a way to resolve the problem.

I get the error: net::ERR_ABORTED at https://beta.character.ai/chat/streaming/

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth')

class Requester {
    browser = undefined;
    page = undefined;

    //mode = "puppeteer"; // or proxy

    #initialized = false;

    constructor() {

    }
    isInitialized() {
        return this.#initialized;
    }
    /*isModeValid() {
        const mode = this.mode;
        return mode === "puppeteer" || mode === "proxy";
    }*/

    async initialize() {
        if (!this.isInitialized());

        console.log("[node_characterai] Puppeteer - This is an experimental feature. Please report any issues on github.");

        puppeteer.use(StealthPlugin())
        const browser = await puppeteer.launch({
            headless: true,
            args: [
                '--fast-start',
                '--disable-extensions',
                '--no-sandbox',
                '--disable-setuid-sandbox',
                '--no-gpu',
                '--disable-background-timer-throttling',
                '--disable-renderer-backgrounding',
                '--override-plugin-power-saver-for-testing=never',
                '--disable-extensions-http-throttling',
            ]
        });
        this.browser = browser;

        let page = await browser.pages();
        page = page[0];
        this.page = page;
        await page.setRequestInterception(true);

        page.setViewport({
            width: 1920 + Math.floor(Math.random() * 100),
            height: 3000 + Math.floor(Math.random() * 100),
            deviceScaleFactor: 1,
            hasTouch: false,
            isLandscape: false,
            isMobile: false,
        });
        await page.setJavaScriptEnabled(true);
        await page.setDefaultNavigationTimeout(0);

        const userAgent = 'CharacterAI/1.0.0 (iPhone; iOS 14.4.2; Scale/3.00)';
        await page.setUserAgent(userAgent);

        console.log("[node_characterai] Puppeteer - Done with setup");

    }

    async request(url, options) {
        const page = this.page;

        const method = 
        (options.method == 'POST' || options.method == undefined || options.method == null
         ? 'POST' : 'GET');

        const body = (method == 'GET' ? {} : options.body);
        const headers = options.headers;

        page.once('request', interceptedRequest => {
            var data = {
                'method': method,
                'postData': body,
                'headers': headers
            };

            try {
                interceptedRequest.continue(data);
            } catch (error) {
                console.log("[node_characterai] Puppeteer - Non Fatal " + error);
            }
        });

        let response

        try {
            response = await page.goto(url, { waitUntil: 'networkidle0' });
        } catch (error) {
            console.log("[node_characterai] Puppeteer - " + error)
        }
        return response;
    }

}

module.exports = Requester;

This is my current code for requester. If anyone knows how to fix this, I really cannot manage to.

[saki64]

Would you mind sending me your currently testing code that includes all the fixes so far as well? My discord is reaeq#1242

[ThantHtetAung101]

Hello, try to install the 'util' package

I've tried install util package and the custom error is gone. But the new error is process is not defined in util.js. What should I do?

[drizzle-mizzle]

Having the same problem with a PuppeteerSharp: #43 I have not found any solution yet too, but here are my attemps to solve it, maybe it will give you some thoughts...

[realcoloride]

Hello, After a long time investigating, I might have found a temporary fix. This may make requests longer but at least it will work.

I will keep you updated. EDIT: Sorry for the lack of communication, I've been very busy lately.

[memmaptensor]

Hello! I was able to use selenium with undetected_chromedriver in python to do the sending part of a message query and used the webdriver's cdp listener to listen to Network.responseReceived events with the response URL being the fetch API endpoint (/streaming). I then extract the requestId and listen to Network.loadingFinished events with the same requestId and decoded the base64 response to get the message responses.

It's not incredibly fast, but it works. Though, the octet-stream response could be captured before the loadingFinished event is received... it's just that the response tokens are incomplete and need to be followed up by newer responses from Network.dataReceived events.

[realcoloride]

Hello & Interesting! Could you send the code for what you've explained?

[memmaptensor]

Send me a friend request on discord! rwussiya#5821

[samching]

+1 to having a way in Python to interact with the API!

[andeyeluguo]

so what is the time to update this repository

[realcoloride]

Hello,

Please know that I'm still working on it. Even though I am not as active and able to work as much, please note that I'm extremely busy and haven't had as much time I wanted to work on the package.

I am still, looking for a way to go around, but the only one I have at the moment is to manually push buttons and send a message, which is not practical for conversations.

Thus, my optimal way of doing it would be using only requests, but it would require to capture the packet and such... I had tried working with Evaluating but it didn't work and I've kept trying experimental ways like for example waiting each x ms to see if message was sent then return, but it isn't efficient.

So if you have any resource or suggestions in the meantime please let me know.

But again, I will keep you updated. Thanks for your patience.

On Sun, 26 Mar 2023, 03:14 andeyeluguo, @.***> wrote:

so what is the time to update this repository

— Reply to this email directly, view it on GitHub https://github.com/realcoloride/node_characterai/issues/17#issuecomment-1483963874, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ4WO5OSCDWCMOZML4ZWDQ3W56J57ANCNFSM6AAAAAAV2UBEL4 . You are receiving this because you were mentioned.Message ID: @.***>

[AzzaDeveloper]

Thank you for your work! Patiently waiting for it to get fixed ^^

[rhenthar]

Definitely waiting. Although I've got it somewhat working with selenium. But not quite. I may have created another issue entirely myself. Cloudflare, yeah. tokens. no guest mode, make sure to not include double quotes for the access token. stick it all in a .env ... I'm so excited to get some of this working.

[drizzle-mizzle]

Got it working in my C# project: CharacterAI.Net