search

niieani / gpt-tokenizer

The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o. Port of OpenAI's tiktoken with additional features.
https://gpt-tokenizer.dev
MIT License
434 stars 35 forks source link

encodeChat: lineToEncode.match is not a function #47

Closed BennyAlex closed 2 weeks ago

BennyAlex commented 3 weeks ago

I get this error when using this als input chat:

     [
            {
                role: 'system',
                content:
                    'Please help with this error.'
            },
            {
                role: 'user',
                content: [
                    {
                        type: 'text',
                        text: 'The image on my website is not rendered'
                    }
                ]
            }
        ]
niieani commented 2 weeks ago

content expects a string, not an array. TypeScript's type checking could have helped you discover this problem.

BennyAlex commented 2 weeks ago

Thats not correct. When using images we must use an array and have {type: text} as well as {type: image_url} inside it.

niieani commented 2 weeks ago

It is correct for what this library supports. If you're after tokenizing images, that is not yet supported: https://github.com/niieani/gpt-tokenizer/issues/41 PRs welcome.