how to have longer output using gpt-3.5-turbo-16k model

[jaodei]

I manually changed the model in settings.js in node-chatgpt-api container to gpt-3.5-turbo-16k model and also specified max_token to say 12000 tokens. See below. However, when I asked the bot to create a 5000 words story, each time the generated text is quite short, maybe around 700 words. Is there elsewhere limiting the output length?

        payload.update(
            {
                "clientOptions": {
                    "clientToUse": "chatgpt",
                    "openaiApiKey": self.openai_api_key,
                    "modelOptions": {
                        "temperature": self.temperature,
                        "max_tokens": 12000,
                        "model": self.chatgpt_model,
                    },
                    "options": {
                        "maxContextTokens": 16384,
                    },
                }
            }

[hibobmaster]

Enable debug mode and get the payload/response from console https://github.com/waylaidwanderer/node-chatgpt-api/blob/217c189cbbc7749da0f86631ca0f6dd7f5a43cb0/settings.example.js#L37

[jaodei]

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant?

Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | |_

[jaodei]

not showing anything about the payload/reponse from console after enabling debug mode. wierd. I am using your forked docker container at https://github.com/hibobmaster/node-chatgpt-api. relevant?

Jun 24 10:24:22 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:22,759 - INFO - prompt: write a 4000 words story about a boy likes fishing Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | 2023-06-24 02:24:40,786 - INFO - Message received in room chatgpt-16k Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | ChatGPT-16K| Once upon a time, in a small coastal town called Seaview, there lived a young boy named Daniel. Daniel had always been fascinated by the sea and all its wonders, but his true passion lay in fishing. From an early age, he would spend hours on end with his father at the local pier, casting lines into the deep blue, hoping for a bite. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | Daniel's love for fishing was not just about catching fish; it went much deeper than that. He found solace in the rhythmic motion of casting the line, the gentle lapping of the waves against the shore, and the peacefulness that enveloped him as he waited patiently for a tug on his line. Fishing became his escape from the chaos of everyday life. Jun 24 10:24:40 arm1 -16k[2686543]: bot-chatgpt | |_

Sorry my fault. I was looking at the log info of matrix_chat_bot, not node-chatgpt-bot.

[jaodei]

Got the log as below. It appears setting max_tokens correctly to 12000. why can't it produce 4000 words as requested? An issue of openai itself?

_Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | https://api.openai.com/v1/chat/completions Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | model: 'gpt-3.5-turbo-16k-0613', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | temperature: 0.8, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | max_tokens: 12000, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | top_p: 1, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | presence_penalty: 1, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | stop: [ '||>', '\nUser:', '<|diff_marker|>' ], Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | messages: [ Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | name: 'instructions', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>Instructions:\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'You are ChatGPT, a large language model trained by OpenAI. Respond conversationally.\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'Current date: June 24, 2023\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '\n' Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | }, Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | { Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | role: 'system', Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | content: '||>User:\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | 'write a 4000 words story about a boy likes fishing\n' + Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | '||>ChatGPT:\n' Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | } Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgpt_api | ] Jun 24 13:53:47 arm1 node_chatgpt_api[2740547]: node_chatgptapi | }

[hibobmaster]

I use another tool and got the same result: prompt: write a 4000 words story about a boy likes fishing

Maybe you could try openai playground to verify it. https://platform.openai.com/playground?mode=chat&model=gpt-3.5-turbo-16k

[jaodei]

wierd. On playground, the maximum token is only 2048 even for gpt-3.5-turbo-16k model.

[hibobmaster]

Some other people have the same question. https://www.reddit.com/r/ChatGPT/comments/10ddllh/got_tips_for_how_to_get_chatgpt_to_write_a_long/