di-sukharev / opencommit

Generate conventional git commit messages with AI in 1 second 🤯🔫
https://www.npmjs.com/package/opencommit
MIT License
5.98k stars 318 forks source link

[Bug]: Error: The text contains a special token that is not allowed: <|endoftext|> #377

Closed hochmannA1 closed 1 month ago

hochmannA1 commented 1 month ago

Opencommit Version

3.0.17

Node Version

v20.12.2

NPM Version

10.5.0

What OS are you seeing the problem on?

Other Linux Distro

What happened?

When using oco to generate commit message with files containing the string "<|endoftext|>" oco fails with the following error:

◓ Generating commit message..
│ └ ✖ Error: The text contains a special token that is not allowed: <|endoftext|>

Expected Behavior

opencommit should generate a commit message even though "forbidden" strings are inside some files

Current Behavior

See in "What happened"

Possible Solution

It could be possible to replace all forbidden tokens with their escaped counterpart or remove them from the text which is getting sent to the AI

Steps to Reproduce

No response

Relevant log output

No response

di-sukharev commented 1 month ago

this must be an openai or other provider constraint, can not really do anything. i think openai makes this to prevent prompt hacking