Wrong character encoding in responses

[CosmicMac]

Before submitting your bug report

• [X] I believe this is a bug. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that reports the same bug
• [X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 10 & 11
- Continue: 0.0.33
- IDE:Jetbrains (C-Lion/PHP-Storm/DataGrip, latest versions, new UI)
- Server: docker ollama:latest

Description

Extended characters in responses are badly encoded (eg. "é" instead of "é"). Encoding is OK in direct responses from ollama when prompting with terminal.

To reproduce

1/ Select any gemma model 2/ Prompt "Translate elegant to french"

Log output

No response

[sestinj]

I tried this in both VS Code and Intellij and found that the encoding looked as expected (though Gemma gives interesting answers).

I'm wondering if this might be the model literally outputting "é" due to something it saw in its dataset. If you say something like "repeat after me: 'é'", can you get it to output the correct encoding?

[CosmicMac]

gemma on acid :)

Unfortunately same problem with the repeat prompt:

A quick test in console:

[CosmicMac]

For me, it looks like a double utf-8 encoding. As I'm using a french OS maybe there is auto encoding occurring before forced encoding (or the other way round)? It would explain why you can't reproduce the glitch on your system.

[sestinj]

Ah this makes sense. Is this built into the OS, or might there be a setting that I could change in order to simulate this?

[CosmicMac]

Unfortunately I have no idea :(