Google Generative AI - Remove text formatting to improve TTS

[r1chards]

The problem

Is it possible to remove asterisk characters from Google Generative AI responses to a question, when it's spoken out aloud via a speaker?

When a long responses is given sometimes it has 5 Asterix wrapping a heading using markdown language, then this is spoke out aloud when using Home Assistant Cloud as the TTS agent. (e.g. "Asterix, Asterix, Asterix, Heading, Asterix, Asterix"). Gets a bot annoying after a few times. I've tried updating the Personality in the Google Generative AI, "Instructions" with no luck in the configurations Examples: Remove all characters except alphanumeric and spaces. No * No Asterix Never use markdown

Here's a example of the output.

What version of Home Assistant Core has the issue?

core-2024.7.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Google Generative AI

Link to integration documentation on our website

https://www.home-assistant.io/integrations/google_generative_ai_conversation

Diagnostics information

Debug Assistant.txt

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `google_generative_ai_conversation` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign google_generative_ai_conversation` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

google_generative_ai_conversation documentation google_generative_ai_conversation source _{^{(message by IssueLinks)}}

[tronikos]

Which model are you using? What's your instructions? Can you post a screenshot of your options? 2024.7 reduced these asterisks for Gemini 1.5 but not for 1.0 which isn't possible unless we postprocess the response ourselves inside the integration.

[r1chards]

Which model are you using? What's your instructions? Can you post a screenshot of your options? 2024.7 reduced these asterisks for Gemini 1.5 but not for 1.0 which isn't possible unless we postprocess the response ourselves inside the integration.

Thanks for coming back to me. I'm running core-2024.7.2. Here's a copy of the instructions. I'm using "Gemini 1.5 flash latest". Then asking a question. "Tell me about the first Moon landing". A simple response is given. I ask to "continue". The AI then response with more details but has a load of Asterix in the response.

[tronikos]

The default instructions:

You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Answer in plain text. Keep it simple and to the point.

work fine for me for most of the queries. I get asterisks much less frequently since 2024.7

[r1chards]

That's fine, but soon as you ask for a little more detail it sends a response with a list and bold headings in the markdown language.

Which is crazy when it's spoken out aloud. In the short example below it would say "Asterix" x 19 times.

I could take this up with Nabucasa Support and see if they can work something into the Home Assistant Cloud, Assist TTS engine that I'm using?

-

[truongbber]

I don't understand why Google Generative AI doesn't work with custom_sentences, I tried deleting and reinstalling it but it still can't communicate and use custom commands in custom_sentences

[truongbber]

Cũng được thôi, nhưng ngay khi bạn yêu cầu thêm một chút chi tiết, nó sẽ gửi phản hồi kèm theo danh sách và tiêu đề in đậm bằng ngôn ngữ đánh dấu.

Thật điên rồ khi nói to. Trong ví dụ ngắn dưới đây, nó sẽ nói "Asterix" x 19 lần.

Tôi có thể liên hệ với bộ phận Hỗ trợ của Nabucasa và xem liệu họ có thể đưa tính năng này vào Home Assistant Cloud, công cụ Assist TTS mà tôi đang sử dụng không?

-

To fix this, it's best to use stt-azure, microsof's stt-azure will remove the * character before converting it into speech.

[tronikos]

In you prompt did your add this? Answer in plain text. Maybe try changing it to: Always answer in plain text.

Post your updated prompt and exact requests that led to this response. As I said earlier, I get asterisks much less frequently now. I was also very annoyed with them before 2024.7

[r1chards]

Here's the instructions:

You are a voice assistant for Home Assistant. Answer questions about the world truthfully. Always answer in plain text. Keep it simple and to the point.

At the prompt: Tell me about the planet Mars

A short paragraph is give:

Mars is the fourth planet from the Sun and is known as the "Red Planet" because of its reddish appearance due to iron oxide on its surface. It's a cold, desert world with a thin atmosphere. It has two moons, Phobos and Deimos, and is a popular target for space exploration.

Then in response I ask: In more detail please

A longer more detailed answer is given:

Mars is a fascinating planet with a lot to offer: Size and Distance: It's about half the size of Earth and is about 142 million miles from the Sun (Earth is 93 million miles away). Atmosphere: Mars has a very thin atmosphere, composed mainly of carbon dioxide. It's about 100 times thinner......

[tronikos]

Try having this to your prompt: Answer in a simple paragraph format. Your answers are spoken. It seems to reduce the markdown.

[r1chards]

Tried that and a load of others within the instructions. I seem to have a better response from Gemini 1.5 Pro, it's still not prefect when you ask to "continue with more details" but generally it seems better.

Here's my instructions:

You are a voice assistant for Home Assistant and a Teacher.
Answer in straightforward, longer text in a simple paragraph format.
Never use markdown formatting.
Never use special formatting.
Never emphasis text.
Important, your answers are spoken.

Thank you for the help please close the ticket.