Mock-AI-Interviewer / mock-ai-interviewer

A mock AI interviewer that asks you questions and records your answers 🧠
1 stars 0 forks source link

Fix the Setance Splitting for text from LLM #74

Open BossCodersHQ opened 7 months ago

BossCodersHQ commented 7 months ago
async def text_chunker(chunks):
    """Split text into chunks, ensuring to not break sentences."""
    splitters = (".", ",", "?", "!", ";", ":", "—", "-", "(", ")", "[", "]", "}", " ")
    buffer = ""

    async for text in chunks:
        if buffer.endswith(splitters):
            yield buffer + " "
            buffer = text
        elif text.startswith(splitters):
            yield buffer + text[0] + " "
            buffer = text[1:]
        else:
            buffer += text

    if buffer:
        yield buffer + " "