Rare Flashcard Generation Failures

[mikhailocampo]

Consider the following snippet from features/Dynamo/tools.py

def generate_flashcards(summary: str, verbose=False) -> list:
    # Receive the summary from the map reduce chain and generate flashcards
    parser = JsonOutputParser(pydantic_object=Flashcard)

    if verbose: logger.info(f"Beginning to process summary")

    template = read_text_file("prompt/dynamo-prompt.txt")
    examples = read_text_file("prompt/examples.txt")

    cards_prompt = PromptTemplate(
        template=template,
        input_variables=["summary", "examples"],
        partial_variables={"format_instructions": parser.get_format_instructions()}
    )

    cards_chain = cards_prompt | model | parser

    try:
        response = cards_chain.invoke({"summary": summary, "examples": examples})
    except Exception as e:
        logger.error(f"Failed to generate flashcards: {e}")
        raise HTTPException(status_code=500, detail=f"Failed to generate flashcards from LLM")

    return response

Where we try except the execution of the chain. There is a rare chance (not documented the rate at which this happens yet) where the parser FAILS and then returns nothing to the user!

There already is a long latency (about 20-30seconds) to use the Map Reduce algorithm and then use the LLM to extract information. Perhaps the summary defined as summarize_transcript within the same tools.py can cache the result and run a retry for the response chain in order to preserve the summarization time and only take <2.5seconds or so to generate new flashcards on the summary.

[mikhailocampo]

Had a new one, I believe this is also an invalid output from the model.

Currently the process: Map Reduce Algo -> Generate Response -> Format

Where the error is happening from the Format itself