[Obs AI Assistant] Improve LLM re-rank consistency

[miltonhultgren]

Summary

@LucaWintergerst has reported a odd behaviour of the LLM, where given a certain prompt and certain Knowledge base content it will sometimes rank a document high (7) and sometimes low (0).

We haven't yet been able to reproduce this but the issue persists across small tweaks to the prompt.

This has real consequences as it makes the recall step (and thus the following conversation) very unpredictable.

AC

• (Ideally) The conditions for this behaviour is identified and reproduced
• Tweaks are made to the system message so that variance in score is limited (6-7, 0-1) across several runs

Test data

Current system message:

Given the following question, score the documents that are relevant to the question. on a scale from 0 to 7,
    0 being completely irrelevant, and 7 being extremely relevant. Information is relevant to the question if it helps in
    answering the question. Judge it according to the following criteria:

    - The document is relevant to the question, and the rest of the conversation
    - The document has information relevant to the question that is not mentioned,
      or more detailed than what is available in the conversation
    - The document has a high amount of information relevant to the question compared to other documents
    - The document contains new information not mentioned before in the conversation

Prompt:

Is there a runbook for the cartservice-otel?

Knowledge base:

[
  {
    id: 'elastic/observability-aiops/ai_assistant/runbooks/slos/cartservice-runbook.md',
    text: '{"mode":"100644","path":"ai_assistant/runbooks/slos/cartservice-runbook.md","extension":".md","size":165,"name":"cartservice-runbook.md","id":"elastic/observability-aiops/ai_assistant/runbooks/slos/cartservice-runbook.md","type":"blob","body":"This is the runbook for the cartservice-otel error. If the cartservice is experiencing errors, do the following: - call Luca, he will fix it - grab a cup of coffee"}',
    score: 10
  }
]

[elasticmachine]

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

[miltonhultgren]

If anyone has any idea about what might influence this behaviour so that we can try to reproduce it, that would be greatly helpful.

[grabowskit]

Need to find a way to reproduce.