search

danswer-ai / danswer

Gen-AI Chat for Teams - Think ChatGPT if it had access to your team's unique knowledge.
https://docs.danswer.dev/
Other
10.37k stars 1.25k forks source link

Slackbot missing persona__prompt entry when using document sets #1584

Closed pbrumblay closed 3 months ago

pbrumblay commented 3 months ago

Steps to reproduce:

  1. Create a slackbot
  2. Constrain the slackbot to a document set instead of using a persona
  3. Attempt to chat with DanswerBot through a configured channel.

Expected result:

DanswerBot responds with an answer constrained to the document set.

Actual result:

danswer/danswerbot/slack/listener.py throws the following exception:

$ python danswer/danswerbot/slack/listener.py
06/06/2024 02:37:00 PM          listener.py 401 : Verifying query preprocessing (NLTK) data is downloaded
06/06/2024 02:37:00 PM     search_runner.py  45 : stopwords is already downloaded.
06/06/2024 02:37:00 PM     search_runner.py  48 : Downloading wordnet...
06/06/2024 02:37:00 PM     search_runner.py  50 : wordnet downloaded successfully.
06/06/2024 02:37:00 PM     search_runner.py  45 : punkt is already downloaded.
06/06/2024 02:37:03 PM          listener.py 384 : Listening for messages from Slack...
06/06/2024 02:43:00 PM    handle_message.py 120 : [Channel ID: C076YT500CB] Scheduled feedback reminder disabled...
06/06/2024 02:43:00 PM    handle_message.py 257 : [Channel ID: C076YT500CB] Found slack bot config for channel. Restricting bot to use document sets: ['django'], validity checks enabled: NA
06/06/2024 02:43:01 PM               api.py  40 : [Channel ID: C076YT500CB] Persona does not have any prompts - this should never happen, retrying in 0.25 seconds...
06/06/2024 02:43:01 PM               api.py  40 : [Channel ID: C076YT500CB] Persona does not have any prompts - this should never happen, retrying in 0.5 seconds...
06/06/2024 02:43:02 PM               api.py  40 : [Channel ID: C076YT500CB] Persona does not have any prompts - this should never happen, retrying in 1.0 seconds...
06/06/2024 02:43:03 PM               api.py  40 : [Channel ID: C076YT500CB] Persona does not have any prompts - this should never happen, retrying in 2.0 seconds...
06/06/2024 02:43:06 PM    handle_message.py 401 : [Channel ID: C076YT500CB] Unable to process message - did not successfully answer in 5 attempts
Traceback (most recent call last):
  File "/Users/pbrumblay/src/danswer/backend/danswer/danswerbot/slack/handlers/handle_message.py", line 390, in handle_message
    answer = _get_answer(
             ^^^^^^^^^^^^
  File "/Users/pbrumblay/src/danswer/.venv/lib/python3.11/site-packages/decorator.py", line 232, in fun
    return caller(func, *(extras + args), **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pbrumblay/src/danswer/.venv/lib/python3.11/site-packages/retry/api.py", line 73, in retry_decorator
    return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pbrumblay/src/danswer/.venv/lib/python3.11/site-packages/retry/api.py", line 33, in __retry_internal
    return f()
           ^^^
  File "/Users/pbrumblay/src/danswer/backend/danswer/danswerbot/slack/handlers/handle_message.py", line 83, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pbrumblay/src/danswer/backend/danswer/danswerbot/slack/handlers/handle_message.py", line 349, in _get_answer
    answer = get_search_answer(
             ^^^^^^^^^^^^^^^^^^
  File "/Users/pbrumblay/src/danswer/backend/danswer/one_shot_answer/answer_question.py", line 308, in get_search_answer
    for packet in results:
  File "/Users/pbrumblay/src/danswer/backend/danswer/one_shot_answer/answer_question.py", line 139, in stream_answer_objects
    raise RuntimeError(
RuntimeError: Persona does not have any prompts - this should never happen
^CTraceback (most recent call last):
  File "/Users/pbrumblay/src/danswer/backend/danswer/danswerbot/slack/listener.py", line 436, in <module>
    Event().wait(timeout=60)
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 629, in wait
    signaled = self._cond.wait(timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/python@3.11/3.11.9/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 331, in wait
    gotit = waiter.acquire(True, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Analysis

create_slack_bot_persona does not properly set a prompt for the document constrained slackbot. https://github.com/danswer-ai/danswer/blob/main/backend/danswer/db/slack_bot_config.py#L34

    # create/update persona associated with the slack bot
    persona_name = _build_persona_name(channel_names)
    persona = upsert_persona(
        user=None,  # Slack Bot Personas are not attached to users
        persona_id=existing_persona_id,
        name=persona_name,
        description="",
        num_chunks=num_chunks,
        llm_relevance_filter=True,
        llm_filter_extraction=True,
        recency_bias=RecencyBiasSetting.AUTO,
        prompts=None, ### This should not be none
        document_sets=document_sets,
        llm_model_provider_override=None,
        llm_model_version_override=None,
        starter_messages=None,
        is_public=True,
        default_persona=False,
        db_session=db_session,
        commit=False,
    )

The above code creates the following configuration in postgres: 

postgres=# select * from slack_bot_config;
 id | persona_id |                                      channel_config                                      | response_type 
----+------------+------------------------------------------------------------------------------------------+---------------
  3 |          3 | {"channel_names": ["dbot2-django"], "respond_to_bots": false, "respond_tag_only": false} | CITATIONS
(1 row)

postgres=# select id, name from persona;
 id |               name                
----+-----------------------------------
  0 | Danswer
 -1 | GPT
 -2 | Paraphrase
  3 | __slack_bot_persona__dbot2-django
(4 rows)

postgres=# select * from persona__prompt;
 persona_id | prompt_id 
------------+-----------
          0 |         0
         -1 |         1
         -2 |         3
(3 rows)
Weves commented 3 months ago

Hey @pbrumblay, thanks so much for pointing this out! Just fixed this in https://github.com/danswer-ai/danswer/pull/1588.

Let me know if you're still running into any issues.