search

truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
2.03k stars 176 forks source link

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly #470

Closed niloychak2022 closed 11 months ago

niloychak2022 commented 11 months ago

I am running in ubuntu 20.04 machine. I have integrated trulens with llama-index for quering. Here is the code - 

from multiprocessing.managers import BaseManager
from trulens_eval import TruLlama, Feedback, Tru, feedback
from sqlalchemy import URL
from llama_index import SimpleWebPageReader
from llama_index import VectorStoreIndex
import numpy as np

def initialize_trulens():
    global feedbacks

    # Initiating the trulens dashboard
    url_object = URL.create(
        "postgresql+psycopg2",
        username=os.environ.get("SUPERBASE_USERNAME"),
        password=os.environ.get("SUPERBASE_PASSWORD"), 
        host=os.environ.get("SUPERBASE_HOST"),
        port=os.environ.get("SUPERBASE_PORT"),
        database=os.environ.get("SUPERBASE_DATABASE")
    )
    tru = Tru(database_url= url_object)
    tru.run_dashboard()

    # Initialize Huggingface-based feedback function collection class:
    hugs = feedback.Huggingface()
    openai = feedback.OpenAI()
    # Define a language match feedback function using HuggingFace.
    f_lang_match = Feedback(hugs.language_match).on_input_output()
    # By default this will check language match on the main app input and main app
    # output.

    # Question/answer relevance between overall question and answer.
    f_qa_relevance = Feedback(openai.relevance).on_input_output()

    # Question/statement relevance between question and each context chunk.
    f_qs_relevance = Feedback(openai.qs_relevance).on_input().on(
        TruLlama.select_source_nodes().node.text
    ).aggregate(np.min)

    feedbacks = [f_lang_match, f_qa_relevance, f_qs_relevance]

def query():
    documents = SimpleWebPageReader(html_to_text=True).load_data(
         ["http://paulgraham.com/worked.html"]
    )
    index = VectorStoreIndex.from_documents(documents)

    query_engine = index.as_query_engine()
    tru_query_engine_recorder = TruLlama(
        query_engine,
        app_id='LlamaIndex_App1',
        feedbacks=feedbacks
    )
    response = tru_query_engine_recorder.query("What did the author do growing up?")
    print(response)
    return response

if __name__ == "__main__":
    print("server started...")
    initialize_trulens()
    manager = BaseManager(('', 5602), b'password')
    manager.register('query_index', query)
    server = manager.get_server()
    server.serve_forever()

After running the server it is working good. I can see the dashboard of trulens. But after some time I am getting the following error and trulens dashboard gets disconnected from database.

Tru was already initialized. Cannot change database_url=postgresql+psycopg2://<database_username>:<database_password>@<database_host>:<database_port>/<database_name> or database_file=None .

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

2023-09-26 14:01:57.225 MediaFileHandler: Missing file 16a57003a519fd6aa13a2f733bd4fa522ba8e58545862765c5d0ce92.png

2023-09-26 14:01:57.235 MediaFileHandler: Missing file 6b6481b1f4db67286783dec664cc63b2153e3c7c262a3bff8a328923.png

2023-09-26 14:01:57.236 MediaFileHandler: Missing file 37d4f4b62a7b6acf699e54df90a7bf371cbab2d17fca60c9372fd503.png

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

Tru was already initialized. Cannot change database_url=postgresql+psycopg2://<database_username>:<database_password>@<database_host>:<database_port>/<database_name> or database_file=None .

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

2023-09-26 14:02:14.752 MediaFileHandler: Missing file 16a57003a519fd6aa13a2f733bd4fa522ba8e58545862765c5d0ce92.png

2023-09-26 14:02:14.753 MediaFileHandler: Missing file 6b6481b1f4db67286783dec664cc63b2153e3c7c262a3bff8a328923.png

2023-09-26 14:02:14.754 MediaFileHandler: Missing file 37d4f4b62a7b6acf699e54df90a7bf371cbab2d17fca60c9372fd503.png

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

Tru was already initialized. Cannot change database_url=postgresql+psycopg2://<database_username>:<database_password>@<database_host>:<database_port>/<database_name> or database_file=None .

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

Tru was already initialized. Cannot change database_url=postgresql+psycopg2://<database_username>:<database_password>@<database_host>:<database_port>/<database_name> or database_file=None .

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))

Tru was already initialized. Cannot change database_url=postgresql+psycopg2://<database_username>:<database_password>@<database_host>:<database_port>/<database_name> or database_file=None .

2023-09-26 14:18:28.504 Session with id 36790e30-3097-4309-acd6-b1d908187e66 is already connected! Connecting to a new session.

Tru was already initialized. Cannot change database_url=postgresql+psycopg2://<database_username>:<database_password>@<database_host>:<database_port>/<database_name> or database_file=None .

2023-09-26 14:18:28.840 Uncaught app exception

Traceback (most recent call last):

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context

    self.dialect.do_execute(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute

    cursor.execute(statement, parameters)

psycopg2.OperationalError: server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/usr/local/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 541, in _run_script

    exec(code, module.__dict__)

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/Leaderboard.py", line 141, in <module>

    main()

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/Leaderboard.py", line 137, in main

    streamlit_app()

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/Leaderboard.py", line 51, in streamlit_app

    df, feedback_col_names = lms.get_records_and_feedback([])

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/database/utils.py", line 60, in wrapper

    callback(*args, **kwargs)

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/database/sqlalchemy_db.py", line 44, in <lambda>

    run_before(lambda self, *args, **kwargs: check_db_revision(self.engine)),

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/database/utils.py", line 112, in check_db_revision

    if is_legacy_sqlite(engine):

  File "/usr/local/lib/python3.10/site-packages/trulens_eval/database/utils.py", line 82, in is_legacy_sqlite

    tables = list(inspector.get_table_names())

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 396, in get_table_names

    return self.dialect.get_table_names(

  File "<string>", line 2, in get_table_names

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/reflection.py", line 97, in cache

    ret = fn(self, con, *args, **kw)

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/base.py", line 3368, in get_table_names

    return self._get_relnames_for_relkinds(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/base.py", line 3364, in _get_relnames_for_relkinds

    return connection.scalars(query).all()

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1344, in scalars

    return self.execute(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1412, in execute

    return meth(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection

    return connection._execute_clauseelement(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1635, in _execute_clauseelement

    ret = self._execute_context(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1844, in _execute_context

    return self._exec_single_context(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1984, in _exec_single_context

    self._handle_dbapi_exception(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2339, in _handle_dbapi_exception

    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1965, in _exec_single_context

    self.dialect.do_execute(

  File "/usr/local/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 921, in do_execute

    cursor.execute(statement, parameters)

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

server closed the connection unexpectedly

    This probably means the server terminated abnormally

    before or while processing the request.

[SQL: SELECT pg_catalog.pg_class.relname 

FROM pg_catalog.pg_class JOIN pg_catalog.pg_namespace ON pg_catalog.pg_namespace.oid = pg_catalog.pg_class.relnamespace 

WHERE pg_catalog.pg_class.relkind = ANY (ARRAY[%(param_1)s, %(param_2)s]) AND pg_catalog.pg_class.relpersistence != %(relpersistence_1)s AND pg_catalog.pg_table_is_visible(pg_catalog.pg_class.oid) AND pg_catalog.pg_namespace.nspname != %(nspname_1)s]

[parameters: {'param_1': 'r', 'param_2': 'p', 'relpersistence_1': 't', 'nspname_1': 'pg_catalog'}]

(Background on this error at: https://sqlalche.me/e/20/e3q8)

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6826: RuntimeWarning: All-NaN slice encountered

  xmin = min(xmin, np.nanmin(xi))

/usr/local/lib/python3.10/site-packages/matplotlib/axes/_axes.py:6827: RuntimeWarning: All-NaN slice encountered

  xmax = max(xmax, np.nanmax(xi))
rshih32 commented 11 months ago

@joshreini1 @piotrm0 This looks relevant; https://stackoverflow.com/questions/55457069/how-to-fix-operationalerror-psycopg2-operationalerror-server-closed-the-conn

Probably need to modify the configurations

niloychak2022 commented 11 months ago

@joshreini1 @piotrm0 @rshih32

I have seen this article. But here I am no idea how to configure psycopg and pass those parameters. Here this package is creating the connection. If you have any idea of how to configure this, that would be great.

piotrm0 commented 11 months ago

Does this happen if you use a local sqlite db?

niloychak2022 commented 11 months ago

Does this happen if you use a local sqlite db?

No it does not happen if I use a local sqlite db.But I need to store the data on a database. @piotrm0