truera / trulens

Evaluation and Tracking for LLM Experiments
https://www.trulens.org/
MIT License
2.05k stars 178 forks source link

quick tutorial: Segmentation fault: 11 #1356

Closed jonathanbouchet closed 1 month ago

jonathanbouchet commented 1 month ago

Bug Description I'm trying the quick start: https://www.trulens.org/trulens_eval/getting_started/quickstarts/quickstart/

To Reproduce

from trulens_eval import Tru
# Segmentation fault: 11
# ename: 'ValueError',
# evalue: 'Expected embeddings to be a list, got str',

Expected behavior expected the line to work

Relevant Logs/Tracebacks Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. If the issue is related to the TruLens dashboard, please also include a screenshot.

Environment:

Additional context I tried through VScode, jupyter-notebook and terminal. All crashed with a Segmentation Fault 11

dosubot[bot] commented 1 month ago

Hey there, @jonathanbouchet! I'm here to help you out with any bugs, questions, or contributions you may have. Let's tackle this issue together!

To resolve the Segmentation fault: 11 error caused by a ValueError when using TruLens, ensure that the database engine is properly initialized before performing any operations. The ValueError might be raised due to an uninitialized database engine, which can lead to unexpected behavior and potentially cause a segmentation fault.

Here is a snippet from the provided code that checks if the database engine is initialized and raises a ValueError if it is not:

def check_db_revision(self):
    """See [DB.check_db_revision][trulens.core.database.base.DB.check_db_revision]."""

    if self.engine is None:
        raise ValueError("Database engine not initialized.")

    alembic_check_db_revision(self.engine, self.table_prefix)

Ensure that the database engine is initialized correctly by calling the _reload_engine method during the initialization of the SQLAlchemyDB class:

def __init__(
    self,
    redact_keys: bool = mod_db.DEFAULT_DATABASE_REDACT_KEYS,
    table_prefix: str = mod_db.DEFAULT_DATABASE_PREFIX,
    **kwargs: Dict[str, Any],
):
    super().__init__(
        redact_keys=redact_keys,
        table_prefix=table_prefix,
        orm=mod_orm.make_orm_for_prefix(table_prefix=table_prefix),
        **kwargs,
    )
    self._reload_engine()
    if is_memory_sqlite(self.engine):
        warnings.warn(
            UserWarning(
                "SQLite in-memory may not be threadsafe. "
                "See https://www.sqlite.org/threadsafe.html"
            )
        )

Make sure that the _reload_engine method is correctly setting up the engine and session:

def _reload_engine(self):
    self.engine = sa.create_engine(**self.engine_params)
    self.session = sessionmaker(self.engine, **self.session_params)

By ensuring that the database engine is properly initialized, you can avoid the ValueError and prevent the Segmentation fault: 11 error [1].

Additionally, ensure that the embeddings passed to the TruLens functions are in the correct format. The embeddings should be a 2D numpy array, reshaped to have the first dimension represent the number of samples:

query_embed = np.asarray(
    self._embed_model.get_query_embedding(query)
).reshape(
    1, -1
)  # sklearn expects 2d array (first dimension number of samples)

document_embed = np.asarray(
    self._embed_model.get_text_embedding(document)
).reshape(
    1, -1
)  # sklearn expects 2d array (first dimension number of samples)

This format ensures compatibility with scikit-learn's distance functions, which expect a 2D array input [2].

To continue talking to Dosu, mention @dosu.

jonathanbouchet commented 1 month ago

I think I found the reason / faulty package I had a similar issue with another RAG evaluation framework where the datasets package is required. I tried to import this package and got the same segmentation fault

These 2 bug reports suggested the version of pyarrow was the reason datasets could not be loaded:

Reverting to pyarrow==16.0.0 instead of 17.0.0 and I'm able to run these 2 imports:

from trulens_eval import Feedback, Select
from trulens_eval.feedback.provider.openai import OpenAI
jonathanbouchet commented 1 month ago

switching to pyarrow==16.0.0 fixed my environment issue