LiteralAI step_to_stepdict doesn't forward metadata

[andrePankraz]

Describe the bug We want to store additional metadata to a Step (e.g. current RAG context or document infos). We want later to reconstruct a Thread with such extra info.

Such metadata is stored and retrieved properly over LiteralAI datalayer, but the step-mapping from the API-fetched step-object to Chainlit-Step-objects "forgets" to map this additional metadata.

To Reproduce 1) Add custom metadata to Step during on_message. 2) Resume some old thread. 3) Metadata in on_resume is missing in Thread/Step

Expected behavior Additional Step-Metadata in on_resume must exist.

Additional context See https://github.com/Chainlit/chainlit/blob/22d198d24d17f04ee65a0d2bcffbd2c1595f32ae/backend/chainlit/data/literalai.py#L83 Some special metadata like language is extracted and added to Step explicitely, other matdata is "forgotten".

[dosubot[bot]]

To address the issue of missing metadata in the step_to_stepdict function, you can modify the function to ensure that all additional metadata from a Step object is retained and mapped to Chainlit Step objects. Here's how you can update the function:

@classmethod
def step_to_stepdict(cls, step: LiteralStep) -> "StepDict":
    metadata = step.metadata or {}
    input = (step.input or {}).get("content") or (
        json.dumps(step.input) if step.input and step.input != {} else ""
    )
    output = (step.output or {}).get("content") or (
        json.dumps(step.output) if step.output and step.output != {} else ""
    )

    user_feedback = (
        next(
            (
                s
                for s in step.scores
                if s.type == "HUMAN" and s.name == "user-feedback"
            ),
            None,
        )
        if step.scores
        else None
    )
    return {
        "createdAt": step.created_at,
        "id": step.id or "",
        "threadId": step.thread_id or "",
        "parentId": step.parent_id,
        "feedback": cls.score_to_feedbackdict(user_feedback),
        "start": step.start_time,
        "end": step.end_time,
        "type": step.type or "undefined",
        "name": step.name or "",
        "generation": step.generation.to_dict() if step.generation else None,
        "input": input,
        "output": output,
        "showInput": metadata.get("showInput", False),
        "indent": metadata.get("indent"),
        "language": metadata.get("language"),
        "isError": bool(step.error),
        "waitForAnswer": metadata.get("waitForAnswer", False),
        # Add all additional metadata fields here
        "additionalMetadata": metadata  # This line ensures all metadata is retained
    }

This modification adds an "additionalMetadata" field to the StepDict, which retains all metadata from the LiteralStep object. This will ensure that when a thread is resumed, all necessary metadata is available ^[1].

To continue talking to Dosu, mention @dosu.