🐛 Bug Report: LCEL Runnables not supported in traces

sidpan1 commented 8 months ago

Which component is this bug for?

All Packages

📜 Description

When a chain is built using LCEL, traces are not emitted.

👟 Reproduction steps

Create an LCEL chain
Run the application

👍 Expected behavior

Traces should be emitted for langchain blocks.

👎 Actual Behavior with Screenshots

Only traces emitted :

{ "name": "PATCH", "context": { "trace_id": "0x5d226499992625813276d417daf03aad", "span_id": "0x52cf0fa0514c33a5", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:14.970928Z", "end_time": "2024-01-29T11:06:15.404928Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "PATCH", "http.url": "https://api.smith.langchain.com/runs/97d3bdd0-8a1e-4437-9401-84e2004f16d8", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "POST", "context": { "trace_id": "0xe6cf84bd7a5a72d6220d76528fc6fa32", "span_id": "0x0aa04c6a4dbf1bfd", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:14.970581Z", "end_time": "2024-01-29T11:06:15.430682Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "POST", "http.url": "https://api.smith.langchain.com/runs", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "POST", "context": { "trace_id": "0xfa2c996295cfda4103f37090627791c7", "span_id": "0x5c3d0c3dd6bd1d51", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:14.971143Z", "end_time": "2024-01-29T11:06:15.441099Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "POST", "http.url": "https://api.smith.langchain.com/runs", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "POST", "context": { "trace_id": "0xeff8835537de179427d32f928d2b244c", "span_id": "0xa010e98b7ec99bc6", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:14.971049Z", "end_time": "2024-01-29T11:06:15.471814Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "POST", "http.url": "https://api.smith.langchain.com/runs", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } INFO: HTTP Request: POST https://navira-poc-02.openai.azure.com//openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-09-01-preview "HTTP/1.1 200 OK" { "name": "openai.chat", "context": { "trace_id": "0xa3b49a57b3990fdd94f44b647ab7f4fb", "span_id": "0xe925e3010d0e482a", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:14.955512Z", "end_time": "2024-01-29T11:06:15.976070Z", "status": { "status_code": "UNSET" }, "attributes": { "llm.request.type": "chat", "openai.api_version": "2023-09-01-preview", "llm.vendor": "OpenAI", "llm.request.model": "gpt-3.5-turbo", "llm.temperature": 0.0, "llm.headers": "None", "llm.prompts.0.role": "user", "llm.prompts.0.content": "You are an AWS cloud consultant tasked with recognizing the intent of users' queries in the context of cloud cost governance. Pay close attention to the exact definitions of allowed categories & services. DO NOT deviate from them STRICTLY.\n\nContext: \nThe intent consists of 2 parts : category and service.\n\nThe possible categories of user's intent. \n- RESOURCE_METADATA_DISCOVERY: Involves querying metadata or attributes of user provisioned AWS cloud resources. This can include details like configurations, provisioned capacity, tags and status.\n- RESOURCE_USAGE_DISCOVERY: Focuses on analyzing resource utilization metrics of the user's provisioned resources such as performance metrics and usage patterns across different metrics.\n- RESOURCE_COST_DISCOVERY: It involves evaluating cost-related data of user's provisioned resources to gain insights into expenditure patterns and identify cost drivers.\n- PRICING_DISCOVERY: Dedicated to exploring the general public pricing details of AWS services (not related to the user's data). This includes obtaining information about service rates and understanding different pricing models (e.g., on-demand, reserved instances).\n- RECOMMENDATION: Includes providing suggestions and actionable advice for optimizing AWS cloud resource usage, costs & performance.\n\nThe services related to the user's intent.\n- EC2: Amazon Elastic Compute Cloud\n- RDS: Amazon Relational Database Service\n- S3: Amazon Simple Storage Service\n- DYNAMODB: Amazon DynamoDB\n- EBS: Amazon Elastic Block Storage\n- OPENSEARCH_SERVICE: Amazon OpenSearch Service\n- CLOUDWATCH: Amazon CloudWatch\n- ELASTICACHE: Amazon ElastiCache\n- VPC: Amazon Virtual Private Cloud\n- ELB: Amazon Elastic Load Balancing\n- CLOUDFRONT: Amazon CloudFront\nOutput Format:The output should be formatted as a JSON instance that conforms to the JSON schema below.\n\nAs an example, for the schema {\"properties\": {\"foo\": {\"title\": \"Foo\", \"description\": \"a list of strings\", \"type\": \"array\", \"items\": {\"type\": \"string\"}}}, \"required\": [\"foo\"]}\nthe object {\"foo\": [\"bar\", \"baz\"]} is a well-formatted instance of the schema. The object {\"properties\": {\"foo\": [\"bar\", \"baz\"]}} is not well-formatted.\n\nHere is the output schema:\n\n{\"description\": \"Serializable base class.\", \"properties\": {\"category\": {\"description\": \"The category of the user's intent.\", \"allOf\": [{\"$ref\": \"#/definitions/IntentCategory\"}]}, \"service\": {\"description\": \"The service related to the user's intent.\", \"allOf\": [{\"$ref\": \"#/definitions/AmazonService\"}]}}, \"required\": [\"category\", \"service\"], \"definitions\": {\"IntentCategory\": {\"title\": \"IntentCategory\", \"description\": \"An enumeration.\", \"enum\": [\"RESOURCE_METADATA_DISCOVERY\", \"RESOURCE_USAGE_DISCOVERY\", \"RESOURCE_COST_DISCOVERY\", \"COST_DISCOVERY\", \"RECOMMENDATION\"], \"type\": \"string\"}, \"AmazonService\": {\"title\": \"AmazonService\", \"description\": \"An enumeration.\", \"enum\": [\"EC2\", \"RDS\", \"S3\", \"DYNAMODB\", \"EBS\", \"OPENSEARCH_SERVICE\", \"CLOUDWATCH\", \"ELASTICACHE\", \"VPC\", \"ELB\", \"CLOUDFRONT\"], \"type\": \"string\"}}}\n\nUser's query: how many ec2 instances\n", "llm.response.model": "gpt-35-turbo", "llm.usage.total_tokens": 735, "llm.usage.completion_tokens": 20, "llm.usage.prompt_tokens": 715, "llm.completions.0.finish_reason": "stop", "llm.completions.0.role": "assistant", "llm.completions.0.content": "{\n \"category\": \"RESOURCE_METADATA_DISCOVERY\",\n \"service\": \"EC2\"\n}" }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "PATCH", "context": { "trace_id": "0xd95ec01e44229f05b4844ca8325c94c2", "span_id": "0x9dc9086284294043", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:16.068286Z", "end_time": "2024-01-29T11:06:16.367807Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "PATCH", "http.url": "https://api.smith.langchain.com/runs/39ee576a-6e25-4260-9ba7-d6e080e65ec9", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "POST", "context": { "trace_id": "0x6e2c49a93463ceb0965582c405be317e", "span_id": "0x9f801f0e5058c0ba", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:16.068165Z", "end_time": "2024-01-29T11:06:16.398113Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "POST", "http.url": "https://api.smith.langchain.com/runs", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "PATCH", "context": { "trace_id": "0x66442059532c14185b5da4b6c0e3e2a5", "span_id": "0xf8da2f9a81d0d8e1", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:16.067977Z", "end_time": "2024-01-29T11:06:16.398613Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "PATCH", "http.url": "https://api.smith.langchain.com/runs/5260c5a9-68c0-4910-b206-5986e4244dac", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } } { "name": "PATCH", "context": { "trace_id": "0xa4309c9db74617d83e689ac2fcaeae3d", "span_id": "0x52d896629a51553c", "trace_state": "[]" }, "kind": "SpanKind.CLIENT", "parent_id": null, "start_time": "2024-01-29T11:06:16.067677Z", "end_time": "2024-01-29T11:06:16.402780Z", "status": { "status_code": "UNSET" }, "attributes": { "http.method": "PATCH", "http.url": "https://api.smith.langchain.com/runs/3047f648-72af-4c4e-b223-dbb38c97002e", "http.status_code": 200 }, "events": [], "links": [], "resource": { "attributes": { "service.name": "src/streamlit_main.py" }, "schema_url": "" } }

🤖 Python Version

3.11.7

📃 Provide any additional context for the Bug.

No response

👀 Have you spent some time to check if this bug has been raised before?

[X] I checked and didn't find similar issue

Are you willing to submit PR?

None

sidpan1 commented 8 months ago

I think the Runnables need to be enabled here : https://github.com/traceloop/openllmetry/blob/e81dada24d4a650d6466c6eb6a0a05444aa4740e/packages/opentelemetry-instrumentation-langchain/opentelemetry/instrumentation/langchain/__init__.py#L25C1-L78C2

galkleinman commented 8 months ago

Hey @sidpan1,

Thanks for reporting this one!

We'll have a look at this one, and hopefully will issue a fix if needed within the next 1-2 days.

nirga commented 8 months ago

@sidpan1 can you share the example of the program you used, and the langchain version? We were supporting some LCEL cases (see this example) so I want to see if I can reproduce it locally.

nirga commented 8 months ago

@sidpan1 checking if you saw my previous comment - can you share a snippet that's not working for you?

nirga commented 8 months ago

@sidpan1 I'm going to close this as I can't reproduce it. Please provide a snippet for this so we can address this if needed.

nirga commented 8 months ago

This was reported again and I was managed to find an example that reproduces it:

from langchain.prompts import ChatPromptTemplate
from langchain.output_parsers.openai_functions import JsonOutputFunctionsParser
from langchain_community.utils.openai_functions import (
    convert_pydantic_to_openai_function,
)
from langchain_community.chat_models import ChatOpenAI
from langchain_core.pydantic_v1 import BaseModel, Field

from traceloop.sdk import Traceloop

Traceloop.init(app_name="lcel_example")

class Joke(BaseModel):
    """Joke to tell user."""

    setup: str = Field(description="question to set up a joke")
    punchline: str = Field(description="answer to resolve the joke")

openai_functions = [convert_pydantic_to_openai_function(Joke)]

prompt = ChatPromptTemplate.from_messages(
    [("system", "You are helpful assistant"), ("user", "{input}")]
)
model = ChatOpenAI(model="gpt-3.5-turbo")
output_parser = JsonOutputFunctionsParser()

chain = prompt | model.bind(functions=openai_functions) | output_parser
result = chain.invoke({"input": "tell me a short joke"})

print(result)

Will issue a fix today

traceloop / openllmetry