I am attempting to run the evaluate() function. I am working in an environment that will not allow me to set environment variables such as with os.environ["LANGCHAIN_API_KEY"] = "lsv...". Instead I am constructing my Client() object with the api key passed in as a parameter via client = Client(api_key="lsv....") and then passing the client into the evaluate() function.
Error running evaluator <DynamicRunEvaluator skill_choice> on run e9d39149-2b41-4e6f-b605-af3ff15b40ae: LangSmithUserError('API key must be provided when using hosted LangSmith API')
Traceback (most recent call last):
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/_runner.py", line 874, in _run_evaluators
evaluator_response = evaluator.evaluate_run(
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.py", line 250, in evaluate_run
result = self.func(
^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 544, in wrapper
run_container = _setup_run(
^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 1065, in _setup_run
new_run = run_trees.RunTree(
^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 339, in __init__
values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 1074, in validate_model
v_, errors_ = field.validate(value, values, loc=field.alias, cls=cls_)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 864, in validate
v, errors = self._apply_validators(v, values, loc, cls, self.pre_validators)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 1154, in _apply_validators
v = validator(cls, v, values, self, self.model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/class_validators.py", line 304, in <lambda>
return lambda cls, v, values, field, config: validator(cls, v)
^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_trees.py", line 63, in validate_client
return Client()
^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 534, in __init__
_validate_api_key_if_hosted(self.api_url, self.api_key)
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 323, in _validate_api_key_if_hosted
raise ls_utils.LangSmithUserError(
langsmith.utils.LangSmithUserError: API key must be provided when using hosted LangSmith API
Error running evaluator <DynamicRunEvaluator skill_choice> on run f4755d4c-9c34-494f-8db5-b3e98ada3b9c: LangSmithUserError('API key must be provided when using hosted LangSmith API')
Traceback (most recent call last):
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/_runner.py", line 874, in _run_evaluators
evaluator_response = evaluator.evaluate_run(
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.py", line 250, in evaluate_run
result = self.func(
^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 544, in wrapper
run_container = _setup_run(
^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 1065, in _setup_run
new_run = run_trees.RunTree(
^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 339, in __init__
values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 1074, in validate_model
v_, errors_ = field.validate(value, values, loc=field.alias, cls=cls_)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 864, in validate
v, errors = self._apply_validators(v, values, loc, cls, self.pre_validators)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 1154, in _apply_validators
v = validator(cls, v, values, self, self.model_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/class_validators.py", line 304, in <lambda>
return lambda cls, v, values, field, config: validator(cls, v)
^^^^^^^^^^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_trees.py", line 63, in validate_client
return Client()
^^^^^^^^
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 534, in __init__
_validate_api_key_if_hosted(self.api_url, self.api_key)
File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 323, in _validate_api_key_if_hosted
raise ls_utils.LangSmithUserError(
langsmith.utils.LangSmithUserError: API key must be provided when using hosted LangSmith API
Setting the environment variable via os.environ["LANGCHAIN_API_KEY"] = gets rid of the error, but I need to be able to run it without the environment variable -- which should be possible by passing the client object into evaluate()
Suggestion:
investigate usage of pydantic validation inside of the evaluate() function
Issue you'd like to raise.
I am attempting to run the
evaluate()
function. I am working in an environment that will not allow me to set environment variables such as withos.environ["LANGCHAIN_API_KEY"] = "lsv..."
. Instead I am constructing myClient()
object with the api key passed in as a parameter viaclient = Client(api_key="lsv....")
and then passing the client into theevaluate()
function.Here is an example of my
evaluate()
call:Here is the resulting error and stack trace:
Setting the environment variable via
os.environ["LANGCHAIN_API_KEY"] =
gets rid of the error, but I need to be able to run it without the environment variable -- which should be possible by passing the client object intoevaluate()
Suggestion:
investigate usage of pydantic validation inside of the
evaluate()
function