Bug: Running evaluate() without an API Key environment variable fails

[PGAnswerRocket]

Issue you'd like to raise.

I am attempting to run the evaluate() function. I am working in an environment that will not allow me to set environment variables such as with os.environ["LANGCHAIN_API_KEY"] = "lsv...". Instead I am constructing my Client() object with the api key passed in as a parameter via client = Client(api_key="lsv....") and then passing the client into the evaluate() function.

Here is an example of my evaluate() call:

experiment_results = evaluate(
    target,
    data = dataset_name,
    evaluators = [skill_choice],
    experiment_prefix = "notebook-skillchoice-test",
    client=client
)

Here is the resulting error and stack trace:

Error running evaluator <DynamicRunEvaluator skill_choice> on run e9d39149-2b41-4e6f-b605-af3ff15b40ae: LangSmithUserError('API key must be provided when using hosted LangSmith API')
Traceback (most recent call last):
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/_runner.py", line 874, in _run_evaluators
    evaluator_response = evaluator.evaluate_run(
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.py", line 250, in evaluate_run
    result = self.func(
             ^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 544, in wrapper
    run_container = _setup_run(
                    ^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 1065, in _setup_run
    new_run = run_trees.RunTree(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 339, in __init__
    values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 1074, in validate_model
    v_, errors_ = field.validate(value, values, loc=field.alias, cls=cls_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 864, in validate
    v, errors = self._apply_validators(v, values, loc, cls, self.pre_validators)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 1154, in _apply_validators
    v = validator(cls, v, values, self, self.model_config)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/class_validators.py", line 304, in <lambda>
    return lambda cls, v, values, field, config: validator(cls, v)
                                                 ^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_trees.py", line 63, in validate_client
    return Client()
           ^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 534, in __init__
    _validate_api_key_if_hosted(self.api_url, self.api_key)
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 323, in _validate_api_key_if_hosted
    raise ls_utils.LangSmithUserError(
langsmith.utils.LangSmithUserError: API key must be provided when using hosted LangSmith API
Error running evaluator <DynamicRunEvaluator skill_choice> on run f4755d4c-9c34-494f-8db5-b3e98ada3b9c: LangSmithUserError('API key must be provided when using hosted LangSmith API')
Traceback (most recent call last):
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/_runner.py", line 874, in _run_evaluators
    evaluator_response = evaluator.evaluate_run(
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.py", line 250, in evaluate_run
    result = self.func(
             ^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 544, in wrapper
    run_container = _setup_run(
                    ^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_helpers.py", line 1065, in _setup_run
    new_run = run_trees.RunTree(
              ^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 339, in __init__
    values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/main.py", line 1074, in validate_model
    v_, errors_ = field.validate(value, values, loc=field.alias, cls=cls_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 864, in validate
    v, errors = self._apply_validators(v, values, loc, cls, self.pre_validators)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/fields.py", line 1154, in _apply_validators
    v = validator(cls, v, values, self, self.model_config)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/pydantic/v1/class_validators.py", line 304, in <lambda>
    return lambda cls, v, values, field, config: validator(cls, v)
                                                 ^^^^^^^^^^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/run_trees.py", line 63, in validate_client
    return Client()
           ^^^^^^^^
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 534, in __init__
    _validate_api_key_if_hosted(self.api_url, self.api_key)
  File "/Users/prestongoren/Desktop/dadproj/venv/lib/python3.12/site-packages/langsmith/client.py", line 323, in _validate_api_key_if_hosted
    raise ls_utils.LangSmithUserError(
langsmith.utils.LangSmithUserError: API key must be provided when using hosted LangSmith API

Setting the environment variable via os.environ["LANGCHAIN_API_KEY"] = gets rid of the error, but I need to be able to run it without the environment variable -- which should be possible by passing the client object into evaluate()

Suggestion:

investigate usage of pydantic validation inside of the evaluate() function

[hinthornw]

Thanks for flagging this - I must have failed to test this case. Will try to get a fix out when I find time