python[patch]: evaluators can return primitives

[baskaryan]

from langsmith import evaluate

def foo(run, example):
    return 0

def bar(run, example):
    return "long"

# removed, needs to be list of dict
# def baz(run, example):
#     return [0, 0.2, "how are ya"]

def app(inputs):
    return inputs

evaluate(app, data="Sample Dataset 3", evaluators=[foo, bar, baz])

[baskaryan]

how ^ example gets logged

[hinthornw]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

[baskaryan]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

are we able to support list values as feedback?

[hinthornw]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

are we able to support list values as feedback?

Only dict or string it seems.

Maybe doing the _ix suffix is preferable. I'm not sure honestly. Probalby less surprising than just logging duplicates to the same key.

It just messes with the experiment averages (we'd be averaging over index 1 and over index 2 )

[baskaryan]

The list return type seems unexpected to me - I wouldn't expect it to make separate keys for each value

are we able to support list values as feedback?

Only dict or string it seems.

Maybe doing the _ix suffix is preferable. I'm not sure honestly. Probalby less surprising than just logging duplicates to the same key.

It just messes with the experiment averages (we'd be averaging over index 1 and over index 2 )

feel less strongly about list behavior either way, think main use case is supporting int/float/bool/str

[hinthornw]

Maybe let's land with the numeric and string value support but hold off on list behavior?

Maybe add support for list[evaluationresultlike] to make the "results" key not necessary