Open yanggaome opened 2 months ago
one observation is that, when I created the target user call (with a class same with the evaluator, provided init and call methods)
in the constructor, if I passed the loaded data (~100 lines jsonl data) as argument explicitly: TargetCallClass(config, input_data)
, then it got this error
~if i passed this loaded data in a config.input_data: TargetCallClass(config)
, it seems better, better means can succeeded on one machine, but still failed on another. not sure if this is related, but just giving this observation~
do you have any idea about this?
also, if i set _use_pf_client to false with 0.3.1 evals, it worked fine (just as 0.3.0 as they both use CodeClient not ProxyClient)
Hi @ninghu , I think i was able to identify the root cause.
The issue was when the evaluate api gets called, the data path should be absolute path, not relative path w.r.t. working directory.
_ = evaluate(
evaluation_name=evaluation_name,
data=data_path,
target=custom_user_call,
evaluators=evaluators_set,
evaluator_config=evaluators_config,
)
When using CodeClient, the dataframe read from data_path is used. When using ProxyClient, data (data path) is used for pfclient run. However, this still puzzles me, because for target call, it is also a pf client run, with data to be relative path, it still works well. but it makes a difference when using ProxyClient.
Another observation is, if we use one evaluator (either built-in or customized), even with ProxyClient + relative path, it still worked.
But when using two evaluators, ProxyClient + relative path didn't work. We have to use absolute path.
These are great findings! The behavior you described is interesting and seems to occur only when using more than one evaluator. We'll try to reproduce this issue and get back to you.
Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!
Describe the bug
After switching promptflow-eval from 0.3.0 to 0.3.1, for same code, it keeps giving me the error:
I noticed some changes related to performance/parallelization between 0.3.0 and 0.3.1. e.g. https://github.com/microsoft/promptflow/commit/32115d397b888be86eaea9b54039f5dd20f55dc0
to
Is this related to the failure? and which client is parallelization and recommended to use?
How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug: 1.
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Running Information(please complete the following information):
pf -v
: [e.g. 0.0.102309906]python --version
: [e.g. python==3.10.12]{ "promptflow": "1.13.0", "promptflow-azure": "1.13.0", "promptflow-core": "1.13.0", "promptflow-devkit": "1.13.0", "promptflow-evals": "0.3.1", or 0.3.0 as comparision "promptflow-tracing": "1.13.0" }
Additional context Add any other context about the problem here.