search

microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
8.7k stars 780 forks source link

[Feature Request] Allow the target function in evaluate to be a local function #3208

Open pamelafox opened 1 month ago

pamelafox commented 1 month ago

I currently get this error when passing a local function to evaluate:

promptflow.exceptions.UserErrorException: Failed to import <function run_evaluation..wrap_target at 0x318bc7ba0> from module <module 'scripts.evaluate' from '/Users/pamelafox/ai-rag-chat-evaluator/scripts/evaluate.py'>, please make sure it's a global function.

That is due to the usage of multiprocessing and its need to serialize the function.

Perhaps you could use asyncio.taskgroup or a different mechanism such that local functions are allowed. That would permit me to write much cleaner and more portable, testable code.

D-W- commented 1 month ago

Hi @pamelafox , thanks for the suggestion. We've already tracking this internally. The reason we did not support this in the first place was to maintain parity between local and cloud (it's hard to run a local function in cloud). However, we will prioritize support for local scenarios first.

github-actions[bot] commented 3 weeks ago

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

pamelafox commented 3 weeks ago

I don't believe any more input is needed from me. (The stalebot in this repo seems to be tuned to respond really quickly to perceived stalneness, btw).