UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://inspect.ai-safety-institute.org.uk/
MIT License
558 stars 93 forks source link

tool, scorer, etc should preseve function annotations #506

Open JasonGross opened 4 days ago

JasonGross commented 4 days ago

VSCode tooltips for wrapped functions are missing argument information in the type, e.g.,: image By contrast, annotating functions with @functools.cache or @torch.no_grad() does not remove type arguments. It seems the issue is the cast in https://github.com/UKGovernmentBEIS/inspect_ai/blob/5eb6190631bd9687ed4812816cc303d78c1f6976/src/inspect_ai/tool/_tool.py#L154, though I'm not 100% sure this diagnosis is right

jjallaire-aisi commented 4 days ago

Thanks for this! I think there is a way to do this with functools.wraps I just need to make sure that we aren't somehow relying downstream on the wrapper (in one case with solvers I know that we are, but I think I have a workaround for that).