UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://UKGovernmentBEIS.github.io/inspect_ai/
MIT License
385 stars 41 forks source link

Add optional tool param "properties" to adhere to OpenAI spec #55

Closed bjoernpl closed 1 week ago

bjoernpl commented 2 weeks ago

Prior, I was unable to have a tool param with type "array" because OpenAI spec requires items={"type": "str"} property to be set. This solution adds the option to specify any optional parameters to pass to the actual API. May also be useful for flexibility in working with different providers. Very open for feedback on how this should be tackled.

This PR contains:

What is the current behavior? (You can also link to an open issue here)

Cannot have a list as the type of my toolparam since it doesn't adhere to OpenAI API spec.

What is the new behavior?

By setting optional properties you can override this behaviour

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No

Other information:

This solution may not be optimal. Probably it could be nicer to have a specific property for this

aisi-inspect commented 1 week ago

Thanks! Would like get this in but need to think about how it might affect other providers that also consume the same JSON schema. Will do some additional research/experimentation then revert back.

aisi-inspect commented 1 week ago

Took a closer look at this and what I think I'd like to do is generally extend our tool use / docstring handling to support arrays (then you wouldn't need a special field here, we would just do the right thing when seeing a Python list). Will likely work on this soon and post an update here once we have one.

bjoernpl commented 1 week ago

Sounds good, thanks for looking at this!