search

UKGovernmentBEIS / inspect_ai

Inspect: A framework for large language model evaluations
https://UKGovernmentBEIS.github.io/inspect_ai/
MIT License
385 stars 41 forks source link

include 'MemoryDataset' in the docs #64

Closed Lovkush-A closed 3 days ago

Lovkush-A commented 5 days ago

TLDR: I think it would be useful to add 'MemoryDataset' in the datasets docs page. https://ukgovernmentbeis.github.io/inspect_ai/datasets.html

I am trying to parse this dataset: https://gist.github.com/nix-apollo/fa562b0269d3e1324ef4fae0902bd1f3

I dont think I can parse using the tools provided in https://ukgovernmentbeis.github.io/inspect_ai/datasets.html, because i need a nested for loop to access the individual records from the json object. But I can do parsing outside of inspect, and then use MemoryDataset to create the Dataset object.

jjallaire commented 4 days ago

Good catch, will do this in our next sync.

jjallaire commented 3 days ago

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/commit/2d7561f02213781d603ed4ff1826f8dfe0963da4