no attribute '__mutable_keys__' on python 3.11 when running the CLI

[adrianlyjak]

I'm attempting to run the demo example in the tutorial. Once running inspect eval test.py --model openai/gpt-3.5, I get a stack trace.

# test.py
from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import (
  chain_of_thought, generate, self_critique
)

@task
def theory_of_mind():
    return Task(
        dataset=example_dataset("theory_of_mind"),
        plan=[
          chain_of_thought(), 
          generate(), 
          self_critique()
        ],
        scorer=model_graded_fact(),
    )                                                                                      

Traceback:

Traceback (most recent call last):
  File "/Users/adrianlyjak/dev/prompty/.venv/bin/inspect", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/inspect_ai/_cli/main.py", line 44, in main
    inspect(auto_envvar_prefix="INSPECT")
  File "/inspect/.venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/inspect_ai/_cli/common.py", line 46, in wrapper
    return cast(click.Context, func(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^
  File "/inspect/.venv/lib/python3.11/site-packages/inspect_ai/_cli/eval.py", line 209, in eval_command
    config_keys = list(GenerateConfigArgs.__mutable_keys__)  # type: ignore

I believe that __mutable_keys__ is a new feature in python 3.13. In the pyproject.toml, it appears that you support python >=3.10

[adrianlyjak]

Hmm, apologies, maybe this is something else. I tried recreating this in a standalone project without other dependencies, and its now working using python 3.11.