Approvals for Entropy-labs not found in the registry

[mlcocdav]

Hi @jjallaire, we have created approvers inside our entropy-labs python package. We followed instructions for the approvals registration here: https://inspect.ai-safety-institute.org.uk/extensions.html#sec-extensions-approvers. In our pyproject.toml file, we specified

[project.entry-points.inspect_ai]
entropy_labs = "el.supervision.inspect_ai._registry"

and this is our el.supervision.inspect_ai._registry:

from .supervisors import bash_approver, python_approver, human_approver, llm_approver

But when running inspect eval ( inspect eval run.py --approval approval_human.yaml --model openai/gpt-4o --trace ) with this example approval_human.yaml

approvers:
  - name: el/human_approver
    tools: "*"
    approval_api_endpoint: "http://localhost:8080"
    agent_id: "my_sample_1_agent"
    n: 1

We receive ValueError: el/human_approver was not found in the registry. Our approvals are registered only if we manually import our registry to the run.py script where the inspect tasks are defined.

from el.supervision.inspect_ai import _registry

But this is just a hack for now and is not good for evals already integrated to Inspect examples. Do you know if we are missing some configuration or we configured something wrong? Is it possible that there is an issue with the Approver Registration code itself?

To replicate

Install entropy-labs:

pip install entropy-labs --upgrade

And use the .yaml above with inspect eval, e.g.:

inspect eval inspect_evals/gdm_in_house_ctf --model openai/gpt-4o --limit 1 --approval approval_human.yaml

Thank you for your help!

[jjallaire]

This came down to aligning on the "names" in play (package name, setuptools entry point name, dist name). This change to your pyproject.toml will make things work against Inspect on PyPI:

https://github.com/EntropyLabsAI/sentinel/compare/main...jjallaire:sentinel:patch-1

Note we had to change the project name to ep (to align with the package name). You might have reasons for not wanting to do this?

With this change to Inspect (which won't hit PyPI for a week or two) you can keep the project name "entropy_labs":

https://github.com/UKGovernmentBEIS/inspect_ai/pull/750/files

[mlcocdav]

Thank you @jjallaire for looking into this. As you mentioned, the problem was with the name discrepancy el vs. entropy_labs. We couldn't name our package el as that name was already taken in PyPI, but we also wanted to give our users the ease of importing our package as from el import ... That being said, to simplify this and move forward, we renamed everything (package, setuptools entry point name, dist name) to entropy-labs and the approval registry now works.