The agent protocol compliance check fails for the minimal reference implementation

[ATheorell]

The compliance test suggested here: https://agentprotocol.ai/compliance fails with the below error for the minimal reference implementation https://github.com/AI-Engineers-Foundation/agent-protocol-sdk-python/blob/main/examples/minimal.py and identically for the api implementation at in PR #698 in https://github.com/AntonOsika/gpt-engineer/pull/698 .

It looks unsurprising that the list check fails, given that the json has both the task_ids and pagination properties.

/home/axel/Software/gpt-engineer/venv/bin/python /home/axel/Software/gpt-engineer/venv/bin/agent-protocol test --url http://127.0.0.1:8000 ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.3.0 -- /home/axel/Software/gpt-engineer/venv/bin/python cachedir: .pytest_cache rootdir: /home/axel/Software plugins: anyio-3.7.1, asyncio-0.21.1 asyncio: mode=strict collecting ... collected 7 items

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_create_agent_task[http:/127.0.0.1:8000] PASSED [ 14%] ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_tasks_ids[http:/127.0.0.1:8000] FAILED [ 28%] ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_get_agent_task[http:/127.0.0.1:8000] PASSED [ 42%] ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_task_steps[http:/127.0.0.1:8000] FAILED [ 57%] ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_execute_agent_task_step[http:/127.0.0.1:8000] PASSED [ 71%] ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_artifacts[http:/127.0.0.1:8000] PASSED [ 85%] ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_get_agent_task_step[http:/127.0.0.1:8000] FAILED [100%]

=================================== FAILURES =================================== ___ TestCompliance.test_list_agent_tasks_ids[http://127.0.0.1:8000] ____

self = <agent_protocol.utils.compliance.main.TestCompliance object at 0x7f6b24c05e10> url = 'http://127.0.0.1:8000'

def test_list_agent_tasks_ids(self, url):
    response = requests.get(f"{url}/ap/v1/agent/tasks")
    assert response.status_code == 200

  assert isinstance(response.json(), list)

E AssertionError

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py:20: AssertionError _ TestCompliance.test_list_agent_tasksteps[http://127.0.0.1:8000]

self = <agent_protocol.utils.compliance.main.TestCompliance object at 0x7f6b24c06560> url = 'http://127.0.0.1:8000'

def test_list_agent_task_steps(self, url):
    # Create task
    response = requests.post(f"{url}/ap/v1/agent/tasks", json=self.task_data)
    task_id = response.json()["task_id"]
    response = requests.get(f"{url}/ap/v1/agent/tasks/{task_id}/steps")
    assert response.status_code == 200

  assert isinstance(response.json(), list)

E AssertionError

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py:36: AssertionError ____ TestCompliance.test_get_agent_task_step[http://127.0.0.1:8000] ____

self = <agent_protocol.utils.compliance.main.TestCompliance object at 0x7f6b24c06b00> url = 'http://127.0.0.1:8000'

def test_get_agent_task_step(self, url):
    # Create task
    response = requests.post(f"{url}/ap/v1/agent/tasks", json=self.task_data)
    task_id = response.json()["task_id"]
    # Get steps
    response = requests.get(f"{url}/ap/v1/agent/tasks/{task_id}/steps")

  step_id = response.json()[0]

E KeyError: 0

../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py:61: KeyError =========================== short test summary info ============================ FAILED ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_tasks_ids[http:/127.0.0.1:8000] FAILED ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_list_agent_task_steps[http:/127.0.0.1:8000] FAILED ../gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py::TestCompliance::test_get_agent_task_step[http:/127.0.0.1:8000] ========================= 3 failed, 4 passed in 0.14s ========================== Traceback (most recent call last): File "/home/axel/Software/gpt-engineer/venv/bin/agent-protocol", line 8, in sys.exit(cli()) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, **kwargs) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/cli.py", line 25, in _check_compliance check_compliance(url, args) File "/home/axel/Software/gpt-engineer/venv/lib/python3.10/site-packages/agent_protocol/utils/compliance/main.py", line 104, in check_compliance assert exit_code == 0, "Your Agent API isn't compliant with the agent protocol" AssertionError: Your Agent API isn't compliant with the agent protocol

[hackgoofer]

We actually fixed it! Please checkout our new SDK in this repo: https://github.com/AI-Engineers-Foundation/agent-protocol-sdk-python/tree/main

[hackgoofer]

Closing, feel free to reopen if the issue persists. :)

[ATheorell]

For me, following the steps:

$ git clone git@github.com:AI-Engineers-Foundation/agent-protocol-sdk-python.git

$ cd agent-protocol-sdk-python

$ poetry install

$ poetry run python examples/minimal.py

And in a separate terminal

$ pip install --upgrade agent-protocol ... Successfully installed agent-protocol-1.0.1

$ agent-protocol test --url http://127.0.0.1:8000

still yields the error in my previous post. Please reopen and let me know what I'm doing wrong @hackgoofer

[ATheorell]

Please reopen or comment :) @hackgoofer

[hackgoofer]

Hi all,

So, here is how we are testing your compliance on agent-protocol: URL=http://127.0.0.1:8000 bash -c "$(curl -fsSL https://raw.githubusercontent.com/AI-Engineers-Foundation/agent-protocol/main/testing_suite/test.sh)"

[ATheorell]

Hi, Thanks for pointing me to the new test and I can confirm that the minimal example is working