awslabs / agent-evaluation

A generative AI-powered framework for testing virtual agents.
https://awslabs.github.io/agent-evaluation/
Apache License 2.0
117 stars 20 forks source link

Bedrock Agent required arguments documented as optional #89

Closed rvernica closed 2 days ago

rvernica commented 3 weeks ago

The following two Bedrock Agent parameters are required in the configuration file but there are documented as optional.

Required arguments in the agenteval.yml file when target type: bedrock-agent:

  bedrock_session_attributes: {}
  bedrock_prompt_session_attributes: {}

Both are listed as optional in the documentation file.

If not specified in the agenteval.yml the following error is reported:

...
  File "/agent-evaluation/src/agenteval/plan/plan.py", line 194, in _run_test
    target = self._target_factory.create()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/agent-evaluation/src/agenteval/targets/target_factory.py", line 42, in create
    return target_cls(**{k: v for k, v in self.config.items() if k != "type"})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: BedrockAgentTarget.__init__() missing 2 required positional arguments: 'bedrock_session_attributes' and 'bedrock_prompt_session_attributes'

The BedrockAgentTarget constructor has the two arguments with no defaults.

This issue is reported against the main branch at the linked revision.

namdoswo commented 4 days ago

I'm having the same error. Currently nobody is working on this?