search

valory-xyz / open-autonomy

A framework for the creation of autonomous agent services.
https://docs.autonolas.network
Apache License 2.0
82 stars 26 forks source link

Service analyzer improvements #1898

Closed Adamantios closed 1 year ago

Adamantios commented 1 year ago

Subject of the issue

The service analysis command is currently difficult to use. At the moment, I have detected the following issues:

Confusing error message

Running the service analyzer:

autonomy analyse service --public-id valory/oracle_hardhat

with consensus_threshold specified on the skill configuration, gives:

[2023-03-24 09:49:06,371][INFO] Validating service overrides
[2023-03-24 09:49:06,371][INFO] Validating (skill, valory/oracle_abci:0.1.0) from the service overrides
Error: ABCI skill validation failed; 'consensus_threshold' is a required property

That is because the service does not specify the consensus_threshold, but the message incorrectly reports that it is missing from the skill.

Configurations required in setup are reported as missing

If we go ahead and add the consensus_threshold in the service configuration, but do not place it under the setup, then it is reported as missing, instead of mentioning that the consensus_threshold was not found in the setup configuration.

Incorrectly reporting as missing

If we fix all the issues reported by the tool, then it incorrectly performs the cross-verification:

[2023-03-24 11:03:51,366][INFO] Cross verifying overrides between skill and agent
[2023-03-24 11:03:51,366][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (skill, valory/oracle_abci:0.1.0)
        Path: models
        Missing parameters: {'price_api', 'randomness_api', 'server_api'}

Too verbose

The tool is too verbose. Even though everything is correct in the configurations on price-oracle@e2519777a136213a9824bf8c6e0b96198b216945, the tool reports all the following:

[2023-03-24 11:03:51,361][INFO] Validating service overrides
[2023-03-24 11:03:51,362][INFO] Validating (skill, valory/oracle_abci:0.1.0) from the service overrides
[2023-03-24 11:03:51,362][INFO] Validating (connection, valory/ledger:0.19.0) from the service overrides
[2023-03-24 11:03:51,362][INFO] Validating agent overrides
[2023-03-24 11:03:51,362][INFO] Validating skill overrides
[2023-03-24 11:03:51,362][INFO] Cross verifying overrides between agent and service
[2023-03-24 11:03:51,363][INFO] Verifying the skill override with index 0
[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args
        Missing parameters: {'response_index'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.headers
        Missing parameters: {'$', 'c', '}', 'd', 'i', '{', 't', ':'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.parameters
        Missing parameters: {'pair', 'symbol'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args
        Missing parameters: {'tendermint_com_url', 'termination_sleep', 'reset_tendermint_after', 'multisend_address', 'validate_timeout', 'init_fallback_gas', 'tendermint_url', 'round_timeout_seconds', 'use_termination', 'max_healthcheck', 'tendermint_p2p_url'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args.setup
        Missing parameters: {'oracle_contract_address'}

[2023-03-24 11:03:51,364][INFO] Verifying the skill override with index 1
[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args
        Missing parameters: {'headers'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.parameters
        Missing parameters: {'ids', 'vs_currencies', 'symbol'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args
        Missing parameters: {'tendermint_com_url', 'termination_sleep', 'reset_tendermint_after', 'multisend_address', 'validate_timeout', 'init_fallback_gas', 'tendermint_url', 'round_timeout_seconds', 'use_termination', 'max_healthcheck', 'tendermint_p2p_url'}

[2023-03-24 11:03:51,364][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args.setup
        Missing parameters: {'oracle_contract_address'}

[2023-03-24 11:03:51,364][INFO] Verifying the skill override with index 2
[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args
        Missing parameters: {'response_index'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.headers
        Missing parameters: {'$', 'c', '}', 'd', 'i', '{', 't', ':'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.parameters
        Missing parameters: {'vs_currencies', 'ids', 'pair', 'symbol'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args
        Missing parameters: {'tendermint_com_url', 'termination_sleep', 'reset_tendermint_after', 'multisend_address', 'validate_timeout', 'init_fallback_gas', 'tendermint_url', 'round_timeout_seconds', 'use_termination', 'max_healthcheck', 'tendermint_p2p_url'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args.setup
        Missing parameters: {'oracle_contract_address'}

[2023-03-24 11:03:51,365][INFO] Verifying the skill override with index 3
[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args
        Missing parameters: {'response_index'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.headers
        Missing parameters: {'$', 'c', '}', 'd', 'i', '{', 't', ':'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.price_api.args.parameters
        Missing parameters: {'ids', 'vs_currencies', 'pair'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args
        Missing parameters: {'tendermint_com_url', 'termination_sleep', 'reset_tendermint_after', 'multisend_address', 'validate_timeout', 'init_fallback_gas', 'tendermint_url', 'round_timeout_seconds', 'use_termination', 'max_healthcheck', 'tendermint_p2p_url'}

[2023-03-24 11:03:51,365][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (service, valory/oracle_hardhat:0.1.0)
        Path: models.params.args.setup
        Missing parameters: {'oracle_contract_address'}

[2023-03-24 11:03:51,366][INFO] Cross verifying overrides between skill and agent
[2023-03-24 11:03:51,366][WARNING] (agent, valory/oracle:0.1.0) contains configuration which is missing from (skill, valory/oracle_abci:0.1.0)
        Path: models
        Missing parameters: {'price_api', 'randomness_api', 'server_api'}

[2023-03-24 11:03:51,366][WARNING] (skill, valory/oracle_abci:0.1.0) contains configuration which is missing from (agent, valory/oracle:0.1.0)
        Path: models.params.args
        Missing parameters: {'cleanup_history_depth_current', 'oracle', 'genesis_config', 'retry_attempts', 'drand_public_key', 'request_retry_delay', 'cleanup_history_depth', 'tendermint_max_retries', 'observation_aggregator_function', 'tendermint_check_sleep_delay', 'finalize_timeout', 'tx_timeout', 'history_check_timeout', 'retry_timeout', 'keeper_timeout', 'request_timeout', 'keeper_allowed_retries', 'sleep_time', 'service_id', 'max_attempts'}

Service is ready to be deployed.

This makes it extremely hard to understand if something is going wrong.

Not being able to run the command on CI defeats its purpose

We should add the tool to our workflows, otherwise, we can still end up deploying a problematic service by human error.

Adamantios commented 1 year ago

Another issue I just came across on https://github.com/valory-xyz/apy-oracle/pull/51/commits/25cde2cfa980f19d7115c33a20dbed45591a6e63:

Running:

autonomy analyse service --public-id valory/apy_estimation_demo

Gives:

[2023-03-24 19:14:31,800][INFO] Validating service overrides
[2023-03-24 19:14:31,800][INFO] Validating (skill, valory/apy_estimation_abci:0.1.0) from the service overrides
[2023-03-24 19:14:31,800][INFO] Validating agent overrides
[2023-03-24 19:14:31,800][INFO] Validating skill overrides
[2023-03-24 19:14:31,800][INFO] Cross verifying overrides between agent and service
Error: Service config has an overrides which are not defined in the agent config; packages with missing overrides={PackageId(skill, valory/apy_estimation_abci:0.1.0)}

This does not report what is missing from the agent config, which ends up requiring manual investigation.

Moreover, the error says that the override is missing from the agent configuration, but the reported package is a skill. Which one is true?

Adamantios commented 1 year ago

Regarding the APY issue, as @angrybayblade correctly pointed, the skill override is incorrect and instead has to be:

public_id: valory/apy_estimation_chained_abci:0.1.0

Relevant fix is on https://github.com/valory-xyz/apy-oracle/pull/51/commits/8a1a6184c8d489a48b57517f962609addabc36b7.

Two points:

  1. The analysis tool did not report this and failed to identify the reason. If we add this to the reported errors it is going to save a lot of time for the devs in the future.
  2. Why did this only start failing after open-autonomy v0.10.0 release?
0xArdi commented 1 year ago

Warnings should be raised in cases when degenerate rounds are not chained.

DavidMinarsch commented 1 year ago

[2023-04-01 21:43:14,049][INFO] Validating skill overrides Error: ABCI skill model parameter validation failed; 'reset_pause_duration' is a required property

It's not telling me which skill...

DavidMinarsch commented 1 year ago

Overall very helpful command though!

DavidMinarsch commented 1 year ago

thon3.10/site-packages/aea/helpers/env_vars.py", line 144, in convert_value_str_to_type raise KeyError(f"{type_str} is not a valid python data type") KeyError: 'null is not a valid python data type'

This was not caught by the analyzer.

DavidMinarsch commented 1 year ago

And one more that could have been spotted:

Error: Package loading error: An error occurred while loading skill valory/governatooorr_abci:0.1.0: Traceback (most recent call last):

File "/home/ubuntu/agent/vendor/valory/skills/abstract_round_abci/models.py", line 108, in _ensure enforce(

File "/home/ubuntu/.local/lib/python3.10/site-packages/aea/exceptions.py", line 106, in enforce raise exception_class(exception_text)

aea.exceptions.AEAEnforceError: 'use_termination' of type '<class 'bool'>' required, but it is not set in models.params.args of skill.yaml of valory/governatooorr_abci:0.1.0

DavidMinarsch commented 1 year ago 
All certificates have been issued.
`--aev` flag is deprecated and will be removed in v2.0.0, usage of envrionment varibales is default now.
[2023-04-02 09:48:16,379] [WARNING] [agent] The kwargs={'broadcast_to_server': False, 'history_end': None, 'observation_interval': 300} passed to params have not been set!
Error: Package loading error: An error occurred while loading skill valory/governatooorr_abci:0.1.0: Traceback (most recent call last):

  File "/home/ubuntu/agent/vendor/valory/skills/abstract_round_abci/models.py", line 342, in _ensure_setup
    check_type(key, value, type_)

  File "/home/ubuntu/agent/vendor/valory/skills/abstract_round_abci/utils.py", line 455, in check_type
    raise err

packages.valory.skills.abstract_round_abci.utils.AutonomyTypeError: Error in field 'safe_contract_address'. Expected type <class 'str'>, got <class 'list'> (value: ['0x0000000000000000000000000000000000000000'])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/ubuntu/agent/vendor/valory/skills/abstract_round_abci/models.py", line 344, in _ensure_setup
    enforce(

  File "/home/ubuntu/.local/lib/python3.10/site-packages/aea/exceptions.py", line 106, in enforce
    raise exception_class(exception_text)

aea.exceptions.AEAEnforceError: 'safe_contract_address' must be a <class 'str'>, but type <class 'list'> was found in `models.params.args.setup` of `skill.yaml` of `valory/governatooorr_abci:0.1.0`
Adamantios commented 1 year ago

Service analyser did not detect this: https://github.com/valory-xyz/governatooorr/pull/14