[bug]SNET-CLI is throwing errors when attempting to publish a service

JoeSHMOEK commented 5 months ago

Note: Issue updated to only include errors with SNET-CLI version 5.0.9

JoeSHMOEK commented 5 months ago

When trying to publish the Neuralprophet service on temporai-services-test, snet-cli is throwing the following error:

$ snet --print-traceback service publish temporai neuralprophet
Traceback (most recent call last):
  File "/home/snet/venv/bin/snet", line 8, in <module>
    sys.exit(main())
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/__init__.py", line 23, in main
    getattr(args.cmd(conf, args), args.fn)()
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/commands/mpe_service.py", line 450, in publish_service_with_metadata
    self.args.org_id, self.args.metadata_file)
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/commands/mpe_service.py", line 434, in _validate_service_group_with_org_group_and_update_group_id
    org_metadata = self._get_organization_metadata_from_registry(org_id)
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/commands/mpe_service.py", line 417, in _get_organization_metadata_from_registry
    rez = self._get_organization_registration(org_id)
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/commands/mpe_service.py", line 427, in _get_organization_registration
    "Registry", "getOrganizationById", params)
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/commands/commands.py", line 202, in call_contract_command
    contract_address = get_contract_address(self, contract_name)
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/utils/config.py", line 33, in get_contract_address
    return read_default_contract_address(w3=cmd.w3, contract_name=contract_name)
  File "/home/snet/venv/lib/python3.7/site-packages/snet_cli/utils/config.py", line 42, in read_default_contract_address
    raise Exception()
Exception

To solve the above error it was determined that the snet-cli version must be upgraded.
- This required an upgrade from python 3.7 to 3.11 and a reinstall of the virtual environment
- The new snet-cli is available on PyPy repos and no longer requires a manual install, the snet-cli package can now be installed with the following:
```
pip install snet-cli
```

JoeSHMOEK commented 5 months ago

After the update of snet-cli we are no longer receiving a contract error, but there appears to be a change in the service_metadata.json schema, as depicted by the following error:

$ snet --print-traceback service publish temporai1 temporAITestService2

Traceback (most recent call last):
  File "/home/snet/venv/bin/snet", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/__init__.py", line 23, in main
    getattr(args.cmd(conf, args), args.fn)()
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/mpe_service.py", line 444, in publish_service_with_metadata
    self._validate_service_group_with_org_group_and_update_group_id(
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/mpe_service.py", line 429, in _validate_service_group_with_org_group_and_update_group_id
    org_metadata = self._get_organization_metadata_from_registry(org_id)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/mpe_service.py", line 415, in _get_organization_metadata_from_registry
    self._get_ipfs_client(), metadata_hash)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/commands.py", line 78, in _get_ipfs_client
    return ipfshttpclient.connect(ipfs_endpoint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/ipfshttpclient/client/__init__.py", line 98, in connect
    client = Client(addr, base, chunk_size, session, **defaults)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/ipfshttpclient/client/base.py", line 124, in __init__
    self._client = self._clientfactory(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/ipfshttpclient/http.py", line 171, in __init__
    addr = multiaddr.Multiaddr(addr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/multiaddr/multiaddr.py", line 126, in __init__
    self._bytes = string_to_bytes(addr)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/multiaddr/transforms.py", line 17, in string_to_bytes
    for proto, codec, value in string_iter(string):
  File "/home/snet/venv/lib/python3.11/site-packages/multiaddr/transforms.py", line 55, in string_iter
    raise exceptions.StringParseError("Must begin with /", string)
multiaddr.exceptions.StringParseError: Invalid MultiAddr 'http://ipfs.singularitynet.io:80': Must begin with /

I attempted to reset the the service_metadata.json to its defaults using metadata-init on the new install of snet-cli, however this also results in an error:

snet --print-traceback service metadata-init service.proto "temporAITestService2"
 --group-name default_groups --fixed-price 0.00000001 --endpoints http://127.0.0.1:8010
Traceback (most recent call last):
  File "/home/snet/venv/bin/snet", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/__init__.py", line 23, in main
    getattr(args.cmd(conf, args), args.fn)()
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/mpe_service.py", line 112, in publish_proto_metadata_init
    self._get_ipfs_client(), self.args.protodir)
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/commands.py", line 78, in _get_ipfs_client
    return ipfshttpclient.connect(ipfs_endpoint)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/ipfshttpclient/client/__init__.py", line 98, in connect
    client = Client(addr, base, chunk_size, session, **defaults)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/ipfshttpclient/client/base.py", line 124, in __init__
    self._client = self._clientfactory(
                   ^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/ipfshttpclient/http.py", line 171, in __init__
    addr = multiaddr.Multiaddr(addr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/multiaddr/multiaddr.py", line 126, in __init__
    self._bytes = string_to_bytes(addr)
                  ^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/multiaddr/transforms.py", line 17, in string_to_bytes
    for proto, codec, value in string_iter(string):
  File "/home/snet/venv/lib/python3.11/site-packages/multiaddr/transforms.py", line 55, in string_iter
    raise exceptions.StringParseError("Must begin with /", string)
multiaddr.exceptions.StringParseError: Invalid MultiAddr 'http://ipfs.singularitynet.io:80': Must begin with /

multiaddr package info:

https://pypi.org/project/multiaddr/
https://github.com/multiformats/py-multiaddr
https://github.com/multiformats/multiaddr
https://github.com/multiformats/multiaddr/blob/master/protocols.csv
Github project status is archived, does this mean that the package is no longer maintained?

To move forward, I believe the URI http://ipfs.singularitynet.io:80 will need to be converted to multiaddr spec which is located here: https://github.com/multiformats/multiformats/blob/master/contributing.md#multiformats-protocol-specifications

More leads:

https://pypi.org/project/ipfshttpclient/

Conclusion:

snet-cli is currently broken and can't be deployed using their releases, I've determined this with the following:
- snet-cli version: 2.0.9 (The version used to deploy production Sybil) errors out when attempting to publish a service
- snet-cli verison: 2.1.0 and 2.1.1 errors out when attempting to publish and initialize new service_metadata.jsons
- Publishing a service requires that snet-cli be used as a IPFS hash must be generated for the service.

Next Steps:

It seems that SNET has updated their IPFS client and routing method without updating their addressing spec.
- Workarounds exist if we hack in multiaddr spec in stead of their URI based spec.
- It may also be possible to bypass the SNET IPFS client and in stead replicate it using a manual process. This will be a lengthy process however.

JoeSHMOEK commented 5 months ago

The following is the translation from the URI based routing spec to multiaddr, based on the documents located at https://pypi.org/project/ipfshttpclient/

http://ipfs.singularitynet.io:80
/dns/ipfs.singularitynet.io/tcp/80/http

From reverse engineering snet-cli, IPFS default endpoint is located in a hidden config file in ~/.snet/config, this line was updated:

default_ipfs_endpoint = http://ipfs.singularitynet.io:80
default_ipfs_endpoint = /dns/ipfs.singularitynet.io/tcp/80/http

Solution: The above worked to resolve the IPFS endpoint issue

JoeSHMOEK commented 5 months ago

Configuration output of command needs to be updated with the HTTP configuration:

{
    "version": 1,
    "display_name": "temporAITestService2",
    "encoding": "proto",
    "service_type": "grpc",
    "model_ipfs_hash": "QmTdhXy4NvAckaRqWB4LRnjZM12d1otCRXBubPynzihDDa",
    "mpe_address": "0x6245F856EFFBDB3ED6a3c64385b27A78B42F65e1",
    "groups": [
        {
            "group_name": "default_groups",
            "endpoints": [
                "http://127.0.0.1:8010"
            ],
            "pricing": [
                {
                    "price_model": "fixed_price",
                    "price_in_cogs": 1,
                    "default": true
                }
            ]
        }
    ],
    "assets": {},
    "media": [],
    "tags": []
}

Updated to:

{
    "version": 1,
    "display_name": "temporAITestService2",
    "encoding": "proto",
    "service_type": "http",
    "model_ipfs_hash": "QmNvALivX5zM7kpDjWNVvLmQboREDvsziMUbWosuoySAPQ",
    "mpe_address": "0x6245F856EFFBDB3ED6a3c64385b27A78B42F65e1",
    "groups": [
        {
            "group_name": "temporAItestgroup",
            "endpoints": [
                "http://127.0.0.1:8010"
            ],
            "pricing": [
                {
                    "price_model": "fixed_price",
                    "price_in_cogs": 1,
                    "default": true
                }
            ]
        }
    ],
    "assets": {},
    "media": [],
    "tags": []
}

Pushed to IPFS via the command:

snet --print-traceback service publish --metadata-file service_metadata.json temporai1 temporAITestService2
# Calculating gas price... one moment..
# gas_price = 7.151220 GWei
    transaction:
        chainId: 5
        data: '0xa4123f0f74656d706f72616931000000000000000000000000000000000000000000000074656d706f72414954657374536572766963653200000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000040697066733a2f2f516d5532793365684e6e4165774655756d484e795372586e3261463932416261345777695878516f6658465445620000000000000000000000'
        from: '0x3c550cf595e8E857082531623a5a174495Aa710d'
        gas: 172114
        gasPrice: 7151220332
        nonce: 29
        to: '0x0DD7feC305f2374d7eed35d6d28134936c025A7A'
        value: 0

Proceed? (y/n): y
Submitting transaction...

JoeSHMOEK commented 5 months ago

SNET-CLI is failing to get the correct gas_price on the ETH network. Even worse it seems that the manual override "--gas-price fast" has been removed in SNET-CLI 2.1.1. See below for details:

$ snet --print-traceback service publish --metadata-file service_metadata.json temporai1 temporAITestService2
# Calculating gas price... one moment..
# gas_price = 1.333333 GWei
    transaction:
        chainId: 5
        data: '0xa4123f0f74656d706f72616931000000000000000000000000000000000000000000000074656d706f72414954657374536572766963653200000000000000000000000000000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000040697066733a2f2f516d5532793365684e6e4165774655756d484e795372586e3261463932416261345777695878516f6658465445620000000000000000000000'
        from: '0x3c550cf595e8E857082531623a5a174495Aa710d'
        gas: 172114
        gasPrice: 1333333338
        nonce: 29
        to: '0x0DD7feC305f2374d7eed35d6d28134936c025A7A'
        value: 0

Proceed? (y/n): y
Submitting transaction...

Traceback (most recent call last):
  File "/home/snet/venv/bin/snet", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/__init__.py", line 23, in main
    getattr(args.cmd(conf, args), args.fn)()
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/mpe_service.py", line 451, in publish_service_with_metadata
    self.transact_contract_command(
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/commands.py", line 213, in transact_contract_command
    is_silent).transact()
               ^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/commands/commands.py", line 372, in transact
    receipt = self.ident.transact(txn, self.err_f)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/identity.py", line 56, in transact
    return send_and_wait_for_transaction(raw_transaction, self.w3, out_f)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/snet_cli/identity.py", line 291, in send_and_wait_for_transaction
    txn_hash = w3.eth.send_raw_transaction(raw_transaction)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/web3/eth/eth.py", line 373, in send_raw_transaction
    return self._send_raw_transaction(transaction)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/web3/module.py", line 75, in caller
    result = w3.manager.request_blocking(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/web3/manager.py", line 325, in request_blocking
    return self.formatted_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/snet/venv/lib/python3.11/site-packages/web3/manager.py", line 288, in formatted_response
    raise ValueError(error)
ValueError: {'code': -32000, 'message': 'replacement transaction underpriced'}

Supporting evidence for SNET-CLI gas calculation changes, from line 86 of snet_cli/commands/commands.py:

"""
# Temporally deprecated

class CachedGasPriceStrategy:
    def __init__(self, gas_price_param):
        self.gas_price_param = gas_price_param
        self.cached_gas_price = None

    def __call__(self, w3, transaction_params):
        if self.cached_gas_price is None:
            self.cached_gas_price = int(
                self.calc_gas_price(w3, transaction_params))
        return self.cached_gas_price

    def calc_gas_price(self, w3, transaction_params):
        gas_price_param = self.gas_price_param
        if gas_price_param.isdigit():
            return int(self.gas_price_param)
        if gas_price_param == "fast":
            return fast_gas_price_strategy(w3, transaction_params)
        if gas_price_param == "medium":
            return medium_gas_price_strategy(w3, transaction_params)
        if gas_price_param == "slow":
            return slow_gas_price_strategy(w3, transaction_params)
        raise Exception("Unknown gas price strategy: %s" % gas_price_param)

    def is_going_to_calculate(self):
        return self.cached_gas_price is None and not self.gas_price_param.isdigit()
"""

A workaround is needed to correctly pay the ETH network and publish services

Solution:

It appears that Goerli has been deprecated on Jan 1st 2024
- Source: https://www.alchemy.com/blog/goerli-faucet-deprecation
Perhaps these gas_price errors are a symptom of the deprecation
We should migrate to Sepolia for UAT.

Kevin-Chen0 commented 5 months ago

Thanks for investigating all this and posting this here @JoeSHMOEK. Can you bring this up to the Platform Onboarding Support on MM to get their attention? These are some very serious issues you brought up, and the SNET dev team must address them immediately. For example, they need to be the ones fixing the issues of the correct gas prices, rather than us having to hack up a workaround, esp if we don't even have the manual override capability anymore.

Feel free to contact Soubir for help and see whether he or his Zero2AI team is also encountering these issues.

JoeSHMOEK commented 5 months ago

Limiting the scope of this ticket to discover the root cause of errors involving SNET-CLI errors. Remediation will be taken in follow-up ticket

Tempor-ai / sybil

[bug]SNET-CLI is throwing errors when attempting to publish a service #104