safe events task - string indices must be integers

[InoMurko]

Describe the bug Is this a known issue? Transaction with which a safe was created on L2 Boba Network (optimistic rollups) https://blockexplorer.boba.network/tx/0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69/internal-transactions

boba_ops-worker-1                          | 2021-12-21 15:20:06,790 [INFO] [a64c0933/index_safe_events_task] SafeEventsIndexer: Found 2 events from block-number=252954 to block-number=252965 for 1 addresses: ['0xfb1bffC9d739B8D520DaF37dF666da4C687191EA']
boba_ops-worker-1                          | 2021-12-21 15:20:06,874 [INFO] [2b290610/index_new_proxies_task] ProxyFactoryIndexer: Processing 1 almost updated addresses
boba_ops-worker-1                          | 2021-12-21 15:20:06,948 [ERROR] [a64c0933/index_safe_events_task] Task safe_transaction_service.history.tasks.index_safe_events_task[a64c0933-add0-4672-ae46-7f692655ec6f] raised unexpected: TypeError('string indices must be integers')
boba_ops-worker-1                          | Traceback (most recent call last):
boba_ops-worker-1                          |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
boba_ops-worker-1                          |     R = retval = fun(*args, **kwargs)
boba_ops-worker-1                          |   File "/usr/local/lib/python3.9/site-packages/celery/app/trace.py", line 734, in __protected_call__
boba_ops-worker-1                          |     return self.run(*args, **kwargs)
boba_ops-worker-1                          |   File "/usr/local/lib/python3.9/site-packages/celery/app/autoretry.py", line 34, in run
boba_ops-worker-1                          |     return task._orig_run(*args, **kwargs)
boba_ops-worker-1                          |   File "/app/safe_transaction_service/history/tasks.py", line 174, in index_safe_events_task
boba_ops-worker-1                          |     number = SafeEventsIndexerProvider().start()
boba_ops-worker-1                          |   File "/app/safe_transaction_service/history/indexers/ethereum_indexer.py", line 383, in start
boba_ops-worker-1                          |     processed_elements, updated = self.process_addresses(
boba_ops-worker-1                          |   File "/app/safe_transaction_service/history/indexers/ethereum_indexer.py", line 341, in process_addresses
boba_ops-worker-1                          |     processed_elements = self.process_elements(elements)
boba_ops-worker-1                          |   File "/app/safe_transaction_service/history/indexers/events_indexer.py", line 206, in process_elements
boba_ops-worker-1                          |     self.index_service.txs_create_or_update_from_tx_hashes(tx_hashes)
boba_ops-worker-1                          |   File "/app/safe_transaction_service/history/services/index_service.py", line 134, in txs_create_or_update_from_tx_hashes
boba_ops-worker-1                          |     self.ethereum_client.get_transaction_receipts(tx_hashes_not_in_db),
boba_ops-worker-1                          |   File "/usr/local/lib/python3.9/site-packages/gnosis/eth/ethereum_client.py", line 1656, in get_transaction_receipts
boba_ops-worker-1                          |     for result in sorted(results, key=lambda x: x["id"]):
boba_ops-worker-1                          |   File "/usr/local/lib/python3.9/site-packages/gnosis/eth/ethereum_client.py", line 1656, in <lambda>
boba_ops-worker-1                          |     for result in sorted(results, key=lambda x: x["id"]):
boba_ops-worker-1                          | TypeError: string indices must be integers
boba_ops-worker-1                          | 2021-12-21 15:20:07,024 [INFO] [2b290610/index_new_proxies_task] ProxyFactoryIndexer: Found 1 events from block-number=252954 to block-number=252965 for 1 addresses: ['0xC22834581EbC8527d974F8a1c97E1bEA4EF910BC']

To Reproduce Creating a safe with 1 owner.

Expected behavior A clear and concise description of what you expected to happen.

Environment (please complete the following information):

• production mainnet
• Boba Network
• OSX
• Brave
• 7ee78f9f80de1b25a1301d8c92c6430e0ef008ea 3.4.17

Additional context Add any other context about the problem here.

[Uxio0]

It's not a known issue. Looks like a problem of the node when getting transaction receipts

[InoMurko]

The Ethereum client serving jsonrpc requests?

[InoMurko]

Gathering some more data:

AttributeDict({'address': '0x9033dA73B23A4778d0E70314D87aae398ECB916D', 'blockHash': HexBytes('0xc173bbc3b27e8b581cffbc941ff9a12f203f4eaf58165c7ef11327d6190322e4'), 'blockNumber': 252964, 'data': '0x000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000017062a1de2fe6b99be3d9d37841fed19f57380400000000000000000000000000000000000000000000000000000000000000010000000000000000000000003c99545602ff72723e813dc31f696cd893e5e525', 
'logIndex': 0, 'removed': False, 'topics': [HexBytes('0x141df868a6331af528e38c83b7aa03edc19be66e37ae67f9285bf4f8e3c6a1a8'), HexBytes('0x000000000000000000000000c22834581ebc8527d974f8a1c97e1bea4ef910bc')], 
'transactionHash': HexBytes('0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69'), 'transactionIndex': 0}), 

AttributeDict({'address': '0xC22834581EbC8527d974F8a1c97E1bEA4EF910BC', 'blockHash': HexBytes('0xc173bbc3b27e8b581cffbc941ff9a12f203f4eaf58165c7ef11327d6190322e4'), 'blockNumber': 252964, 'data': '0x0000000000000000000000009033da73b23a4778d0e70314d87aae398ecb916d000000000000000000000000fb1bffc9d739b8d520daf37df666da4c687191ea', 
'logIndex': 1, 'removed': False, 'topics': [HexBytes('0x4f51faf6c4561ff95f067657e43439f0f856d97c04d9ec9070a6199ad418e235')], 
'transactionHash': HexBytes('0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69'), 'transactionIndex': 0})

[Uxio0]

@InoMurko Could you provide a RPC so I can debug the issue?

[InoMurko]

https://docs.boba.network/developer-docs/006_parameters-mainnet

Mainnet RPC https://mainnet.boba.network

[InoMurko]

It seems like there's no transaction_receipt for a tx_hash.

boba_ops-worker-1                          | 2021-12-22 09:56:07,674 [INFO] [c9908b3e/index_safe_events_task] tx_hashes_not_in_db ['0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69']

because later on, this fails:

        logger.info("get_transaction_receipts %s",
                    self.ethereum_client.get_transaction_receipts(tx_hashes_not_in_db))

curl https://mainnet.boba.network/ -X POST  -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":1}'

{
  "jsonrpc": "2.0",
  "result": {
    "blockHash": "0xc173bbc3b27e8b581cffbc941ff9a12f203f4eaf58165c7ef11327d6190322e4",
    "blockNumber": "0x3dc24",
    "contractAddress": null,
    "cumulativeGasUsed": "0x402fa",
    "from": "0x3c99545602ff72723e813dc31f696cd893e5e525",
    "gasUsed": "0x402fa",
    "l1Fee": "0x2bef",
    "l1FeeScalar": "1.5",
    "l1GasPrice": "0x1",
    "l1GasUsed": "0x1d4a",
    "logs": [
      {
        "address": "0x9033da73b23a4778d0e70314d87aae398ecb916d",
        "blockHash": "0xc173bbc3b27e8b581cffbc941ff9a12f203f4eaf58165c7ef11327d6190322e4",
        "blockNumber": "0x3dc24",
        "data": "0x000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000017062a1de2fe6b99be3d9d37841fed19f57380400000000000000000000000000000000000000000000000000000000000000010000000000000000000000003c99545602ff72723e813dc31f696cd893e5e525",
        "logIndex": "0x0",
        "removed": false,
        "topics": [
          "0x141df868a6331af528e38c83b7aa03edc19be66e37ae67f9285bf4f8e3c6a1a8",
          "0x000000000000000000000000c22834581ebc8527d974f8a1c97e1bea4ef910bc"
        ],
        "transactionHash": "0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69",
        "transactionIndex": "0x0"
      },
      {
        "address": "0xc22834581ebc8527d974f8a1c97e1bea4ef910bc",
        "blockHash": "0xc173bbc3b27e8b581cffbc941ff9a12f203f4eaf58165c7ef11327d6190322e4",
        "blockNumber": "0x3dc24",
        "data": "0x0000000000000000000000009033da73b23a4778d0e70314d87aae398ecb916d000000000000000000000000fb1bffc9d739b8d520daf37df666da4c687191ea",
        "logIndex": "0x1",
        "removed": false,
        "topics": [
          "0x4f51faf6c4561ff95f067657e43439f0f856d97c04d9ec9070a6199ad418e235"
        ],
        "transactionHash": "0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69",
        "transactionIndex": "0x0"
      }
    ],
    "logsBloom": "0x00080000000000000000000000000000000000000000000000000000000000000000000000000040000000000200000000000000000000000000000000000000000010000000000000000000000000000010000000000008000000000000000000000000008000000500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000202000000000000000000000000000000000000000000000000000000080000000000000000000000000020000000000008000000000000000001",
    "status": "0x1",
    "to": "0xc22834581ebc8527d974f8a1c97e1bea4ef910bc",
    "transactionHash": "0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69",
    "transactionIndex": "0x0"
  },
  "id": 1
}

[InoMurko]

I think that perhaps what is going on is that the python ethereum client library is sending RPC requests with a string ID:

When we receive a string id, we also return a string id (there's a proxy in between). Because when we receive an integer id, we return an integer id.

curl https://mainnet.boba.network/ -X POST  -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":1}' | jq .id
1

curl https://mainnet.boba.network/ -X POST  -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":"1"}' | jq .id
"1"

Hence the lambda sorting sorted(results, key=lambda x: x["id"]) fails because the ID's in response are strings. Wdyt?

[Uxio0]

No, the problem is with the batch requests:

curl https://mainnet.boba.network/ -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":1}]'

{"jsonrpc":"2.0","error":{"code":-32014,"message":"invalid bacth"},"id":null}

Example with mainnet network:

curl https://mainnet.infura.io/v3/<your-id> -X POST -H "Content-Type: application/json" -d '[{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":1},{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":2}]'

[{"jsonrpc":"2.0","id":1,"result":null},{"jsonrpc":"2.0","id":2,"result":null}]

[InoMurko]

@Uxio0 thanks.

curl https://lightning-replica.boba.network/ -X POST -H "Content-Type: application/json" -d '[{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":1},{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params": ["0xc2c9b597c49238d05c2e18ae6ac72b45f0fc16989cf04a27ed923ada06dfad69"],"id":2}]'

The other node has an updated proxy. 🤦🏻

[Uxio0]

Nice, I'm glad it's fixed