SIM-Error when retreiving result from llm

[AgustinRamiroDiaz]

On staging

jsonrpc-1             | Based on the web page data, I found that all the games are still ongoing since none of them have a score other than "0-0". Therefore, I will respond with the following JSON format:
jsonrpc-1             | 
jsonrpc-1             | ```
jsonrpc-1             | {
jsonrpc-1             |     "score": "-",
jsonrpc-1             |     "winner": -1
jsonrpc-1             | }
jsonrpc-1             | ```
jsonrpc-1             | Error running contract Expecting value: line 1 column 1 (char 0)
jsonrpc-1             | Traceback (most recent call last):
jsonrpc-1             |   File "/app/backend/node/base.py", line 76, in run_contract
jsonrpc-1             |     receipt = await self.genvm.run_contract(
jsonrpc-1             |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonrpc-1             |   File "/app/backend/node/genvm/base.py", line 121, in run_contract
jsonrpc-1             |     await function_to_run(*args)
jsonrpc-1             |   File "<string>", line 66, in resolve
jsonrpc-1             |   File "/usr/local/lib/python3.12/json/__init__.py", line 346, in loads
jsonrpc-1             |     return _default_decoder.decode(s)
jsonrpc-1             |            ^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonrpc-1             |   File "/usr/local/lib/python3.12/json/decoder.py", line 337, in decode
jsonrpc-1             |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
jsonrpc-1             |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonrpc-1             |   File "/usr/local/lib/python3.12/json/decoder.py", line 355, in raw_decode
jsonrpc-1             |     raise JSONDecodeError("Expecting value", s, err.value) from None
jsonrpc-1             | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
jsonrpc-1             | 
jsonrpc-1             | DBClient ~ ~ query: UPDATE transactions SET status = %s WHERE id = 53
jsonrpc-1             | DBClient ~ ~ params: ['COMMITTING']
jsonrpc-1             | Error running contract 'NoneType' object is not subscriptable
jsonrpc-1             | Traceback (most recent call last):
jsonrpc-1             |   File "/app/backend/node/base.py", line 76, in run_contract
jsonrpc-1             |     receipt = await self.genvm.run_contract(
jsonrpc-1             |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonrpc-1             |   File "/app/backend/node/genvm/base.py", line 116, in run_contract
jsonrpc-1             |     leader_receipt_eq_result = leader_receipt["result"]["eq_outputs"]["leader"]
jsonrpc-1             |                                ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
jsonrpc-1             | TypeError: 'NoneType' object is not subscriptable
jsonrpc-1             | 
jsonrpc-1             | Error running contract 'NoneType' object is not subscriptable
jsonrpc-1             | Traceback (most recent call last):
jsonrpc-1             |   File "/app/backend/node/base.py", line 76, in run_contract
jsonrpc-1             |     receipt = await self.genvm.run_contract(
jsonrpc-1             |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jsonrpc-1             |   File "/app/backend/node/genvm/base.py", line 116, in run_contract
jsonrpc-1             |     leader_receipt_eq_result = leader_receipt["result"]["eq_outputs"]["leader"]
jsonrpc-1             |                                ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
jsonrpc-1             | TypeError: 'NoneType' object is not subscriptable
jsonrpc-1             | 
jsonrpc-1             | DBClient ~ ~ query: UPDATE transactions SET status = %s WHERE id = 53
jsonrpc-1             | DBClient ~ ~ params: ['REVEALING']
jsonrpc-1             | DBClient ~ ~ query: UPDATE transactions SET status = %s WHERE id = 53
jsonrpc-1             | DBClient ~ ~ params: ['ACCEPTED']
jsonrpc-1             | Error running consensus 'NoneType' object is not subscriptable
jsonrpc-1             | Traceback (most recent call last):
jsonrpc-1             |   File "/app/backend/consensus/base.py", line 71, in _run_consensus
jsonrpc-1             |     await asyncio.gather(*tasks)
jsonrpc-1             |   File "/app/backend/consensus/base.py", line 183, in exec_transaction
jsonrpc-1             |     leader_receipt["result"]["contract_state"]
jsonrpc-1             |     ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
jsonrpc-1             | TypeError: 'NoneType' object is not subscriptable
jsonrpc-1             | 
jsonrpc-1             | INFO | 172.23.0.1 - - [11/Jul/2024 16:43:17] "OPTIONS /api HTTP/1.1" 200 -
jsonrpc-1             | INFO | 172.23.0.1 - - [11/Jul/2024 16:43:17] "OPTIONS /api HTTP/1.1" 200 -
jsonrpc-1             | INFO | 172.23.0.1 - - [11/Jul/2024 16:43:17] "OPTIONS /api HTTP/1.1" 200 -
jsonrpc-1             | INFO | 172.23.0.1 - - [11/Jul/2024 16:43:17] "OPTIONS /api HTTP/1.1" 200 -
jsonrpc-1             | INFO | get_transaction_by_id: {'status': 'info', 'message': 'Starting...', 'data': {}}
jsonrpc-1             | DBClient ~ ~ query: SELECT * FROM transactions WHERE id = 13
jsonrpc-1             | DBClient ~ ~ params: None

[AgustinRamiroDiaz]

After a thorough testing with @bradleySuira we found that the issue is due to problems with the contract when interacting with LLMs: LLMs sometimes produce output that doesn't follow the JSON schema that you ask them for. This is usually because of word padding like "Here's your answer". We found out that expressing the necessity of not putting more words helps the LLMs produce exactly the JSON and nothing more, but it's not a solution and rather a mitigation

I was able to reproduce by using tinillama and also llama3 models.

This should be fixed in many ways:

• errors in the contract should be handled in such a way that they don't impact the consensus algorithm
• transactions should never get stuck (in this case they get stuck in "ACCEPTING"):
  • probably the solution for this would be to have multiple queues for each state
  • transactions should have a maximum attempts

[AgustinRamiroDiaz]

@cristiam86 when was this fixed?

[cristiam86]

@cristiam86 when was this fixed?

Sorry, my mistake

[AgustinRamiroDiaz]

I see that you've closed it again :confused: