Investigate 'Transaction has a bad signature' error on Foucoco

[ebma]

We are currently facing issues with the following error message in our spacewalk testing-service and faucet service which are trying to submit extrinsics on Foucoco:

## Faucet Service
2024-02-13 14:00:50        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Transaction has a bad signature

## Testing Service
2024-02-13 23:32:06        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Transaction has a bad signature
2024-02-13 23:32:06        RPC-CORE: submitAndWatchExtrinsic(extrinsic: Extrinsic): ExtrinsicStatus:: 1010: Invalid Transaction: Transaction has a bad signature
RpcError: 1010: Invalid Transaction: Transaction has a bad signature
    at file:///app/spacewalk-testing-service/dist/vault_service/vault.js:50:24
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///app/spacewalk-testing-service/dist/vault_service/vault.js:18:13 {
  extrinsicCalled: 'Issue Request',
  sender: '6kmmmu4nTuELg5xXqLvfK9TzMsUgmqZEX4iGpMXmQWm71271'
}

We don't know the reason yet, but it seems to have first appeared on Feb 6, see this reported incident. I think at that time we redeployed the collator nodes on Foucoco so it might be related to that collator redeployment. This would mean that there were pending changes to the node which were only applied after the redeployment. But this is not certain.

Submitting extrinsics still seems to work with polkadot.js explorer. So we have to assume that extrinsic submission is only broken when done programmatically (ie. when done with the api exposed by the polkadot.js npm package).

TODO

We can try to boil down the problem by testing the following:

• Try to submit an extrinsic on Foucoco via polkadot.js explorer and programmatically with the polkadot.js package. For this, you can try to create and submit a very basic transaction with the transaction-builder repository. Here, we want to test if even very simple extrinsics (like transferring tokens) fail when done programmatically.
• Try to run Foucoco locally in zombienet and see if the same error is thrown when submitting transactions programmatically.

[ebma]

@pendulum-chain/product we are facing some issues with extrinsic submissions on Foucoco so we are investigating in this ticket.

[prayagd]

Hey team! Please add your planning poker estimate with Zenhub @b-yap @bogdanS98 @ebma @gianfra-t @TorstenStueber

[TorstenStueber]

Was there a runtime upgrade in between? The signature of a transaction is not just over the transaction itself but uses more signing material, such as the current spec version and transaction version. A transaction created before a runtime upgrade will have an invalid signature if executed after a runtime upgrade.

This happens in

• the runtime executive
• which calls the check function of UncheckedExtrinsic
• which adds all additional signing material from all signed extensions
• and the CheckSpecVersion and CheckTxVersion will add the spec and transaction versions

I also suggest to check whether any of the other signed extensions provides signing material that is unexpected by the chain. Maybe easiest to rerun the block with blockchain but use a locally compiled version of the runtime with extra log output (see here).

[ebma]

The fix was simpler than we thought. All we had to do was restart both the testing service and faucet and the error was gone(thanks @gianfra-t for the idea). The problem was probably related to the fact that the api objects are cached and re-used in both services. As you pointed out @TorstenStueber, we probably executed a runtime upgrade in the meantime which made the cached api object unable to create valid signatures.

With this in mind, we can close this ticket as completed but I'll create follow-up tickets to add handling for this case to the services.