Open petermetz opened 3 months ago
Looks like Orion isn't configured correctly in the cactus-besu-all-in-one:2024-07-04-8c030ae
.
@ruzell22 please check if the issue persists on the latest tag as well or not. If the error still persists, we will increase the scope of this task with a potential fix for the same.
UPDATE: I will create an issue to remove tessera, orion base images from the besu aio (besu mp has it and is used for private tx)
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After Retry-After - HTTP | MDN The Retry-After response HTTP header indicates how long the user agent should wait before making a follow-up request. There are three main cases this header is used:
Exponential back-off - the delay between retries is increasing exponentially with every retry. So first you wait 2 seconds, 4, 8, etc.
Let's say start with the run-transaction endpoint of the besu connector and do these things:
BesuLedgerInaccessibleException
which is a sub-class of Error
and you throw this when you detect the issue in the connector that a transaction was not possible to be executed because the network connection to the ledger is broken.catch
block that sets the retry-after headers (linked above) but it only should do this if the exception you caught (that was thrown by the connector) is an instanceof BesuLedgerInaccessibleException
so that way we know for sure that this was the problem.@ruzell22 I figured it out! The issue is deep within the web3 library itself that was causing the crashes. I pushed a commit to your branch with fixes that make it work so that it doesn't crash the entire nodejs process anymore when the connection is down to the ledger.
The other issue with the code was that you were having the re-try logic in the catch block within the endpoint request handler and this was trying to modify the http headers after the response was sent which was another thing that was ALSO crashing the entire nodejs process (which is why seemingly none of my fixes worked at first that were targeted at the web3 library itself).
So now the code is like this:
What I need you to do to finish the task:
To verify the changes you make, take a look at this snippet (that I'll also add to the cmd-api-server readme). You can use this to mount your local changes on the besu connector code into the api server container and then configure the api server to not install from the registry but instead import the local package you have on your file-system (this is what I used to verify my fixes instead of publishing to npm every time)
--volume
parameter that mounts the project directory from the host machine into the container. Second the packageSrc
parameter pointing to the plugin's source directory on the file-system.
docker run \
--rm \
--volume ${PWD}:/usr/src/cacti \
--publish 3000:3000 \
--publish 4000:4000 \
--env AUTHORIZATION_PROTOCOL='NONE' \
--env AUTHORIZATION_CONFIG_JSON='{}' \
--env GRPC_TLS_ENABLED=false \
--env API_TLS_CERT_PEM=- \
--env API_TLS_CLIENT_CA_PEM=- \
--env API_TLS_KEY_PEM=- \
--env API_TLS_ENABLED=false \
--env API_MTLS_ENABLED=false \
--env API_HOST=0.0.0.0 \
--env LOG_LEVEL=DEBUG \
--env PLUGINS='[{"packageName": "@hyperledger/cactus-plugin-ledger-connector-besu", "type": "org.hyperledger.cactus.plugin_import_type.LOCAL", "action": "org.hyperledger.cactus.plugin_import_action.INSTALL", "options": { "packageSrc": "/usr/src/cacti/packages/cactus-plugin-ledger-connector-besu", "rpcApiWsHost": "http://127.0.0.1:8546", "rpcApiHttpHost": "http://127.0.0.1:8545", "instanceId": "some-unique-besu-connector-instance-id"}}]' \
cas
Description
Right now the API server will exit with a crash if the Besu connector receives a request and the backing ledger is not accessible. Instead of a full-on crash, we need to treat this as a recoverable error and send back
HTTP 503 - Service Unavailable
with a Retry-After header with exponential backoff where it starts from 1 second and the doubles every re-try.The easiest way to reproduce is to follow this guide and remove the part of the command from step 2 that binds the API server to the host machine's network:
--network=host \
https://github.com/hyperledger/cacti/tree/main/packages/cactus-plugin-ledger-connector-besu#testing-api-calls-with-the-container
Acceptance Criteria