Lido Validator Ejector

Daemon service which loads LidoOracle events for validator exits and sends out exit messages when necessary.

On start, it will load events from a configurable amount of blocks behind and then poll for new events.

Requirements

• Folder of pre-signed exit messages as individual JSON files in either spec format (generic) or ethdo output format
• Execution node
• Consensus node

This service has to be run in a single instance as it expects to fulfil every request to exit. Each unfulfilled request (no exit message being present for required validator) will log an error.

Configuration

Operation Modes

For both modes, Ejector will monitor exit request events, but react to them differently.

Messages Mode

In this mode, Ejector will load pre-signed exit messages from .json files on start, validate them, and submit them to a CL node when necessary.

Mode is activated by setting the MESSAGES_LOCATION variable.

Webhook Mode

In this mode, Ejector will make a request to a specified endpoint when an exit needs to be made instead of submitting a pre-signed exit message to a CL node.

Mode is activated by setting the VALIDATOR_EXIT_WEBHOOK variable.

This allows NOs to implement JIT approach by offloading exiting logic to an external service and using the Ejector as a secure exit events reader.

On the endpoint, JSON will be POSTed with the following structure:

{
  "validatorIndex": "123",
  "validatorPubkey": "0x123"
}

200 response from the endpoint will be counted as a successful exit, non-200 as a fail.

Environment Variables

Options are configured via environment variables.

Messages can also be loaded from remote storages: AWS S3 and Google Cloud Storage.

Simply set a url with an appropriate protocol in MESSAGES_LOCATION:

• s3:// for S3
• gs:// for GCS

Authentication setup: GCS, S3.

Preparing Exit Messages

Once you generate and sign exit messages, you can encrypt them for storage safety.

Exit messages are encrypted and decrypted following the EIP-2335 spec.

You can check a simple example in JS in encryptor folder:

Simply copy JSON exit message files to encryptor/input, set encryption password as MESSAGES_PASSWORD in .env and run:

yarn encrypt

Done, your encrypted files will be in encryptor/output.

Running

Either:

• Use a Docker image from Docker Hub
• Clone repo, install dependencies, build and start the service:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector
yarn
yarn build
yarn start

Don't forget env variables in the last command.

Metrics

Enable metrics endpoint by setting HTTP_PORT=1234 and RUN_METRICS=true environment variables.

Metrics will be available on $HOST:$HTTP_PORT/metrics.

Available metrics:

• exit_messages: ['valid'] - Exit messages and their validity: JSON parseability, structure and signature
• exit_actions: ['result'] - Statuses of initiated validator exits
• event_security_verification: ['result'] - Statuses of exit event security verifications
• polling_last_blocks_duration_seconds: ['eventsNumber'] - Duration of pooling last blocks in microseconds
• execution_request_duration_seconds: ['result', 'status', 'domain'] - Execution node request duration in microseconds
• consensus_request_duration_seconds: ['result', 'status', 'domain'] - Consensus node request duration in microseconds
• job_duration_seconds: ['name', 'interval', 'result'] - Duration of Ejector cycle cron job
• job_message_reloader_duration_seconds: ['name', 'interval', 'result'] - Duration of Pre-signed message reloader cron job
• exit_messages_left_number - Number of exit messages left
• exit_messages_left_percent - Percentage of exit messages left

Safety Features

• Encrypted messages allow for secure file storage
• Invalid files in messages folder are noticed
• Exit JSON structure is checked
• Exit signature is fully validated
• Exit event pubkeys are checked to exist in transaction data
• Exit event report data hashes are checked to match hashes in original submitReport() Oracle transactions
• Exit events original consensus transactions are checked to be signed by allowlisted Oracles
• Node requests are repeated on error or timeouts
• Amount of messages left to send out can be checked using metrics
• Dry run mode to test setup

Troubleshooting

When you try to use Lido Validator Ejector on ARM, you may encounter an unexpected problem related to the inability to install @chainsafe/blst dependencies under darwin arm64.

Why does it happen?!

It happens because https://www.npmjs.com/package/@chainsafe/blst doesn't provide native C binding to https://github.com/supranational/blst under darwin arm64. Such as there no native binding, a user has to compile C binding to blst lab manually for darwin arm64. @chainsafe/blst has compile option but inside itself for downloading dependencies this lib uses Python language. Historically MacOs uses alias python3 for python. So then @chainsafe/blst fails with an error that it could not install all dependencies. To fix it on MacOs just create alias python for python3.

ln -s /opt/homebrew/bin/python3 /usr/local/bin/python

More info here - https://github.com/ChainSafe/lodestar/issues/4767#issuecomment-1640631566