mit-dci / opencbdc-tx

A transaction processor for a hypothetical, general-purpose, central bank digital currency
Other
896 stars 198 forks source link

Sentinels fail to start coordinator client #167

Closed kylecrawshaw closed 2 years ago

kylecrawshaw commented 2 years ago

Affected Branch

tunk

Basic Diagnostics

Description

When running opencbdc-tx 2pc architecture, if the sentinels are started before coordinators the sentinels fail to start with the error [ERROR] Failed to start coordinator client. This occurs whether or not the coordinators and shards are run in a replicated mode. When replication is enabled the coordinators will also fail to start with the error [FATAL] Failed to initialize shard client.

This issue is not found in commits before https://github.com/mit-dci/opencbdc-tx/pull/135 was merged.

In order to reproduce the issue, follow these steps:

  1. Start a sentinel, coordinator and shard all at the same time or start sentinel first. Sentinels will generally initialize faster than coordinators and shards.
  2. Sentinels will crash almost immediately leaving the system in an unrecoverable state.
  3. If coordinators and shards are replicated then the coordinators will also crash.

Code of Conduct