mit-dci / opencbdc-tx

A transaction processor for a hypothetical, general-purpose, central bank digital currency
Other
896 stars 198 forks source link

Decouple atomizer component dependencies #158

Open jallen-frb opened 2 years ago

jallen-frb commented 2 years ago

Affected Branch

trunk

Basic Diagnostics

Description

In order to reproduce the issue, follow these steps:

  1. Run docker compose --file docker-compose-atomizer.yml up
  2. The watchtower comes online slowly which chains failures to connect to it and containers exit.

A quick fix is to add restart:always to each service in docker-compose-atomizer.yml, but @metalicjames mentioned that we should not rely on restarting containers/services. Each service should poll to see if the others are online without exiting.

Code of Conduct

mszulcz-mitre commented 2 years ago

Pull Request #157 seems relevant. I'm not familiar with the expected output from Step 1 in the bug description, but when I do Step 1 with the commit in #157, the output eventually stops with the sentinel being "Done":

image

Without the commit, the output continues for a long time (maybe forever) as the watchtower repeatedly attempts to connect:

image

Maybe we should consider adding the changes in #157 for the atomizer controller to the controllers of the other components so that they try multiple times to connect without failing. Here's the changes to the atomizer controller: image