[ERROR] Failed to connect to any atomizers at Setup test wallets and test them

[UrsaEli]

Affected Branch

main @ https://github.com/mit-dci/opencbdc-tx.git

Basic Diagnostics

• [X] I've pulled the latest changes on the affected branch and the issue is still present.

• [ ] The issue is reproducible in docker

Description

While running "Setup test wallets and test them": ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat mint 10 5

[2022-02-09 19:08:44.231] [WARN ] Existing wallet file not found [2022-02-09 19:08:44.232] [WARN ] Existing client file not found

following, we see an error: """ ./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat info Balance: $0.50, UTXOs: 10, pending TXs: 0 root@88cc0c05fa6d:/opt/tx-processor# Balance: $0.50, UTXOs: 10, pending TXs: 0 bash: Balance:: command not found root@88cc0c05fa6d:/opt/tx-processor# ./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat sync [2022-02-09 19:10:34.623] [ERROR] Failed to connect to any atomizers terminate called after throwing an instance of 'std::system_error' what(): Invalid argument """

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[HalosGhost]

There are two problems here:

1. You ran the output of the info command as a command itself (that's why the shell reported Balance:: command not found: it attempted to, at your request, run Balance: $0.50, UTXOs: 10, pending TXs: 0 as a command).
2. You're running the 2PC architecture, which doesn't use atomizers. You can run multiple copies of the system at once (thanks to docker), but a container can only be connected to one copy of the system, so whichever one you start is the one that is being used (as I said above, in this case, that's the 2PC architecture).

[UrsaEli]

docker run --network atomizer-network -ti opencbdc-tx /bin/bash docker: Error response from daemon: network atomizer-network not found. ERRO[0000] error waiting for container: context canceled

[HalosGhost]

Mmm, that's interesting. Did you rerun the docker-compose command to bring up the Atomizer version of the system? (on first blush, it looks like not).

[UrsaEli]

yes, the error did not happen for: --network 2pc-network but error happens for: --network atomizer-network

[HalosGhost]

Just to be clear, you cannot just run the docker run command you mentioned. You'll also need to run the Atomizer version of the system. So, the two commands you would run are as follows:

$ docker-compose --file docker-compose-atomizer.yml up --detach
[output snipped]
$ docker run --network atomizer-network -ti opencbdc-tx /bin/bash

That will run the system with the Atomizer architecture and a new container connected to the atomizer network (which is setup by the first command).

[UrsaEli]

It runs now without error,

docker run --network atomizer-network -ti opencbdc-tx /bin/bash
root@38ca8d7cc70e:/opt/tx-processor

however, now at

./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat mint 10 5

[2022-02-09 20:35:17.724] [WARN ] Existing wallet file not found
[2022-02-09 20:35:17.724] [WARN ] Existing client file not found
[2022-02-09 20:35:18.145] [ERROR] Failed to send mint tx

same error follows:

./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat sync
[2022-02-09 20:35:32.607] [ERROR] Failed to connect to any atomizers
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument
Aborted

[HalosGhost]

You need to consistently refer to the configuration that you're interacting with. In the first command, you referenced the 2pc-compose.cfg (which won't work when you're interacting with the Atomizer system).

I would suggest starting fresh, choosing which version of the system you want to test, and then working through the commands consistently referring to the correct configuration.

[UrsaEli]

pardon, I meant, I run:

docker-compose --file docker-compose-atomizer.yml up
docker run --network atomizer-network -ti opencbdc-tx /bin/bash

[UrsaEli]

docker-compose --file docker-compose-atomizer.yml up --remove-orphans --detach

[output snipped]

docker run --network atomizer-network -ti opencbdc-tx /bin/bash
./build/src/uhs/client/client-cli 2pc-compose.cfg mempool0.dat wallet0.dat mint 10 5
[2022-02-09 22:02:23.299] [WARN ] Existing wallet file not found
[2022-02-09 22:02:23.299] [WARN ] Existing client file not found
[2022-02-09 22:02:23.334] [ERROR] Failed to send mint tx

./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat sync

**[2022-02-09 22:02:35.901] [ERROR] Failed to connect to any atomizers
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument
Aborted**

[HalosGhost]

Your mint command is trying to use the 2PC compose configuration.

[UrsaEli]

./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat mint 10 5
[2022-02-09 22:26:19.386] [WARN ] Existing wallet file not found
[2022-02-09 22:26:19.387] [WARN ] Existing client file not found
[2022-02-09 22:26:19.404] [ERROR] Failed to connect to any atomizers
terminate called after throwing an instance of 'std::system_error'
  what():  Invalid argument
Aborted

[HalosGhost]

This still looks like the system running isn't right. Can you please post the commands from the beginning and include all output (it's also helpful if you put them in code blocks so they'll be rendered pleasantly):

1. docker-compose --file docker-compose-atomizer.yml up --remove-orphans --detach
2. docker run --network atomizer-network -ti opencbdc-tx /bin/bash
3. (inside the second container): ./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat mint 10 5

[UrsaEli]

I am still experiencing the error.

[HalosGhost]

You show two open containers there (from which you could run client commands), the one on the left was definitely started after the system was run (we know this because it was run in the same window). Is that also true of the one on the right? Is there a reason you opened two containers for client commands at the same time?

[UrsaEli]

Isn't that what it means

(inside the second container):

./build/src/uhs/client/client-cli atomizer-compose.cfg mempool0.dat wallet0.dat mint 10 5

?

I did the same for the first window, and it gives the same error.

[HalosGhost]

I suppose “second” is a little unclear above. I was referring to running the docker run command one time and running the commands in that resulting shell.

This is now enough information to demonstrate that the code appears to be being run as-expected (sorry that was kind of a chore to get to that point!); so I've reopened the issue.

Can you provide (ideally in text code-blocks, not screenshots) the output of docker container ls after you run the docker-compose command?

[UrsaEli]

bash-3.2# docker-compose --file docker-compose-atomizer.yml up --remove-orphans --detach
[+] Running 5/5
 ⠿ Container opencbdc-tx-watchtower0-1  Started                            0.4s
 ⠿ Container opencbdc-tx-atomizer0-1    Started                            4.6s
 ⠿ Container opencbdc-tx-archiver0-1    Started                            0.0s
 ⠿ Container opencbdc-tx-shard0-1       Started                            0.0s
 ⠿ Container opencbdc-tx-sentinel0-1    Started                            0.0s

bash-3.2# docker container ls
CONTAINER ID   IMAGE         COMMAND                  CREATED       STATUS                            PORTS     NAMES
67d4a1666a2e   opencbdc-tx   "/bin/bash"              7 hours ago   Up 7 hours                                  festive_nobel
5bb42b1fa7c3   opencbdc-tx   "/bin/bash"              7 hours ago   Up 7 hours                                  sleepy_bhabha
a2902ab5194d   opencbdc-tx   "/bin/bash"              8 hours ago   Up 8 hours                                  hardcore_dhawan
3c34f930aa33   opencbdc-tx   "/bin/bash"              8 hours ago   Up 8 hours                                  gracious_maxwell
1d0d2d2ee6f6   opencbdc-tx   "/bin/bash"              8 hours ago   Up 8 hours                                  focused_liskov
8041eb15d11a   opencbdc-tx   "./build/src/uhs/ato…"   8 hours ago   Restarting (255) 37 seconds ago             opencbdc-tx-sentinel0-1
7b5f1f582a07   opencbdc-tx   "./build/src/uhs/ato…"   8 hours ago   Restarting (255) 42 seconds ago             opencbdc-tx-shard0-1
d25b0049d7dd   opencbdc-tx   "/bin/bash"              9 hours ago   Up 9 hours                                  ecstatic_poincare
38ca8d7cc70e   opencbdc-tx   "/bin/bash"              9 hours ago   Up 9 hours                                  compassionate_goldwasser
0822ba9350e1   opencbdc-tx   "./build/src/uhs/ato…"   9 hours ago   Restarting (255) 48 seconds ago             opencbdc-tx-archiver0-1

[HalosGhost]

Hmm, something is definitely not right there. You can see that there doesn't seem to be a watchtower or atomizer running despite their having been started. (This is probably also why the running containers are failing out.)

I will take another look at this tomorrow morning, but the next step is to kill/remove all the containers (so that you can start completely fresh), and then run the docker-compose command without --detach so that you can see the live output. Then, if the atomizer fails out, we should see the error message it outputs.

[UrsaEli]

1 more observation:

bash-3.2# docker-compose --file docker-compose-atomizer.yml up -d
[+] Running 5/5
 ⠿ Container opencbdc-tx-watchtower0-1  Started                            1.2s
 ⠿ Container opencbdc-tx-atomizer0-1    Started                            5.4s
 ⠿ Container opencbdc-tx-archiver0-1    Started                            6.2s
 ⠿ Container opencbdc-tx-shard0-1       Started                            7.0s
 ⠿ Container opencbdc-tx-sentinel0-1    Started                            7.9s

bash-3.2# docker-compose --file docker-compose-atomizer.yml down
[+] Running 5/5
 ⠿ Container opencbdc-tx-sentinel0-1    Removed                            0.3s
 ⠿ Container opencbdc-tx-shard0-1       Removed                            0.9s
 ⠿ Container opencbdc-tx-archiver0-1    Removed                            0.3s
 ⠿ Container opencbdc-tx-atomizer0-1    Removed                            0.1s
 ⠿ Container opencbdc-tx-watchtower0-1  Removed                            0.1s
 ⠿ Network atomizer-network             Erro...                            0.0s
failed to remove network aa8d52372d34764e159d76e18e45101022e06a0cdf8643d0fd21637f86ac6b83: Error response from daemon: error while removing network: network atomizer-network id aa8d52372d34764e159d76e18e45101022e06a0cdf8643d0fd21637f86ac6b83 has active endpoints

[UrsaEli]

I tested with another machine which is on ubuntu. Though the 1st run gave a connection time out error (as below stated), it ran as stated on a 2nd attempt.

How is this failing for the Mac OS?

Creating network "atomizer-network" with the default driver
Creating opencbdc-tx_watchtower0_1 ... done
Creating opencbdc-tx_atomizer0_1   ...

ERROR: for opencbdc-tx_atomizer0_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for atomizer0  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

[HalosGhost]

We've decided to take this discussion offline, but will report back with any insights we find in case anyone else runs into this issue!