search

hyperledger / indy-test-automation

Automation tools for testing of Indy Project components.
https://wiki.hyperledger.org/display/indy
Apache License 2.0
6 stars 23 forks source link

Run tests with system-based Ubuntu 20.04 image #113

Closed udosson closed 2 years ago

udosson commented 2 years ago

It seems like the Nodes in the test network don't reach consensus according to their logs.

Steps to reproduce:

  1. Check out https://github.com/udosson/indy-test-automation/tree/test-systemd-ubuntu-2004
  2. Follow the readme and use the ubuntu 20.04 commands

When running the tests and thereby starting the docker containers it seems like the nodes connect, lose connection, and reconnect to the nodes. The tests get stuck in the following function https://github.com/udosson/indy-test-automation/blob/test-systemd-ubuntu-2004/system_node_only/utils.py#L522

Node1.log Node2.log Node3.log Node4.log Node5.log Node6.log Node7.log

Any ideas @WadeBarnes and @pSchlarb?

WadeBarnes commented 2 years ago

It looks like they can't stablish a connection with each other. Investigating.

WadeBarnes commented 2 years ago

The indy-node-control.service is scheduling restarts of indy-node for some reason.

WadeBarnes commented 2 years ago

The indy-node-control.service is crashing and restarting, which may be contributing to the indy-node restarts. I'm looking into why it's crashing and seeing if fixing that solves the indy-node restart issues.

WadeBarnes commented 2 years ago

Fixing the crash in indy-node-control.service worked for a shorter set of tests UBUNTU_VERSION="ubuntu-2004" ./run.sh system_node_only/indy-node-tests/test_ledger.py "-l -v --junit-xml=test_ledger-report.xml -k test_send_and_get_nym_positive". I've run the longer set of tests to confirm they work too; UBUNTU_VERSION="ubuntu-2004" ./run.sh system_node_only/indy-node-tests/test_ledger.py "-l -v --junit-xml=test_ledger-report.xml --log-cli-level 0"

WadeBarnes commented 2 years ago

When the indy-node-control.service starts up it attempts to hold indy and plenum related packages from this list, https://github.com/hyperledger/indy-node/blob/ubuntu-20.04-upgrade/indy_common/config.py#L100-L107. In the case of the 20.04 version of node, the python3-pygments and python3-intervaltree packages do not exist. When the indy-node-control.service tries to hold these packages it crashes and restarts. The restart of the indy-node-control.service seems to also trigger a restart of the node-service. I'm not sure exactly why or how, as I have not dug into the specifics, but on the surface it does not appear to be related to the indy-node-control.service code itself.

WadeBarnes commented 2 years ago

I tested this by installing the two packages into the node container at the indy-test-automation level, as the real fix requires a change to indy-node: image

WadeBarnes commented 2 years ago

Fix is here; https://github.com/hyperledger/indy-node/pull/1741

WadeBarnes commented 2 years ago

@udosson, I've updated the instructions with the fixed version on indy-node on your branch here; https://github.com/udosson/indy-test-automation/pull/1.