search

ExchangeUnion / xud

Exchange Union Daemon 🔁 ⚡️
https://exchangeunion.com
GNU Affero General Public License v3.0
115 stars 49 forks source link

Find a stable setup to run simulation tests locally #1995

Closed sangaman closed 3 years ago

sangaman commented 3 years ago

I get this every time I try to run simulation tests locally:

=== RUN   TestIntegration
2020/11/17 07:04:10 xud: creating network
2020/11/17 07:04:13 ltcd: launching node...
2020/11/17 07:04:14 ltcd: 200 blocks generated
2020/11/17 07:04:14 lnd-ltc: launching network...
2020/11/17 07:04:27 btcd: launching node...
2020/11/17 07:04:27 btcd: 200 blocks generated
2020/11/17 07:04:27 lnd-btc: launching network...
2020/11/17 07:04:38 connext: launching network...
2020/11/17 07:04:41 xud: launching network...
2020/11/17 07:05:01 cannot start xud network: timeout waiting for TLS cert file to be created after 20 seconds: /app/temp/xuddatadir-Bob/tls.cert
exit status 1
FAIL    github.com/ExchangeUnion/xud-simulation 51.112s

docker-run.sh: test failed, re-running... (1/3)

Starting simulation_gomod_1   ... done
Starting simulation_btcd_1    ... done
Starting simulation_connext_1 ... done
Starting simulation_lnd_1     ... done
Starting simulation_xud_1     ... done
=== RUN   TestIntegration
2020/11/17 07:05:31 xud: creating network
2020/11/17 07:05:34 ltcd: launching node...
2020/11/17 07:05:35 ltcd: 200 blocks generated
2020/11/17 07:05:35 lnd-ltc: launching network...
2020/11/17 07:05:49 btcd: launching node...
2020/11/17 07:05:49 btcd: 200 blocks generated
2020/11/17 07:05:49 lnd-btc: launching network...
2020/11/17 07:06:01 connext: launching network...
2020/11/17 07:06:04 xud: launching network...
2020/11/17 07:06:24 cannot start xud network: timeout waiting for TLS cert file to be created after 20 seconds: /app/temp/xuddatadir-Dave/tls.cert

Attempting to run on a resource constrained (but clean/fresh) VM results in similar timeout errors.

sangaman commented 3 years ago

Oddly I get this even after setting the tls cert timeout periods to 300 seconds, implying xud isn't starting at all, but I can't figure out why. I think at it would be good to get some error logging here to detect if xud isn't starting if possible and print any error messages.

kilrau commented 3 years ago

Building images with npm fails instantly for me, probably something wrong with my node/npm which I just installed. bash docker-build.sh failed at the step Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory.

kilrau commented 3 years ago

Rebooted and tried again, exact same outcome as above.

michael1011 commented 3 years ago

Building and running them worked fine on my machine:

LePremierHomme commented 3 years ago

Building images with npm fails instantly for me, probably something wrong with my node/npm which I just installed. bash docker-build.sh failed at the step Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory.

Looks like the problem is sh: 1: tsc: not found. Try to run npm install -g typescript.

LePremierHomme commented 3 years ago

Oddly I get this even after setting the tls cert timeout periods to 300 seconds, implying xud isn't starting at all, but I can't figure out why. I think at it would be good to get some error logging here to detect if xud isn't starting if possible and print any error messages.

Indeed the problem is probably that xud fails to start. If no log file exist, we can print xud's stdout. I haven't done that on default because it can be annoying. In case you're working on a custom branch which refactored xud's cmd args, then that might be the problem. Please let me know if so, and if not, i'll add support for printing xud's stdout.

kilrau commented 3 years ago

I finally got simulation tests to build successfully with the exact same setup that worked for @michael1011 :

    Ubuntu 20.10
    Node v14.15.1
    NPM v6.14.8
    Docker v19.03.13

What I did:

xud@1.2.0 test:sim:run /home/kilrau/xud (cd test/simulation && ./docker-run.sh)

~/xud/test/simulation/temp/indra ~/xud/test/simulation bash ops/start-indra.sh Using docker images indra_name:latest Can't find image redis:5-alpine locally, attempting to pull redis:5-alpine 5-alpine: Pulling from library/redis 188c0c94c7c5: Pull complete fb6015f7c791: Pull complete f8890a096979: Pull complete 4d275bc77206: Pull complete b6ead6773e5a: Pull complete 8375f47daa52: Pull complete Digest: sha256:e0256846c84d07095778890a8c481eceaca9a46a9d292a499555f40d248dfb29 Status: Downloaded newer image for redis:5-alpine docker.io/library/redis:5-alpine Proxy configured Node configured Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. make: *** [Makefile:48: start] Error 1 ~/xud/test/simulation ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again. ~/xud/test/simulation/temp/indra ~/xud/test/simulation bash ops/stop.sh indra Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. No stack, service or running container names match: indra bash ops/stop.sh daicard Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. No stack, service or running container names match: daicard bash ops/stop.sh testnet Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. No stack, service or running container names match: testnet docker container prune -f Total reclaimed space: 0B docker network rm indra indra_cf_tester indra_node_tester indra_test_store 2> /dev/null || true docker secret rm indra_database_dev 2> /dev/null || true docker volume rm indra_database_dev 2> /dev/null || true docker volume rm docker volume ls -q -f name=indra_database_test_* 2> /dev/null || true rm -rf .chaindata/* rm -rf modules/bot/.bot-store ~/xud/test/simulation npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! xud@1.2.0 test:sim:run: (cd test/simulation && ./docker-run.sh) npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the xud@1.2.0 test:sim:run script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /home/kilrau/.npm/_logs/2020-11-19T11_32_50_983Z-debug.log

kilrau commented 3 years ago

2020-11-19T11_32_50_983Z-debug.log:

 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   '/home/kilrau/.nvm/versions/node/v14.15.1/bin/node',
1 verbose cli   '/home/kilrau/.nvm/versions/node/v14.15.1/bin/npm',
1 verbose cli   'run',
1 verbose cli   'test:sim:run'
1 verbose cli ]
2 info using npm@6.14.8
3 info using node@v14.15.1
4 verbose run-script [ 'pretest:sim:run', 'test:sim:run', 'posttest:sim:run' ]
5 info lifecycle xud@1.2.0~pretest:sim:run: xud@1.2.0
6 info lifecycle xud@1.2.0~test:sim:run: xud@1.2.0
7 verbose lifecycle xud@1.2.0~test:sim:run: unsafe-perm in lifecycle true
8 verbose lifecycle xud@1.2.0~test:sim:run: PATH: /home/kilrau/.nvm/versions/node/v14.15.1/lib/node_modules/np>
9 verbose lifecycle xud@1.2.0~test:sim:run: CWD: /home/kilrau/xud
10 silly lifecycle xud@1.2.0~test:sim:run: Args: [ '-c', '(cd test/simulation && ./docker-run.sh)' ]
11 silly lifecycle xud@1.2.0~test:sim:run: Returned: code: 1  signal: null
12 info lifecycle xud@1.2.0~test:sim:run: Failed to exec test:sim:run script
13 verbose stack Error: xud@1.2.0 test:sim:run: `(cd test/simulation && ./docker-run.sh)`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/home/kilrau/.nvm/versions/node/v14.15.1/lib/node_modules/np>
13 verbose stack     at EventEmitter.emit (events.js:315:20)
13 verbose stack     at ChildProcess.<anonymous> (/home/kilrau/.nvm/versions/node/v14.15.1/lib/node_modules/np>
13 verbose stack     at ChildProcess.emit (events.js:315:20)
13 verbose stack     at maybeClose (internal/child_process.js:1048:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
14 verbose pkgid xud@1.2.0
15 verbose cwd /home/kilrau/xud/test/simulation
16 verbose Linux 5.8.0-29-generic
17 verbose argv "/home/kilrau/.nvm/versions/node/v14.15.1/bin/node" "/home/kilrau/.nvm/versions/node/v14.15.1/>
18 verbose node v14.15.1
19 verbose npm  v6.14.8
20 error code ELIFECYCLE
21 error errno 1
22 error xud@1.2.0 test:sim:run: `(cd test/simulation && ./docker-run.sh)`
22 error Exit status 1
23 error Failed at the xud@1.2.0 test:sim:run script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]
kilrau commented 3 years ago

Should we also bump simulation tests to current versions? E.g. to go 1.14 and lnd to 0.11.1 , guess there is more to be updated.

https://github.com/ExchangeUnion/xud/blob/master/test/simulation/go.mod

kilrau commented 3 years ago

Tried a couple of more runs with docker system prune -a before, no luck: always failing with the Network indra declared as external, but could not be found as above.

kilrau commented 3 years ago

Can you set up a VM with this "default developer setup":

    Ubuntu 20.10
    Node v14.15.1 (via nvm)
    NPM v6.14.8 (via nvm)
    Docker v19.03.13 (installed via `apt install docker.io`)

And harden simulation tests for this? You should be able to reproduce the issues above without problems. Our goal is to have the setup above with these specific version work reliably @LePremierHomme

erkarl commented 3 years ago

Can you try:

docker system prune -a
npm run test:sim:clean
npm run test:sim:clean:indra
npm run test:sim
kilrau commented 3 years ago

Did that, same:

> xud@1.2.0 test:sim:run /home/kilrau/xud
> (cd test/simulation && ./docker-run.sh)

~/xud/test/simulation/temp/indra ~/xud/test/simulation
bash ops/start-indra.sh
Using docker images indra_name:latest 
Can't find image redis:5-alpine locally, attempting to pull redis:5-alpine
5-alpine: Pulling from library/redis
188c0c94c7c5: Pull complete 
fb6015f7c791: Pull complete 
f8890a096979: Pull complete 
4d275bc77206: Pull complete 
b6ead6773e5a: Pull complete 
8375f47daa52: Pull complete 
Digest: sha256:e0256846c84d07095778890a8c481eceaca9a46a9d292a499555f40d248dfb29
Status: Downloaded newer image for redis:5-alpine
docker.io/library/redis:5-alpine
Proxy configured
Node configured
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
make: *** [Makefile:48: start] Error 1
~/xud/test/simulation
ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.
~/xud/test/simulation/temp/indra ~/xud/test/simulation
bash ops/stop.sh indra
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: indra
bash ops/stop.sh daicard
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: daicard
bash ops/stop.sh testnet
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: testnet
docker container prune -f
Total reclaimed space: 0B
docker network rm indra indra_cf_tester indra_node_tester indra_test_store 2> /dev/null || true
docker secret rm indra_database_dev 2> /dev/null || true
docker volume rm indra_database_dev  2> /dev/null || true
docker volume rm `docker volume ls -q -f name=indra_database_test_*` 2> /dev/null || true
rm -rf .chaindata/*
rm -rf modules/bot/.bot-store
~/xud/test/simulation
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! xud@1.2.0 test:sim:run: `(cd test/simulation && ./docker-run.sh)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the xud@1.2.0 test:sim:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-19T12_48_25_297Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! xud@1.2.0 test:sim: `(npm run test:sim:build && npm run test:sim:run)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the xud@1.2.0 test:sim script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-19T12_48_25_318Z-debug.log
michael1011 commented 3 years ago

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

I also had these issues. A reboot somehow solved them

kilrau commented 3 years ago

Let me try that, also good hint for @LePremierHomme

kilrau commented 3 years ago

Reboot didn't solve it for me.

LePremierHomme commented 3 years ago

What I did: npm run test:sim:build still first failed instantly for me just like before (and yes I installed typescript via npm install -g typescript beforehand).

It still just looks as if you don’t have access to tsc executable. I verified it and it should work fine if you installed it via npm install -g typescript (and it’s accessible on your $PATH), or you run npm install (if you didn’t, you’ll have more issues, but that will be the first one).

  • bash docker-build.sh fails at the same step as above Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory

Why do you run this directly? It should always be run only as part of npm run test:sim:build. It failed because the compilation failed, and xud dir files were not copied to the expected location. So this is not a problem, and we just need to solve the previous one.

Regarding your running failure - something is wrong with indra’s docker setup (“This node is not a swarm manager…”). I’m trying to reproduce it on gcloud VM now.

michael1011 commented 3 years ago

Regarding your running failure - something is wrong with indra’s docker setup (“This node is not a swarm manager…”)

@erkarl do you think debugging the existing Indra setup is worth it since Connext v2 is just around the corner?

erkarl commented 3 years ago

@erkarl do you think debugging the existing Indra setup is worth it since Connext v2 is just around the corner?

No, disable it.

LePremierHomme commented 3 years ago

Ubuntu 20.10 has an issue with docker installation (“Package 'docker-ce' has no installation candidate”), so I used Ubuntu 20.04 LTS instead. Looks like it had no issue the swarm manager. This is what I got:

> xud@1.2.1 test:sim:run /home/premier_gmail_com/xud
> (cd test/simulation && ./docker-run.sh)

/home/premier_gmail_com/xud/test/simulation/temp/indra /home/premier_gmail_com/xud/test/simulation
bash ops/start-indra.sh
Swarm initialized: current node (vr1m05tumic9rzks9mzy5375t) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-62m6b062m40s8cmxqsa1cyp33tg6aioncmkss3akvnjw6peqko-4b1q6jatui1jsgw99fjtyj7jd 10.128.0.19:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Requirements:

Setup script (some sudo permissions are required due to indra’s docker setup):

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
export NVM_DIR="$HOME/.nvm"    
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

nvm install 14.15.1

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

sudo apt-get install build-essential python npm jq

https://github.com/ExchangeUnion/xud.git
cd xud
npm i
sudo npm run test:sim:build
sudo npm run test:sim:run

TODO (@LePremierHomme):

kilrau commented 3 years ago

Ubuntu 20.10 has an issue with docker installation (“Package 'docker-ce' has no installation candidate”)

Docker install scripts are always behind. Please install docker on ubuntu 20.10 via ubuntu repo: sudo apt install docker.io which should get you Docker version 19.03.13, build 4484c46.

kilrau commented 3 years ago

That looks like A LOT of additional dependencies that are needed. Are we sure we need all of these?

kilrau@K-Yoga:~$ bash test.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11329  100 11329    0     0  80921      0 --:--:-- --:--:-- --:--:-- 80921
=> nvm is already installed in /home/kilrau/.nvm, trying to update using git
=> error: pathspec 'v0.33.0' did not match any file(s) known to git
=> Compressing and cleaning up git repository
Enumerating objects: 317, done.
Counting objects: 100% (317/317), done.
Delta compression using up to 8 threads
Compressing objects: 100% (292/292), done.
Writing objects: 100% (317/317), done.
Total 317 (delta 49), reused 268 (delta 0), pack-reused 0

=> nvm source string already in /home/kilrau/.bashrc
=> bash_completion source string already in /home/kilrau/.bashrc
=> You currently have modules installed globally with `npm`. These will no
=> longer be linked to the active version of Node when you install a new node
=> with `nvm`; and they may (depending on how you construct your `$PATH`)
=> override the binaries of modules installed with `nvm`:

/home/kilrau/.nvm/versions/node/v14.15.1/lib
└── typescript@4.0.5
=> If you wish to uninstall them at a later point (or re-install them under your
=> `nvm` Nodes), you can remove them from the system Node as follows:

     $ nvm use system
     $ npm uninstall -g a_module

=> Close and reopen your terminal to start using nvm or run the following to use it now:

export NVM_DIR="/home/kilrau/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
v14.15.1 is already installed.
Now using node v14.15.1 (npm v6.14.8)
# Executing docker install script, commit: 26ff363bcf3b3f5a00498ac43694bf1c7d9ce16c
Warning: the "docker" command appears to already exist on this system.

If you already have Docker installed, this script can cause trouble, which is
why we're displaying this warning and provide the opportunity to cancel the
installation.

If you installed the current Docker package using this script and are using it
again to update Docker, you can safely ignore this message.

You may press Ctrl+C now to abort this script.
+ sleep 20
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
+ sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu groovy stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ [ -n  ]
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
E: Package 'docker-ce' has no installation candidate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   651  100   651    0     0   5761      0 --:--:-- --:--:-- --:--:--  5761
100 11.6M  100 11.6M    0     0  7778k      0  0:00:01  0:00:01 --:--:-- 12.2M
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'python-is-python2' instead of 'python'
build-essential is already the newest version (12.8ubuntu3).
build-essential set to manually installed.
The following additional packages will be installed:
  gyp libjq1 libjs-inherits libjs-is-typedarray libjs-psl libjs-typedarray-to-buffer libnode-dev libnode72 libonig5 libpython2-stdlib libpython2.7-minimal
  libpython2.7-stdlib libssl-dev libuv1-dev node-abbrev node-ajv node-ansi node-ansi-align node-ansi-regex node-ansi-styles node-ansistyles node-aproba node-archy
  node-are-we-there-yet node-asap node-asn1 node-assert-plus node-asynckit node-aws-sign2 node-aws4 node-balanced-match node-bcrypt-pbkdf node-bl node-bluebird node-boxen
  node-brace-expansion node-builtin-modules node-builtins node-cacache node-call-limit node-camelcase node-caseless node-chalk node-chownr node-ci-info node-cli-boxes
  node-cliui node-clone node-co node-color-convert node-color-name node-colors node-columnify node-combined-stream node-concat-map node-concat-stream node-config-chain
  node-configstore node-console-control-strings node-copy-concurrently node-core-util-is node-cross-spawn node-crypto-random-string node-cyclist node-dashdash
  node-debbundle-es-to-primitive node-debug node-decamelize node-decompress-response node-deep-extend node-defaults node-define-properties node-delayed-stream
  node-delegates node-detect-indent node-detect-newline node-dot-prop node-duplexer3 node-duplexify node-ecc-jsbn node-editor node-encoding node-end-of-stream
  node-err-code node-errno node-es6-promise node-escape-string-regexp node-execa node-extend node-extsprintf node-fast-deep-equal node-find-up node-flush-write-stream
  node-forever-agent node-form-data node-from2 node-fs-vacuum node-fs-write-stream-atomic node-fs.realpath node-function-bind node-gauge node-genfun node-get-caller-file
  node-get-stream node-getpass node-glob node-got node-graceful-fs node-gyp node-har-schema node-har-validator node-has-flag node-has-symbol-support-x
  node-has-to-string-tag-x node-has-unicode node-hosted-git-info node-http-signature node-iconv-lite node-iferr node-import-lazy node-imurmurhash node-inflight
  node-inherits node-ini node-invert-kv node-ip node-ip-regex node-is-npm node-is-obj node-is-object node-is-path-inside node-is-plain-obj node-is-retry-allowed
  node-is-stream node-is-typedarray node-isarray node-isexe node-isstream node-isurl node-jsbn node-json-parse-better-errors node-json-schema node-json-schema-traverse
  node-json-stable-stringify node-json-stringify-safe node-jsonify node-jsonparse node-jsonstream node-jsprim node-latest-version node-lazy-property node-lcid node-libnpx
  node-locate-path node-lockfile node-lodash node-lodash-packages node-lowercase-keys node-lru-cache node-make-dir node-mem node-mime node-mime-types node-mimic-fn
  node-mimic-response node-minimatch node-minimist node-mississippi node-mkdirp node-move-concurrently node-ms node-mute-stream node-nopt node-normalize-package-data
  node-npm-bundled node-npm-package-arg node-npm-run-path node-npmlog node-number-is-nan node-oauth-sign node-object-assign node-once node-opener node-os-locale
  node-os-tmpdir node-osenv node-p-cancelable node-p-finally node-p-is-promise node-p-limit node-p-locate node-p-timeout node-package-json node-parallel-transform
  node-path-exists node-path-is-absolute node-path-is-inside node-performance-now node-pify node-prepend-http node-process-nextick-args node-promise-inflight
  node-promise-retry node-promzard node-proto-list node-prr node-pseudomap node-psl node-pump node-pumpify node-punycode node-qs node-qw node-rc node-read
  node-read-package-json node-readable-stream node-registry-auth-token node-registry-url node-request node-require-directory node-require-main-filename node-resolve
  node-resolve-from node-retry node-rimraf node-run-queue node-safe-buffer node-semver node-semver-diff node-set-blocking node-sha node-shebang-command node-shebang-regex
  node-signal-exit node-slash node-slide node-sorted-object node-spdx-correct node-spdx-exceptions node-spdx-expression-parse node-spdx-license-ids node-sshpk node-ssri
  node-stream-each node-stream-iterate node-stream-shift node-strict-uri-encode node-string-decoder node-string-width node-strip-ansi node-strip-eof
  node-strip-json-comments node-supports-color node-tar node-term-size node-text-table node-through node-through2 node-timed-out node-tough-cookie node-tunnel-agent
  node-tweetnacl node-typedarray node-typedarray-to-buffer node-uid-number node-unique-filename node-unique-string node-unpipe node-uri-js node-url-parse-lax
  node-url-to-options node-util-deprecate node-uuid node-validate-npm-package-license node-validate-npm-package-name node-verror node-wcwidth.js node-which
  node-which-module node-wide-align node-widest-line node-wrap-ansi node-wrappy node-write-file-atomic node-xdg-basedir node-xtend node-y18n node-yallist node-yargs
  node-yargs-parser nodejs nodejs-doc python2 python2-minimal python2.7 python2.7-minimal
Suggested packages:
  libssl-doc python2-doc python-tk python2.7-doc
The following packages will be REMOVED:
  python-is-python3
The following NEW packages will be installed:
  gyp jq libjq1 libjs-inherits libjs-is-typedarray libjs-psl libjs-typedarray-to-buffer libnode-dev libnode72 libonig5 libpython2-stdlib libpython2.7-minimal
  libpython2.7-stdlib libssl-dev libuv1-dev node-abbrev node-ajv node-ansi node-ansi-align node-ansi-regex node-ansi-styles node-ansistyles node-aproba node-archy
  node-are-we-there-yet node-asap node-asn1 node-assert-plus node-asynckit node-aws-sign2 node-aws4 node-balanced-match node-bcrypt-pbkdf node-bl node-bluebird node-boxen
  node-brace-expansion node-builtin-modules node-builtins node-cacache node-call-limit node-camelcase node-caseless node-chalk node-chownr node-ci-info node-cli-boxes
  node-cliui node-clone node-co node-color-convert node-color-name node-colors node-columnify node-combined-stream node-concat-map node-concat-stream node-config-chain
  node-configstore node-console-control-strings node-copy-concurrently node-core-util-is node-cross-spawn node-crypto-random-string node-cyclist node-dashdash
  node-debbundle-es-to-primitive node-debug node-decamelize node-decompress-response node-deep-extend node-defaults node-define-properties node-delayed-stream
  node-delegates node-detect-indent node-detect-newline node-dot-prop node-duplexer3 node-duplexify node-ecc-jsbn node-editor node-encoding node-end-of-stream
  node-err-code node-errno node-es6-promise node-escape-string-regexp node-execa node-extend node-extsprintf node-fast-deep-equal node-find-up node-flush-write-stream
  node-forever-agent node-form-data node-from2 node-fs-vacuum node-fs-write-stream-atomic node-fs.realpath node-function-bind node-gauge node-genfun node-get-caller-file
  node-get-stream node-getpass node-glob node-got node-graceful-fs node-gyp node-har-schema node-har-validator node-has-flag node-has-symbol-support-x
  node-has-to-string-tag-x node-has-unicode node-hosted-git-info node-http-signature node-iconv-lite node-iferr node-import-lazy node-imurmurhash node-inflight
  node-inherits node-ini node-invert-kv node-ip node-ip-regex node-is-npm node-is-obj node-is-object node-is-path-inside node-is-plain-obj node-is-retry-allowed
  node-is-stream node-is-typedarray node-isarray node-isexe node-isstream node-isurl node-jsbn node-json-parse-better-errors node-json-schema node-json-schema-traverse
  node-json-stable-stringify node-json-stringify-safe node-jsonify node-jsonparse node-jsonstream node-jsprim node-latest-version node-lazy-property node-lcid node-libnpx
  node-locate-path node-lockfile node-lodash node-lodash-packages node-lowercase-keys node-lru-cache node-make-dir node-mem node-mime node-mime-types node-mimic-fn
  node-mimic-response node-minimatch node-minimist node-mississippi node-mkdirp node-move-concurrently node-ms node-mute-stream node-nopt node-normalize-package-data
  node-npm-bundled node-npm-package-arg node-npm-run-path node-npmlog node-number-is-nan node-oauth-sign node-object-assign node-once node-opener node-os-locale
  node-os-tmpdir node-osenv node-p-cancelable node-p-finally node-p-is-promise node-p-limit node-p-locate node-p-timeout node-package-json node-parallel-transform
  node-path-exists node-path-is-absolute node-path-is-inside node-performance-now node-pify node-prepend-http node-process-nextick-args node-promise-inflight
  node-promise-retry node-promzard node-proto-list node-prr node-pseudomap node-psl node-pump node-pumpify node-punycode node-qs node-qw node-rc node-read
  node-read-package-json node-readable-stream node-registry-auth-token node-registry-url node-request node-require-directory node-require-main-filename node-resolve
  node-resolve-from node-retry node-rimraf node-run-queue node-safe-buffer node-semver node-semver-diff node-set-blocking node-sha node-shebang-command node-shebang-regex
  node-signal-exit node-slash node-slide node-sorted-object node-spdx-correct node-spdx-exceptions node-spdx-expression-parse node-spdx-license-ids node-sshpk node-ssri
  node-stream-each node-stream-iterate node-stream-shift node-strict-uri-encode node-string-decoder node-string-width node-strip-ansi node-strip-eof
  node-strip-json-comments node-supports-color node-tar node-term-size node-text-table node-through node-through2 node-timed-out node-tough-cookie node-tunnel-agent
  node-tweetnacl node-typedarray node-typedarray-to-buffer node-uid-number node-unique-filename node-unique-string node-unpipe node-uri-js node-url-parse-lax
  node-url-to-options node-util-deprecate node-uuid node-validate-npm-package-license node-validate-npm-package-name node-verror node-wcwidth.js node-which
  node-which-module node-wide-align node-widest-line node-wrap-ansi node-wrappy node-write-file-atomic node-xdg-basedir node-xtend node-y18n node-yallist node-yargs
  node-yargs-parser nodejs nodejs-doc npm python-is-python2 python2 python2-minimal python2.7 python2.7-minimal
0 upgraded, 300 newly installed, 1 to remove and 0 not upgraded.
Need to get 20.7 MB of archives.
After this operation, 102 MB of additional disk space will be used.
Do you want to continue? [Y/n]
LePremierHomme commented 3 years ago

Docker install scripts are always behind. Please install docker on ubuntu 20.10 via ubuntu repo: sudo apt install docker.io which should get you Docker version 19.03.13, build 4484c46.

I thought I tried that first and it didn't work, will try it again.

That looks like A LOT of additional dependencies that are needed. Are we sure we need all of these?

These dependencies are installed due to sudo apt-get install npm which is used because nvm doesn't put npm to be accessible via sudo (needed for indra docker. setup). I tried to solve it differently without success, but i'll look into it again.

kilrau commented 3 years ago

Todov2:

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using docker network create indra and try again. /home/kilrau/xud/test/simulation/temp/indra /home/kilrau/xud/test/simulation bash ops/stop.sh indra Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. No stack, service or running container names match: indra bash ops/stop.sh daicard Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. No stack, service or running container names match: daicard bash ops/stop.sh testnet Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again. No stack, service or running container names match: testnet docker container prune -f Total reclaimed space: 0B docker network rm indra indra_cf_tester indra_node_tester indra_test_store 2> /dev/null || true docker secret rm indra_database_dev 2> /dev/null || true docker volume rm indra_database_dev 2> /dev/null || true docker volume rm docker volume ls -q -f name=indra_database_test_* 2> /dev/null || true rm -rf .chaindata/* rm -rf modules/bot/.bot-store /home/kilrau/xud/test/simulation npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! xud@1.2.0 test:sim:run: (cd test/simulation && ./docker-run.sh) npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the xud@1.2.0 test:sim:run script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in: npm ERR! /root/.npm/_logs/2020-11-20T12_43_15_758Z-debug.log

kilrau commented 3 years ago

And I have my first successful run. What I did:

  1. docker swarm init
  2. npm run test:sim:clean
  3. docker system prune
  4. make sure the /test/simulation/temp folder is not there
  5. npm run test:sim:build

And it went through! Full log: https://paste.ubuntu.com/p/HBZysWs4HV/

kilrau commented 3 years ago

Second run which I started immediately after without cleaning anything succeeded too.

kilrau commented 3 years ago

Third run which I started after reboot (without cleaning or any further actions) succeeded too.

kilrau commented 3 years ago

Let's finalize the todo's @LePremierHomme and then let @sangaman do an acceptance test - he's working on xud mainly and needs to run simulation tests locally.

erkarl commented 3 years ago

I suggest to disable connext v1/indra simulation tests if they are causing issues. We'll be switching to V2 this week.

kilrau commented 3 years ago

Well they are 100% reliable for me now after https://github.com/ExchangeUnion/xud/issues/1995#issuecomment-731612871, I opened https://github.com/ExchangeUnion/xud/pull/2001 to close this issue for now adding hints how to prep your environment to make tests pass reliably. If I manage, everyone should ;)

We'll anyways take another look at simulation tests after you did the switch to vector @erkarl

kilrau commented 3 years ago

I cleaned my environment with steps as per steps of my PR. Now the script fails to create the docker indra network again and the only solution seems to be to create it manually:

kilrau@K-Yoga:~/xud$ npm run test:sim:run

> xud@1.2.1 test:sim:run /home/kilrau/xud
> (cd test/simulation && ./docker-run.sh)

~/xud/test/simulation/temp/indra ~/xud/test/simulation
make: *** No rule to make target 'start'.  Stop.
~/xud/test/simulation
ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.
~/xud/test/simulation/temp/indra ~/xud/test/simulation
make: *** No rule to make target 'reset'.  Stop.
~/xud/test/simulation
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! xud@1.2.1 test:sim:run: `(cd test/simulation && ./docker-run.sh)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the xud@1.2.1 test:sim:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-24T09_12_06_185Z-debug.log
kilrau@K-Yoga:~/xud$ docker network create indra
707122d065aa464ea8e753cb701f943a65a8453e7b9bd35213facd587f88e9d9
sangaman commented 3 years ago

Following all the steps in the README in #2001 still gives me the error in the OP here:

2020/11/27 19:18:52 cannot start xud network: timeout waiting for TLS cert file to be created after 20 seconds: /app/temp/xuddatadir-Dave/tls.cert
exit status 1

Should we reopen?

kilrau commented 3 years ago

Well it doesn't fail for anyone else with that, thus isn't easy to debug for others. Rather open a separate issue and I guess you'll have to look into it yourself :/

Only thing that comes to mind are resources/Internet: You made sure Internet was stable and you have 10GB+ RAM free when you ran the tests?

LePremierHomme commented 3 years ago

@sangaman are you on master or a feature branch?