search

PharmaLedger-IMI / ctr-workspace

UC8 Clinical Trials Recruitment
MIT License
2 stars 0 forks source link

Real ETH integration tests with https://github.com/PharmaLedger-IMI/blockchain-node-workspace and quorum-docker #93

Open joaoluis-pdm opened 2 years ago

joaoluis-pdm commented 2 years ago

Perform real blockchain integration tests using the https://github.com/PharmaLedger-IMI/blockchain-node-workspace and https://github.com/PharmaLedger-IMI/quorum-docker tools.

From these tests, and actual deployment plan should be drafted.

joaoluis-pdm commented 2 years ago

The /borest:3000 services seem to conflict with the last https://github.com/PharmaLedger-IMI/quorum-docker 

blockchain-node-workspace/quorum-docker$ bash ./ethAdapter/deploy-ethAdapter.sh
...
... (takes a long time to build dockers...)
...
ERROR: for ethAdapter  Cannot start service ethAdapter: driver failed programming external connectivity on endpoint ethAdapter (c62b4372c9bbda96b7ffc38b50e544ece8dd2fe46e5f4171e2d2e3c5307dfc91): Error starting userland proxy: listen tcp4 0.0.0.0:3000: bind: address already in use

ERROR: for ethAdapter  Cannot start service ethAdapter: driver failed programming external connectivity on endpoint ethAdapter (c62b4372c9bbda96b7ffc38b50e544ece8dd2fe46e5f4171e2d2e3c5307dfc91): Error starting userland proxy: listen tcp4 0.0.0.0:3000: bind: address already in use
ERROR: Encountered errors while bringing up the project.
joaoluis-pdm commented 2 years ago

After blockchain-node-workspace v0.2.1 it should be ok to install a stand-alone blockchain-node-workspace on DEV and deploy a CTR working with it.

joaoluis-pdm commented 2 years ago

On ctr-dev-pl, docker-compose and nvm + node (same version as in the docker containers) installed as described in private VMWare.docx.

joaoluis-pdm commented 2 years ago

See the private VMWare.docx document on setting up ctr-dev-pl for blockchain-node-workspace on day 2022-02-08 and 2022-02-14.

TST installed on 2022-02-17

Deployed https://github.com/PharmaLedger-IMI/blockchain-node-workspace on the homedir.

git clone https://github.com/PharmaLedger-IMI/blockchain-node-workspace
cd blockchain-node-workspace
npm install
npm run config-blockchain -- --name=ctr
npm run deploy-blockchain

Space available before deploying blockchain:

pharmaledger@ctr-tst-pl:~$ df -k
Filesystem                        1K-blocks     Used Available Use% Mounted on
...
/dev/mapper/ubuntu--vg-ubuntu--lv  60759916 15952516  42007248  28% /
...

Space available after deploying blockchain:

pharmaledger@ctr-tst-pl:~/blockchain-node-workspace$ df -k /
Filesystem                        1K-blocks     Used Available Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  60759916 20087320  37872444  35% /

Crontab was edited for restart, but left commented out.

pharmaledger@ctr-tst-pl:~$ crontab -e
PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#01 07 * * *     ( cd $HOME/blockchain-node-workspace/quorum-docker/quorum && bash -x ../../bin/reCreateQuorum.sh >>./log 2>&1 )

The ethAdapter was not tested on TST as ctr-patient was still at v0.10.16

Blockchain was left stopped on TST to economize resources.

pharmaledger@ctr-tst-pl:~/blockchain-node-workspace$ npm run stop-blockchain

Will need to be restarted when deploying v0.11.4 or higher.

joaoluis-pdm commented 2 years ago

Released v0.11.3 to ctr-dev.

Initial internal feedback: image

Options for internal discussion:

joaoluis-pdm commented 2 years ago

Doubts about persistence of blockchain-node-worspace remains. See https://github.com/PharmaLedger-IMI/blockchain-node-workspace/issues/1#issuecomment-1033571735

Problems when upgrading from blockchain-node v0.2.1 to v0.2.2, smart contract was recreated (which implies a new chain), but creation of default "John McTester" account had a problem, and it cannot be deleted now.

joaoluis-pdm commented 2 years ago

Issue #60 seems to happen more frequently using a real blockchain: https://github.com/PharmaLedger-IMI/ctr-workspace/issues/60#issuecomment-1035017756

joaoluis-pdm commented 2 years ago

https://github.com/PharmaLedger-IMI/blockchain-node-workspace/tree/v0.2.3 seems to consume a lot of docker space.

Starting up on 2022-02-14 14h50m

pharmaledger@ctr-dev-pl:~$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   20G   37G  35% /

root@ctr-dev-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
109M    /var/lib/docker/containers
35M /var/lib/docker/image
124K    /var/lib/docker/network
18G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
405M    /var/lib/docker/volumes

Testing stop/start:

pharmaledger@ctr-dev-pl:~/blockchain-node-workspace$ npm run stop-blockchain
pharmaledger@ctr-dev-pl:~/blockchain-node-workspace$ npm run start-blockchain

Seems ok.

Testing a reboot.

root@ctr-dev-pl:~# reboot

Tuesday, 15 February 2022, 10:01:31 After 19h hours

root@ctr-dev-pl:~# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv   58G   26G   30G  47% /

root@ctr-dev-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
6.3G    /var/lib/docker/containers
35M /var/lib/docker/image
124K    /var/lib/docker/network
18G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
949M    /var/lib/docker/volumes

Seems that /var/lib/docker/containers grows 6GB each 19h.

joaoluis-pdm commented 2 years ago

Re-creating quorum docker images every day at 7AM to purge docker container space

pharmaledger@ctr-dev-pl:~$ crontab -e
# ensure docker-compose is in PATH
PATH=/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
01 07 * * *     ( cd $HOME/blockchain-node-workspace/quorum-docker/quorum && bash -x ../../bin/reCreateQuorum.sh >>./log 2>&1 )
joaoluis-pdm commented 2 years ago

After 3 weeks uptime, the memory space (physical and virtual ram) seem exhausted.

image

quorum (geth) was restarted. Not much RAM was recovered, blockchain-explorer was restarted. A bit of RAM was recovered. ... but soon RAM was exhausted again.

Something is consuming more resources.

Also geth is taking much more time to recover. Takes a lot more time doing this (log from node1-qbn):

...
TRACE[03-16|11:02:19.436] Block state missing, rewinding further   number=176986 hash="415703…8d48b4"
TRACE[03-16|11:02:19.436] Block state missing, rewinding further   number=176985 hash="c92b17…5f159d"
TRACE[03-16|11:02:19.436] Block state missing, rewinding further   number=176984 hash="ae92e9…edbd7d"
TRACE[03-16|11:02:19.436] Block state missing, rewinding further   number=176983 hash="ef3389…196f06"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176982 hash="47ce38…bb805f"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176981 hash="6be3a7…a94460"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176980 hash="d44123…4c3a1a"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176979 hash="5a4791…12abfc"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176978 hash="19784d…d29976"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176977 hash="841a92…6e321e"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176976 hash="f92c3c…8373b1"
TRACE[03-16|11:02:19.437] Block state missing, rewinding further   number=176975 hash="76c56b…5638b8"
...

Does this mean that geth will take even more time to recover the next time it is restarted ??

ctr-dev-pl had RAM raised to 16GB RAM and was rebooted. (The root disk is still kept on HDD).

A "top" screenshot taken before geth recovery is finished:

image

Note the high IO wait due to geth.

"iotop":

Total DISK READ:         2.39 M/s | Total DISK WRITE:        17.90 K/s
Current DISK READ:       2.39 M/s | Current DISK WRITE:      17.90 K/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                     
   3877 be/4 root      798.45 K/s    0.00 B/s  0.00 % 96.91 % geth --allow-insecure-u~in=* --graphql.vhosts=*
   9581 be/4 root      744.75 K/s    0.00 B/s  0.00 % 95.72 % geth --allow-insecure-u~in=* --graphql.vhosts=*
   3855 be/4 root      902.29 K/s    0.00 B/s  0.00 % 92.62 % geth --allow-insecure-u~in=* --graphql.vhosts=*
  16331 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/u4:1-events_unbound]
   4241 be/4 root        0.00 B/s   17.90 K/s  0.00 %  0.00 % mongod --logpath=/dev/n~-bind_ip 0.0.0.0 [ftdc]
      1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init maybe-ubiquity
      2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
...

We now have a bit more free RAM, but it is still unclear why is geth using a lot of IO, and what processes will use more RAM next.

joaoluis-pdm commented 2 years ago

There is a potential issue with the trust-loader, that, when stoping/starting the ctr-patient docker several times, the trust loader will fail with an error regarding the loader of a ctr-patient/apihub-root/patient-ssapp/loader/manifest.webmanifest file.

To workaround that, you need to re-create the docker. stop/start won't work.

pharmaledger@ctr-dev-pl:~$ UCNAME="ctr-patient"
pharmaledger@ctr-dev-pl:~$ docker stop $UCNAME
pharmaledger@ctr-dev-pl:~$ docker rm ctr-patient
pharmaledger@ctr-dev-pl:~$ docker run --detach --hostname $UCNAME --network="host" --publish 8080:8080 --mount source=external-volume,target=/ctr-workspace/ctr-patient/apihub-root/external-volume --name $UCNAME --restart always pharmaledger/$UCNAME /bin/bash -c 'cd ctr-workspace/ctr-patient && bash startup-eth-script.sh'

It should be analysed in more detail the next time it happens.

joaoluis-pdm commented 1 year ago

Out-of-disk space on ctr-dev-pl. Lets check where it is being used.

root@ctr-dev-pl:~# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  118G  117G     0 100% /

root@ctr-dev-pl:~# du -sh /var/lib/docker/*
92K /var/lib/docker/buildkit
72G /var/lib/docker/containers
36M /var/lib/docker/image
132K    /var/lib/docker/network
15G /var/lib/docker/overlay2
16K /var/lib/docker/plugins
4.0K    /var/lib/docker/runtimes
4.0K    /var/lib/docker/swarm
4.0K    /var/lib/docker/tmp
4.0K    /var/lib/docker/trust
25G /var/lib/docker/volumes

root@ctr-dev-pl:~# du -sh /var/lib/docker/volumes/*
0   /var/lib/docker/volumes/backingFsBlockDev
8.0K    /var/lib/docker/volumes/be927a522e23f48e1cfa5a1583779ff56270927f2977a79cec1200ac71550b40
7.2G    /var/lib/docker/volumes/d3329c281a736b765149a27d602b2202e817a877ca207db74fdb82781f8879be
69M /var/lib/docker/volumes/external-volume
32K /var/lib/docker/volumes/metadata.db
3.9G    /var/lib/docker/volumes/quorum_vol1
4.1G    /var/lib/docker/volumes/quorum_vol2
3.9G    /var/lib/docker/volumes/quorum_vol3
5.7G    /var/lib/docker/volumes/quorum_vol4

root@ctr-dev-pl:~# du -sh /var/lib/docker/containers/*
44M /var/lib/docker/containers/2c600bcb3534ee7b5612cd2c0f5a44b1942f39656858bb704b10008537ac2745
72G /var/lib/docker/containers/33d02e39c5452cea3c376125d603320a3fc7515e94f796cacf1f1222c6d5d79f
9.9M    /var/lib/docker/containers/580d5dbe5b4f3bbe881f95f109ed7dc46cb281dd49e2cac30e1937e80027bdd7
2.3M    /var/lib/docker/containers/5d2f793971ab24100e357e2706c600dc6ae9d56a2813d472da59b5291c4af8cd
42M /var/lib/docker/containers/733c9c2cb0e1ec6197ebabd3f0d0026fa2e1a7541a9f35b19d44b592b4209ca9
45M /var/lib/docker/containers/7ccdf584842049831c28da1db048378d1b4f29da052855a38dfa15cb45186b98
256K    /var/lib/docker/containers/83c679403ac4623beb30e2e413cceccc5b1958e5f4dba5b170594e13fbbf0711
244K    /var/lib/docker/containers/881d99c03c4e5fc58ba48e78b10bc8a8fa91d34aba32e5e7ef4ac2745d4fee68
36K /var/lib/docker/containers/92815ef51225f6b065f7bab20a306c8deb8cbf7bb31c501930de75f3037b241d
44M /var/lib/docker/containers/a3e23aacb6e6ccedd27c8d063d188a25ae2e257b032e62f1d1bdca0e56721c15
76K /var/lib/docker/containers/a84cd016a0852d0908978d6513df3cd667703a5cd29df84ac2bd5e63bbfb6ecf
224K    /var/lib/docker/containers/c7c04a287e72ee7b4c49e070ed0160278da5e48352caeb2314b7cbd2af9d0391
64K /var/lib/docker/containers/d67e27ba56e264b1485805ebc962a4b81ccfc5d3cf291ba579387a782127af28
5.0M    /var/lib/docker/containers/e2845210251c14fa6684dba9dbf9426c647c7b26a816d48c8842a272a30567cd

root@ctr-dev-pl:~# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          14        11        7.707GB   2.162GB (28%)
Containers      14        14        96.66MB   0B (0%)
Local Volumes   7         7         26.53GB   0B (0%)
Build Cache     0         0         0B        0B

root@ctr-dev-pl:~# docker ps --size
CONTAINER ID   IMAGE                                   COMMAND                  CREATED         STATUS                   PORTS                                                                                             NAMES                             SIZE
c7c04a287e72   nginx:latest                            "/docker-entrypoint.…"   3 months ago    Up 3 months              0.0.0.0:8700->80/tcp                                                                              blockchain-explorer_nginx_1       1.09kB (virtual 142MB)
881d99c03c4e   web3labs/epirus-free-web:latest         "/bin/sh -c ./entryp…"   3 months ago    Up 3 months              3000/tcp                                                                                          blockchain-explorer_web_1         7.87MB (virtual 386MB)
a84cd016a085   web3labs/epirus-free-api:latest         "/bin/sh -c 'exec ja…"   3 months ago    Up 3 months                                                                                                                blockchain-explorer_api_1         15.6MB (virtual 369MB)
33d02e39c545   web3labs/epirus-free-ingestion:latest   "/bin/sh -c 'exec ja…"   3 months ago    Up 3 months                                                                                                                blockchain-explorer_ingestion_1   15.6MB (virtual 366MB)
92815ef51225   mongo:latest                            "mongod --logpath=/d…"   3 months ago    Up 3 months              0.0.0.0:27017->27017/tcp                                                                          blockchain-explorer_mongodb_1     0B (virtual 700MB)
5d2f793971ab   pharmaledger/ctr-patient                "docker-entrypoint.s…"   3 months ago    Up 3 months                                                                                                                ctr-patient                       55.7MB (virtual 2.07GB)
e2845210251c   ethadapter:latest                       "node index.js"          3 months ago    Up 3 months              3000/tcp                                                                                          ethAdapter                        0B (virtual 1.43GB)
a3e23aacb6e6   quorumengineering/quorum:21.7.1         "/bin/sh -c 'UDS_WAI…"   3 months ago    Up 3 months (healthy)    30303/tcp, 30303/udp, 0.0.0.0:22003->8545/tcp, 0.0.0.0:23003->8546/tcp, 0.0.0.0:24003->8547/tcp   node4-qbn                         0B (virtual 85.3MB)
2c600bcb3534   quorumengineering/quorum:21.7.1         "/bin/sh -c 'UDS_WAI…"   3 months ago    Up 3 months (healthy)    30303/tcp, 30303/udp, 0.0.0.0:22002->8545/tcp, 0.0.0.0:23002->8546/tcp, 0.0.0.0:24002->8547/tcp   node3-qbn                         0B (virtual 85.3MB)
7ccdf5848420   quorumengineering/quorum:21.7.1         "/bin/sh -c 'UDS_WAI…"   3 months ago    Up 3 months (healthy)    30303/tcp, 30303/udp, 0.0.0.0:22001->8545/tcp, 0.0.0.0:23001->8546/tcp, 0.0.0.0:24001->8547/tcp   node2-qbn                         0B (virtual 85.3MB)
733c9c2cb0e1   quorumengineering/quorum:21.7.1         "/bin/sh -c 'UDS_WAI…"   3 months ago    Up 5 weeks (unhealthy)   30303/tcp, 30303/udp, 0.0.0.0:22000->8545/tcp, 0.0.0.0:23000->8546/tcp, 0.0.0.0:24000->8547/tcp   node1-qbn                         0B (virtual 85.3MB)
580d5dbe5b4f   pharmaledger/ctr-backoffice-backend     "docker-entrypoint.s…"   6 months ago    Up 3 months                                                                                                                ctr-backoffice-backend            1.86MB (virtual 1.36GB)
d67e27ba56e2   pharmaledger/ctr-backoffice-frontend    "httpd-foreground"       6 months ago    Up 3 months              0.0.0.0:8000->80/tcp                                                                              ctr-backoffice-frontend           2B (virtual 158MB)
83c679403ac4   pharmaledger/formbuilder-lhcforms       "docker-entrypoint.s…"   15 months ago   Up 3 months              0.0.0.0:9030->9030/tcp                                                                            formbuilder-lhcforms              11.5kB (virtual 1.19GB)

Seems that no one understands what the --size (Virtual size means), because it does not add up at all.

But the big fat container eating disk space seems to be 33d02e39c545 web3labs/epirus-free-ingestion with 72GB of local disk writes... (The all other docker volumes, including GoQuorum sum up ony 25GB of disk space).

Let's stop and remove containers.

root@ctr-dev-pl:~# docker kill $(docker ps -q)
...
Error response from daemon: Cannot kill container: 83c679403ac4: Container 83c679403ac4623beb30e2e413cceccc5b1958e5f4dba5b170594e13fbbf0711 is not running

root@ctr-dev-pl:~# docker rm $(docker ps -a -q)
c7c04a287e72
881d99c03c4e
a84cd016a085
33d02e39c545
92815ef51225
5d2f793971ab
e2845210251c
a3e23aacb6e6
2c600bcb3534
7ccdf5848420
733c9c2cb0e1
580d5dbe5b4f
d67e27ba56e2
83c679403ac4

root@ctr-dev-pl:~# df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  118G   45G   68G  40% /

pharmaledger@ctr-dev-pl:~/blockchain-node-workspace$ npm run stop-blockchain

> blockchain-node-workspace@0.2.5 stop-blockchain /home/pharmaledger/blockchain-node-workspace
> cd quorum-docker/ethAdapter && docker-compose stop && cd ../blockchain-explorer && docker-compose -f docker-compose.yml stop && cd ../quorum && docker-compose stop

pharmaledger@ctr-dev-pl:~/blockchain-node-workspace$ npm run down-blockchain

> blockchain-node-workspace@0.2.5 down-blockchain /home/pharmaledger/blockchain-node-workspace
> cd quorum-docker/ethAdapter && docker-compose down && cd ../blockchain-explorer && docker-compose -f docker-compose.yml down && cd ../quorum && docker-compose down

Network qbn-net is external, skipping
Removing network epirus-net
WARNING: Network epirus-net not found.
Removing network qbn-net

pharmaledger@ctr-dev-pl:~/blockchain-node-workspace$ docker volume prune
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
quorum_vol3
be927a522e23f48e1cfa5a1583779ff56270927f2977a79cec1200ac71550b40
external-volume
quorum_vol1
quorum_vol2
quorum_vol4
d3329c281a736b765149a27d602b2202e817a877ca207db74fdb82781f8879be

Total reclaimed space: 26.53GB
pharmaledger@ctr-dev-pl:~/blockchain-node-workspace$ df -h /
Filesystem                         Size  Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv  118G   21G   92G  18% /

Restarted without blockchain.