Spin up a node with docker having issue

[Keth88]

Describe the bug Im, not a dev, trying to setup a orchid provider node with docker, but getting error. Can you help on this.

To Reproduce Steps to reproduce the behavior:

1. Launch fresh AWS EC2 Ubuntu
2. Install docker, test and make sure docker instances are running.
3. Clone orchid repo, and run ./env/docker.sh
4. See error

Screenshots

root@ip-172-31-45-92:/home/ubuntu/orchid# ls
AUTHORS    SECURITY.md     app-linux    cli-shared          env          lot-ethereum  min-krypton    min-zlib               pac-ethereum    setup.mk         srv-shared    tst-finance   tst-ticket   vpn-windows      
COPYING    app-android     app-macos    code-of-conduct.md  eth-sender   makefile      min-pcre2      mkdocs.yml             pac-serverless  sim-payprotocol  tag32         tst-metadata  vpn-android  web-ethereum     
LICENSE    app-flutter.mk  app-shared   default.mk          github.sh    min-cairo     min-v8         modules                pkg-lnx         snap             tok-ethereum  tst-network   vpn-apple
NOTES.txt  app-flutter.sh  app-windows  dir-ethereum        gui-orchid   min-glib      min-webrtc     openzeppelin-solidity  remake          srv-docker       tst-ethereum  tst-openvpn3  vpn-linux
README.md  app-ios         chains.mk    docs                local.mk.in  min-icu4c     min-wireshark  p2p                    remix           srv-gateway      tst-explore   tst-share     vpn-shared
root@ip-172-31-45-92:/home/ubuntu/orchid#
root@ip-172-31-45-92:/home/ubuntu/orchid#
root@ip-172-31-45-92:/home/ubuntu/orchid#
root@ip-172-31-45-92:/home/ubuntu/orchid# ./env/docker.sh
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/mnt/env/setup-dkr.sh": stat /mnt/env/setup-dkr.sh: no such file or directory: unknown.
ERRO[0000] error waiting for container: context canceled
root@ip-172-31-45-92:/home/ubuntu/orchid#

[saurik]

The instructions say you can use env/docker.sh instead of make, but the instructions in the preceeding paragraph for running make specify that, before "and run make", "you can then just go into any subfolder you want (such as app-{android,ios}, or {cli,srv}-shared)", which you did not do. As you mentioned wanting to build the server, you should first change into the srv-shared folder.

[Keth88]

Thanks for checking it. I changed to srv-shared folder, then run env/docker.sh, installed most of the tools, but erroring below.

Should I install ndk separately?

..... 
 inflating: /usr/local/lib/android/sdk/tools/support/typos-hu.txt
Warning: Unable to find package ndk-bundle
Warning: File /root/.android/repositories.cfg could not be loaded.
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  sudo
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 428 kB of archives.
After this operation, 1765 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 sudo amd64 1.8.21p2-3ubuntu1.4 [428 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 428 kB in 2s (271 kB/s)
Selecting previously unselected package sudo.
(Reading database ... 34539 files and directories currently installed.)
Preparing to unpack .../sudo_1.8.21p2-3ubuntu1.4_amd64.deb ...
Unpacking sudo (1.8.21p2-3ubuntu1.4) ...
Setting up sudo (1.8.21p2-3ubuntu1.4) ...
make: Entering directory '/mnt/srv-shared'
find: './p2p/rtc/webrtc/net': No such file or directory
bash: line 0: cd: ./cv8/v8/src: No such file or directory
find: './mediasoup/worker/deps/libwebrtc/libwebrtc': No such file or directory
make: *** No rule to make target 'out-dkr/x86_64/orchidd', needed by 'all'.  Stop.
make: Leaving directory '/mnt/srv-shared'
ubuntu@ip-172-31-45-92:~/orchid/srv-shared$ 

[saurik]

This looks like you didn't run git submodule update --init --recursive (per the instructions) to check out all of the source code.

[Keth88]

Oky, that is fixed. env/docker.sh image building taking so long (3+hours now). clang is taking 100% cpu of 8vCPU (c1.xlarge). is it expected?

top - 16:36:26 up  1:43,  2 users,  load average: 1.00, 1.00, 1.00
Tasks: 165 total,   2 running, 163 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.3 us,  0.2 sy,  0.0 ni, 87.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   6940.7 total,    115.4 free,   1369.7 used,   5455.6 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5269.9 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                  
  39849 ubuntu    20   0  932116 877416  42020 R 100.0  12.3   1:23.48 clang-tidy                                                                                                               
  39832 ubuntu    20   0   10916   3908   3296 R   0.7   0.1   0:00.48 top                                                                                                                      
    133 root      20   0       0      0      0 S   0.3   0.0   0:00.38 kswapd0
   1471 ubuntu    20   0 1716944  50528  33132 S   0.3   0.7   0:02.15 docker
      1 root      20   0  167264  12572   8164 S   0.0   0.2   0:05.05 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp
      5 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 netns
      6 root      20   0       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0-events
      7 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 kworker/0:0H-events_highpri
      8 root      20   0       0      0      0 I   0.0   0.0   0:00.29 kworker/0:1-events
     10 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 mm_percpu_wq
     11 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_rude_
     12 root      20   0       0      0      0 S   0.0   0.0   0:00.00 rcu_tasks_trace
     13 root      20   0       0      0      0 S   0.0   0.0   0:00.29 ksoftirqd/0
     14 root      20   0       0      0      0 I   0.0   0.0   0:02.80 rcu_sched
     15 root      rt   0       0      0      0 S   0.0   0.0   0:00.04 migration/0
     16 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/0
     17 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/0
     18 root      20   0       0      0      0 S   0.0   0.0   0:00.00 cpuhp/1
     19 root     -51   0       0      0      0 S   0.0   0.0   0:00.00 idle_inject/1
     20 root      rt   0       0      0      0 S   0.0   0.0   0:00.48 migration/1
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.33 ksoftirqd/1

[saurik]

You can probably make the build at least somewhat faster by passing debug=notidy to the env/docker.sh script (which will skip the clang-tidy parts... though those frankly don't take that much of the total time), but, yes: in order to do a fully-deterministic build of Orchid with all of its dependencies included--which includes more of Chrome than you might expect (as it uses a copy of both WebRTC and V8)--certainly takes hours, and of course is going to use all of your CPU as compiling is a CPU-limited process.

Oh, that said, looking at what you pasted, it is actually only using 12% of your CPU, not 100% of your CPU. If you pass -j8 to env/docker.sh it might compile a lot faster, as it is currently doing a single-threaded build. (Essentially, env/docker.sh is able to stand in for make, and "just running make" is sufficient, but most people who use make do configure make to better take advantage of the resources on their computer. I guess I could at least try to automatically tune make for your machine if you aren't a developer, but frankly the goal of this is just to get it built and ensure that the build actually works every time rather than try to figure out how a non-developer can build the product from source code they don't understand anyway faster ;P. If you want to have the binary right now you can directly download a copy of the build output for any version you want using GitHub from the CI build artifacts, and in practice you are going to need to trust largely the same people by the same amount.)

[Keth88]

Thanks for the reply. After a long break, I cloned repo today. Tried to build docker image with a C4Xlarge AWS instance, build process looks faster than before. Still getting some error, i could not offer complete logs here. any idea how we can fix this. Im a non-dev enthusiasitic about the project. Its much appricated if you could create a branch so i can clone with any chages required on the code, thanks.

ubuntu@ip-172-31-25-234:~$ cd orchid/srv-shared/
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ git submodule update --init --recursive
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ git submodule update --init --recursive
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ 
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ ./env/docker.sh
Unable to find image 'ubuntu:bionic' locally
bionic: Pulling from library/ubuntu
a404e5416296: Pull complete
Digest: sha256:ca70a834041dd1bf16cc38dfcd24f0888ec4fa431e09f3344f354cf8d1724499
Status: Downloaded newer image for ubuntu:bionic
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [3094 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [83.3 kB]
Get:6 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [1294 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1568 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [22.8 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [3519 kB]
Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [29.8 kB]
Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [1334 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2342 kB]
Get:17 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [20.5 kB]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [64.0 kB]
Fetched 26.7 MB in 7s (3627 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  autotools-dev binfmt-support binutils binutils-common
  binutils-x86-64-linux-gnu build-essential ca-certificates clang-6.0
  clang-tidy-6.0 cpp cpp-7 dbus debugedit dh-python dirmngr dpkg-dev fakeroot
  file fontconfig fontconfig-config fonts-dejavu-core fonts-droid-fallback
  fonts-noto-mono g++ g++-7 g++-7-multilib gcc gcc-7 gcc-7-base gcc-7-multilib
----<snipped>----
webrtc/modules/remote_bitrate_estimator/ -x c++ -
1.      <eof> parser at end of file
make: *** [out-dkr/x86_64/./p2p/rtc/webrtc/modules/remote_bitrate_estimator/remote_estimator_proxy.cc.o] Error 1
env/output.mk:63: recipe for target 'out-dkr/x86_64/./p2p/rtc/webrtc/modules/remote_bitrate_estimator/remote_estimator_proxy.cc.o' failed
make: Leaving directory '/mnt/srv-shared'
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ 
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ 
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ docker image ls -a
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
ubuntu       bionic    71eaf13299f4   5 weeks ago   63.1MB
ubuntu@ip-172-31-25-234:~/orchid/srv-shared$ 

[saurik]

I feel like you snipped something extremely important. FWIW, the beginning is pretty irrelevant: I only need the stuff at the end.

[saurik]

Can you maybe also include the output from a git status?

[Keth88]

I gave another fresh shot on c5.4xlarge server. It progressed bit further than last time.

1. after git submodule update --init --recursive update , no changes in git status
2. ./env/docker.sh is errored, seeing git status, p2p/c-ares modified

Attached full logs all the way from cloning repo: building-docker-image.log

[saurik]

I see no evidence in this log that "./env/docker.sh is errored": in fact, the build seems to have succeeded.

[LD] out-dkr/x86_64/orchidd
b8fd089e9b26e4f378f395f028b9b36acd241310e31ea3a7de7a5e96e8375420 *out-dkr/x86_64/orchidd
-rwxr-xr-x 1 1000 root 60365232 Dec  3 01:56 out-dkr/x86_64/orchidd
make: Leaving directory '/mnt/srv-shared'

[Keth88]

Oh, nice (I didn't realize build was succeed). I guess, passed docker build phase, posted in DH too. I can now run the container successfully.

ubuntu@ip-172-31-23-171:~$ docker run -it keth88/orchid:latest bash
Unable to find image 'keth88/orchid:latest' locally
latest: Pulling from keth88/orchid
a404e5416296: Already exists
Digest: sha256:/snip/
Status: Downloaded newer image for keth88/orchid:latest
root@5187457c3839:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
root@5187457c3839:/#

For this testing, How do I configure node, stake Oxt, and operate.

• Can you give some instructions or steps.
• What parameters need to configure,
• When staking OXT, can I avoid bring private key in to this live node.

[saurik]

Can you tell me what you mean by "the container"? Running env/docker.sh doesn't claim to build a container for Orchid... it just creates a transient build environment--in a container, sure--which it then uses to run the normal Orchid build process, which should result in a binary, as you'd get if you built any other project. The binary was left--per the last few lines of that output--at out-dkr/x86_64/orchidd. If you want a docker container version of Orchid (which I highly dis-recommend and cannot guarantee support for in the future) you can go into srv-docker (where there is a Dockerfile) and run "docker build" (as you'd expect to build any normal docker container).

[Keth88]

sorry for the confusion. Im not indented to run it as container, looking for safe and easy way to run the node. Oky, I have binary out-dkr/x86_64/orchidd now, how to move forward. can I run this on any node that dont have docker installed?

[Keth88]

Can you give some guide, how to configure node from here. thank in advance.