search

harmony-one / experiment-deploy

Repo for conducting benchmark experiments
20 stars 15 forks source link

When building OSTN - want to provide ability for Validators to build and run #139

Closed john-harmony closed 4 years ago

john-harmony commented 4 years ago

Currently when building OSTN - There is no ability for validators to build with the correct binary. Want the ability for them to do this

Background 1) Documentation on process being developed can be found here 2) @LeoHChen originally suggested using harmony.one/node2.sh 3) As part of the build the binary should be pushed to AWS bucket - @AndyBoWu is now using this jenkins job to do this 4) Pangea Ops developed a workaround using this script 5) @fxfactorial shared this way of invoking it

Edgar | HarmonyYesterday at 11:09 AM
@here for anyone interested in running a node, seb from p-ops made it super easy to run a node 

Here's how to run the latest build:

bash <(curl -sSL https://raw.githubusercontent.com/SebastianJ/harmony-tools/master/install/install.sh) --node

If you don't want to fetch the latest node.sh from harmony-one/harmony (master) then just skip '--node' and node.sh won't get downloaded.

Make sure to use -D when starting your node since otherwise the custom binaries might get overwritten!

Example:

./hmy keys generate-bls-key

ln -s *.key bls.key

./node.sh -k bls.key -N staking -z -D -S
AndyBoWu commented 4 years ago

apparently, node2.sh retrieve the latest required binaries and dependency files from this path: https://s3.console.aws.amazon.com/s3/buckets/pub.harmony.one/release/linux-x86_64/testnet/?region=us-west-1&tab=overview, while jenkins job uploaded the files to differant path: s3://pub.harmony.one/release/linux-x86_64/testnet/static/harmony BECAUSE we checked the static binary option.

AndyBoWu commented 4 years ago

new job output:

Copying s3://pub.harmony.one/release/linux-x86_64/v5559-master-20191208.0-182-g50b6df28/harmony to s3://pub.harmony.one/release/linux-x86_64/testnet/harmony Copy complete [PipelUload

john-harmony commented 4 years ago

Latest error

[ec2-user@ip-172-31-45-56 ~]$ LD_LIBRARY_PATH=. ./harmony -version
./harmony: error while loading shared libraries: libbls384_256.so: cannot open shared object file: No such file or directory
AndyBoWu commented 4 years ago

The missing file does exist in the s3 bucket.

Screen Shot 2020-02-12 at 2 59 46 PM

LeoHChen commented 4 years ago

@john-harmony I don't see any errors in the script, nor the s3 bucket. Can you start with a fresh directory?

LeoHChen commented 4 years ago

Try 

./node2.sh -k bls.key -N staking -z -S

to download the binary and libraries.

john-harmony commented 4 years ago

Downloads and runs but version doesn't look right

[ec2-user@ip-172-31-40-139 ~]$ curl -LO https://harmony.one/node2.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 20582  100 20582    0     0  43605      0 --:--:-- --:--:-- --:--:--  154k
[ec2-user@ip-172-31-40-139 ~]$ mv node2.sh node.sh
[ec2-user@ip-172-31-40-139 ~]$ chmod a+x node.sh
[ec2-user@ip-172-31-40-139 ~]$ ./node2.sh -k 843a3bccb40da9920b9ee4c763644032a4284c48a819fba4ee7cf809ae9eb1df8c7d62e4852b3a79b136f50a2809bf95.key -N staking -z -S
-bash: ./node2.sh: No such file or directory
[ec2-user@ip-172-31-40-139 ~]$ ./node.sh -k 843a3bccb40da9920b9ee4c763644032a4284c48a819fba4ee7cf809ae9eb1df8c7d62e4852b3a79b136f50a2809bf95.key -N staking -z -S
node.sh: using manually specified BLS key file: 843a3bccb40da9920b9ee4c763644032a4284c48a819fba4ee7cf809ae9eb1df8c7d62e4852b3a79b136f50a2809bf95.key
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   799  100   799    0     0   1795      0 --:--:-- --:--:-- --:--:--  1795
diff: md5sum.txt: No such file or directory
node.sh: downloaded harmony
node.sh: downloaded wallet
node.sh: downloaded bootnode
node.sh: downloaded libmcl.so
node.sh: downloaded libbls384_256.so
node.sh: public IP address autodetected: 13.58.96.11
Enter passphrase for the BLS key file 843a3bccb40da9920b9ee4c763644032a4284c48a819fba4ee7cf809ae9eb1df8c7d62e4852b3a79b136f50a2809bf95.key:
node.sh: ############### Running Harmony Process ###############
Staking mode; node key 843a3bccb40da9920b9ee4c763644032a4284c48a819fba4ee7cf809ae9eb1df8c7d62e4852b3a79b136f50a2809bf95 -> shard 2
^[^[[O^[[I^[[O^[[I^CGot interrupt signal. Dumping state to DB...
^Z
[1]+  Stopped                 ./node.sh -k 843a3bccb40da9920b9ee4c763644032a4284c48a819fba4ee7cf809ae9eb1df8c7d62e4852b3a79b136f50a2809bf95.key -N staking -z -S
[ec2-user@ip-172-31-40-139 ~]$  LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5559-master-20191208.0-182-g50b6df28 (jenkins@ 2020-02-12T22:47:45+0000)
john-harmony commented 4 years ago

I spoke a little early it looks like the version that it downloaded is different from the version of the nodes when checked in http://watchdog.hmny.io/report-staking We get

Version-v5571-master-20191208.0-194-ga653dabe
Node Count:24

But when checking the node I see

[ec2-user@ip-172-31-40-139 ~]$  LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5559-master-20191208.0-182-g50b6df28 (jenkins@ 2020-02-12T22:47:45+0000)
john-harmony commented 4 years ago

Just downloaded again and still getting an old version from Feb 12th

[ec2-user@ip-172-31-46-53 ~]$ LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5559-master-20191208.0-182-g50b6df28 (jenkins@ 2020-02-12T22:47:45+0000)

Looks to align from this branch https://github.com/harmony-one/harmony/commit/50b6df289e9b5d07d2ac5e70b8a3674ec3f55382

LeoHChen commented 4 years ago

http://jenkins.harmony.one/job/harmony-release/

Just rebuild it

LeoHChen commented 4 years ago

try

node2.sh -d -N testnet

to download the latest binary.

john-harmony commented 4 years ago

Version is not being updated when running that jenkins job

Downloaded and got this version (3 days old) 

[ec2-user@ip-172-31-39-108 ~]$ LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5588-master-20191208.0-211-g3be4594e (jenkins@ 2020-02-22T06:06:54+0000)

Then went in to Jenkins and rebuilt see build 264

Downloaded again and got the same result

[ec2-user@ip-172-31-39-108 ~]$ LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5588-master-20191208.0-211-g3be4594e (jenkins@ 2020-02-22T06:06:54+0000)
LeoHChen commented 4 years ago

@john-harmony , you need to click branch release.

john-harmony commented 4 years ago

Just did a build using the

Screen Shot 2020-02-25 at 12 03 23 PM

Downloaded using 

[ec2-user@ip-172-31-39-108 staging]$ curl -LO https://harmony.one/node2.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 20582  100 20582    0     0  44262      0 --:--:-- --:--:-- --:--:-- 44262
[ec2-user@ip-172-31-39-108 staging]$ mv node2.sh node.sh
[ec2-user@ip-172-31-39-108 staging]$ chmod a+x node.sh
./node.sh -S -N staking -k 166beb7d1b5887898bcc2ba05bad812358238b59403fcfcd7aca6d13628762b20100bdfecdf2e9b7ffeda5103b713484.key -z

Still get an old version

[ec2-user@ip-172-31-39-108 staging]$ LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5588-master-20191208.0-211-g3be4594e (jenkins@ 2020-02-22T06:06:54+0000)

Also tried as above (note I have renamed node2.sh to node.sh on my AWS instance

[ec2-user@ip-172-31-39-108 staging]$ ./node.sh -d -N testnet
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   799  100   799    0     0   2113      0 --:--:-- --:--:-- --:--:-- 10797
node.sh: binaries did not change in staging
[ec2-user@ip-172-31-39-108 staging]$ cd staging
[ec2-user@ip-172-31-39-108 staging]$ LD_LIBRARY_PATH=. ./harmony -version
-bash: ./harmony: No such file or directory
john-harmony commented 4 years ago

Retried with STATIC Binary unchecked 268 and then downloaded again and it worked, i.e. got the latest version

./node.sh -S -N staking -k 2593fdbd97091d0498368626e2b79d7f3bd92d92dc63ddeb137ede571c6a3459990016b25f7f2fad200197db5682aa16.key -z
[ec2-user@ip-172-31-39-108 staging]$ LD_LIBRARY_PATH=. ./harmony -version
Harmony (C) 2020. harmony, version v5595-master-20191208.0-218-g380449bb (jenkins@ 2020-02-25T20:15:22+0000)
AndyBoWu commented 4 years ago

I tried on my side, it is working as expected.

We can close this ticket if you want.

john-harmony commented 4 years ago

Happy to close this and we'll ensure we run the jenkins job when we build OSTN