Building with newDo on w/frolvlad/alpine-gcc [PureGO example]

Not as much as issue as a HOWTO, but here are some EASY steps you can use to get a PKT suite built with newDo - in case you can't build "pure go" versions on your distro for some reason, need to avoid linking against glibc (as even static glibc binaries are not actually staticly linked) or you require the types of builds newDo can produce.

FYI - some reason, the Golang provided by Google (1.15.3) no longer works correctly on Alpine, nor via the Docker image 'frolvlad/alpine-gcc', but you can workaround this quite easily (you actually get better results with the pre-1.6 Golang toolchainas of this writing, but Go 1.15.3+ is OK too.)

The version I'm building here is essentially develop with PR#70 + 71 applied as of this writing (10-21-2020), using Alpine via the 'frolvlad/alpine-gcc' Docker image (has some of it's own issues, but none of which will affect the build):

$ docker run -it frolvlad/alpine-gcc /bin/sh and when inside:

1) apk update 2) apk upgrade 3) apk add git 4) apk add go (You need their old Go to compile your newer Go) 5) apk add bash (Sadly, the Go build system is written in Bash - no Bash required by any part of PKT suite or newDo.) 6) wget and extract the Go sources from golang.org or git clone https://go.googlesource.com/go 7) cd go/src 8) bash ./all.bash (and wait awhile) 9) export PATH=/root/go/bin:$PATH

Then from outside:

1) docker ps to get the ID, then:
2) docker cp your-pktd-source-tree.tar [IDHERE]:/root/pktd-src.tar

Back on the "inside", three steps:

1) cd /root ; tar xvf your-pktd-source-tree.tar && cd pktd 2) export PKT_BUILD_PUREGO=1 ; export PKT_BUILD_REPROD=1 ; export PKT_FAIL_DIRTY=0 ; export PKT_OWN_GOPATH=1 ; export PKT_OWN_GOPATH_DIR=/gopath ; export PKT_SKIP_TESTS=0 ; export PKT_FROM_SCRATCH=0 3) GOPATH=/gopath ./contrib/newDo/newDo

Here is the "reproducible" "Pure Go" build - (yes - newDo does work just fine with BusyBox ash.)

newDo: This is PKT newDo; version 0.6-20200830-R_PRODUCTION.
newDo: POSIX compliance claim: 200809

newDo: Locating required POSIX utilities:
newDo: Looking for cat: /bin/cat
newDo: Looking for chmod: /bin/chmod
newDo: Looking for comm: /usr/bin/comm
newDo: Looking for cut: /usr/bin/cut
newDo: Looking for grep: /bin/grep
newDo: Looking for mkdir: /bin/mkdir
newDo: Looking for mktemp: /bin/mktemp
newDo: Looking for rm: /bin/rm
newDo: Looking for rmdir: /bin/rmdir
newDo: Looking for sed: /bin/sed
newDo: Looking for sort: /usr/bin/sort
newDo: Looking for tr: /usr/bin/tr
newDo: Looking for true: true
newDo: Looking for uname: /bin/uname
newDo: Looking for xargs: /usr/bin/xargs

newDo: Locating required build tools:
newDo: Looking for git: /usr/bin/git
newDo: Looking for go: /root/go/bin/go

newDo: Locating optional utilities:
newDo: Looking for ccache: not found
newDo: Looking for checksec: not found
newDo: Looking for lsb_release: not found
newDo: Looking for sw_vers: not found
newDo: Looking for tput: not found
newDo: Looking for upx: not found

newDo: Warning: Previous config "/root/.pkt-do.conf" ignored, no execute attribute.

newDo: Operating system claims to be: linux
newDo: Machine hardware claims to be: amd64

newDo: Checking basic PKT build configuration: passed
newDo: Checking Go version: Go 1.15.x
newDo: Checking PKT build configuration compatibility with Go 15: passed
newDo: Checking operating system compatibility with Go 15: passed (linux)
newDo: Checking machine hardware compatibility with Go 15: passed (amd64)
newDo: Checking overall Go configuration: passed
newDo: Notice: PKT build has been successfully configured.

newDo: Warning: Removing existing configuration at "/root/.pkt-do.conf"
newDo: Notice: Current configuration written to "/root/.pkt-do.conf"
newDo: Notice: To enable it, execute: "chmod +x /root/.pkt-do.conf"

newDo: PKT_BUILD_REPROD: Build should be reproducible when compiling same sources.
newDo: PKT_BUILD_PUREGO: Making best efforts to avoid CGo, static linking if possible.

newDo: Notice: Building pktd
go: downloading github.com/jessevdk/go-flags v1.4.1-0.20200711081900-c17162fe8fd7
go: downloading github.com/gorilla/websocket v1.4.3-0.20200912193213-c3dd95aea977
go: downloading github.com/sethgrid/pester v1.1.1-0.20200617174401-d2ad9ec9a8b6
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/decred/go-socks v1.1.0
go: downloading github.com/decred/dcrd/lru v1.1.1-0.20201017002141-c07fa09c584d
go: downloading golang.org/x/crypto v0.0.0-20201016220609-9e8e0b390897
go: downloading github.com/dchest/blake2b v1.0.0
go: downloading github.com/golang/snappy v0.0.2
go: downloading github.com/aead/chacha20 v0.0.0-20180709150244-8b13a72661da
go: downloading github.com/kkdai/bstream v1.0.0
go: downloading github.com/aead/siphash v1.0.1
go: downloading golang.org/x/sys v0.0.0-20201018230417-eeed37f84f13

newDo: Notice: Building pktwallet
go: downloading google.golang.org/grpc v1.34.0-dev.0.20201019205402-ce5e36655676
go: downloading github.com/emirpasic/gods v1.12.1-0.20200630092735-7e2349589531
go: downloading go.etcd.io/bbolt v1.3.6-0.20200807205753-f6be82302843
go: downloading github.com/lightningnetwork/lnd/queue v1.0.5-0.20201016111222-d12f76fd6d48
go: downloading github.com/LK4D4/trylock v0.0.0-20191027065348-ff7e133a5c54
go: downloading github.com/lightningnetwork/lnd/ticker v1.0.0
go: downloading github.com/golang/protobuf v1.4.3
go: downloading google.golang.org/genproto v0.0.0-20201019141844-1ed22bb0c154
go: downloading golang.org/x/net v0.0.0-20201020065357-d65d470038a5
go: downloading google.golang.org/protobuf v1.25.0
go: downloading golang.org/x/text v0.3.4-0.20200826142016-a8b467125457

newDo: Notice: Building pktctl
newDo: Notice: Regenerating tests.

The following values are the computed linearly independent vectors needed to make use of the secp256k1 endomorphism:
a1: 3086d221a7d46bcde86c90e49284eb15
b1: -e4437ed6010e88286f547fa90abfe4c3
a2: 114ca50f7a8e2f3f657c1108d9d44cfd8
b2: 3086d221a7d46bcde86c90e49284eb15
Generating seed words

newDo: Notice: Running all tests.
ok      github.com/pkt-cash/pktd    0.113s
ok      github.com/pkt-cash/pktd/addrmgr    0.451s
ok      github.com/pkt-cash/pktd/blockchain 11.558s
ok      github.com/pkt-cash/pktd/blockchain/indexers    0.152s
ok      github.com/pkt-cash/pktd/blockchain/packetcrypt 0.082s
ok      github.com/pkt-cash/pktd/btcec  2.617s
ok      github.com/pkt-cash/pktd/btcjson    0.136s
ok      github.com/pkt-cash/pktd/btcutil    0.137s
ok      github.com/pkt-cash/pktd/btcutil/base58 0.030s
ok      github.com/pkt-cash/pktd/btcutil/bech32 0.090s
ok      github.com/pkt-cash/pktd/btcutil/bloom  0.100s
ok      github.com/pkt-cash/pktd/btcutil/coinset    0.041s
ok      github.com/pkt-cash/pktd/btcutil/gcs    4.227s
ok      github.com/pkt-cash/pktd/btcutil/gcs/builder    0.083s
ok      github.com/pkt-cash/pktd/btcutil/hdkeychain 0.153s
ok      github.com/pkt-cash/pktd/btcutil/txsort 0.032s
ok      github.com/pkt-cash/pktd/chaincfg   0.017s
ok      github.com/pkt-cash/pktd/chaincfg/chainhash 0.040s
ok      github.com/pkt-cash/pktd/connmgr    0.149s
ok      github.com/pkt-cash/pktd/database   0.237s
ok      github.com/pkt-cash/pktd/database/ffldb 3.688s
ok      github.com/pkt-cash/pktd/database/internal/treap    0.081s
ok      github.com/pkt-cash/pktd/mempool    0.454s
ok      github.com/pkt-cash/pktd/mining 0.061s
ok      github.com/pkt-cash/pktd/neutrino   8.535s
ok      github.com/pkt-cash/pktd/neutrino/banman    1.099s
ok      github.com/pkt-cash/pktd/neutrino/blockntfns    0.016s
ok      github.com/pkt-cash/pktd/neutrino/cache 0.007s
ok      github.com/pkt-cash/pktd/neutrino/cache/lru 0.008s
ok      github.com/pkt-cash/pktd/neutrino/chainsync 0.022s
ok      github.com/pkt-cash/pktd/neutrino/filterdb  0.064s
ok      github.com/pkt-cash/pktd/neutrino/headerfs  0.384s
ok      github.com/pkt-cash/pktd/neutrino/headerlist    0.019s
ok      github.com/pkt-cash/pktd/neutrino/pushtx    0.111s
ok      github.com/pkt-cash/pktd/peer   0.113s
ok      github.com/pkt-cash/pktd/pktwallet/chain    0.073s
ok      github.com/pkt-cash/pktd/pktwallet/internal/legacy/keystore 6.282s
ok      github.com/pkt-cash/pktd/pktwallet/internal/zero    0.028s
ok      github.com/pkt-cash/pktd/pktwallet/rpc/legacyrpc    0.094s
ok      github.com/pkt-cash/pktd/pktwallet/snacl    1.120s
ok      github.com/pkt-cash/pktd/pktwallet/waddrmgr 2.625s
ok      github.com/pkt-cash/pktd/pktwallet/wallet   7.296s
ok      github.com/pkt-cash/pktd/pktwallet/wallet/internal/txsizes  0.301s
ok      github.com/pkt-cash/pktd/pktwallet/wallet/seedwords 10.422s
ok      github.com/pkt-cash/pktd/pktwallet/wallet/txauthor  0.096s
ok      github.com/pkt-cash/pktd/pktwallet/walletdb 0.049s
ok      github.com/pkt-cash/pktd/pktwallet/walletdb/bdb 0.188s
ok      github.com/pkt-cash/pktd/pktwallet/walletdb/migration   0.005s
ok      github.com/pkt-cash/pktd/pktwallet/wtxmgr   0.260s
ok      github.com/pkt-cash/pktd/txscript   1.851s
ok      github.com/pkt-cash/pktd/txscript/scriptbuilder 0.027s
ok      github.com/pkt-cash/pktd/wire   1.053s
ok      github.com/pkt-cash/pktd/wire/protocol  0.010s

do: [Executing pktd --version:]: pktd version 1.1.1-0993fc3c

newDo: Build complete

newDo: All build actions completed.

newDo: Running cleanup tasks.
newDo: All cleanup tasks completed.

The proof's in the pudding:

# cd bin ; ls -l
-rwxr-xr-x    1 root     root       9487260 Oct 21 00:20 pktctl
-rwxr-xr-x    1 root     root      18296420 Oct 21 00:20 pktd
-rwxr-xr-x    1 root     root      22211755 Oct 21 00:20 pktwallet

# ./pktd 
1603239964 [INF] BTCD config.go:740: Creating a .cookie file
1603239964 [INF] BTCD btcd.go:61: Version 1.1.1-0993fc3c
1603239964 [INF] BTCD version.go:97: This is a pre-release version
1603239964 [INF] BTCD btcd.go:272: Loading block database from '/root/.pktd/data/pkt/blocks_ffldb'
1603239964 [INF] BTCD btcd.go:292: Block database loaded
1603239964 [INF] INDX server.go:2629: Committed filter index is enabled
1603239964 [INF] INDX manager.go:400: Catching up indexes from height -1 to 0
1603239964 [INF] INDX manager.go:452: Indexes caught up to height 0
1603239964 [INF] CHAN chain.go:1867: Chain state (height 0, hash 0bdc1712a46194e552cf417ab0439c2d4f456c35cf63a0a406964c6f93432d85, totaltx 1, work 4096)
1603239964 [WRN] RPCS server.go:2532: Can't listen on [::1]:64765: listen tcp6 [::1]:64765: bind: cannot assign requested address
1603239964 [INF] CMGR connmanager.go:499: Server listening on 0.0.0.0:64764
1603239964 [INF] CMGR connmanager.go:499: Server listening on [::]:64764
1603239964 [INF] RPCS rpcserver.go:4498: RPC server listening on 127.0.0.1:64765
1603239964 [INF] CMGR seed.go:57: 1 addresses found from DNS seed seed.anode.co
1603239964 [INF] CMGR seed.go:57: 26 addresses found from DNS seed seed.gridfinity.com
1603239964 [INF] CMGR seed.go:57: 22 addresses found from DNS seed seed.cjd.li
1603239975 [INF] SYNC manager.go:426: New valid peer 198.58.101.91:64764 (outbound) (/btcwire:0.5.0/pktd:1.1.1-942bda7d/)
[SNIP]
1603239976 [INF] SYNC manager.go:979: Verified downloaded block header against checkpoint at height 8192/hash 2b6b3d2951d365f26fd5b5362ab8dd5695ea6f5bd6c19ea9b16eb63a3de95ede
1603239976 [INF] SYNC manager.go:1004: Received 8192 block headers: Fetching blocks
1603239986 [INF] SYNC blocklogger.go:69: Processed 1091 blocks in the last 10s (1091 transactions, height 1091, 2019-08-20 17:35:50 +0000 UTC)

and yer off to the races.....

[FYI apk add tmux helps here]

now the pktwallet example:

# ./pktwallet --create                                   

/root/.pktwallet/pktwallet.conf does not exist, creating it from default
Enter the private passphrase for your new wallet: ********
Confirm passphrase: ********
Do you want to add an additional layer of encryption for public data? (n/no/y/yes) [no]: no                                                                       
Do you have an existing wallet seed you want to use? (n/no/y/yes) [no]: no
Encrypting your seed...                                                         
Your wallet generation seed is:
[redacted]

# ./pktwallet --userpc
1603240137 [INF] BTCW btcwallet.go:115: Version 1.1.1-0993fc3c
1603240137 [INF] BTCW version.go:97: This is a pre-release version
1603240137 [INF] BTCW rpcserver.go:57: Generating TLS certificates...
1603240138 [INF] BTCW rpcserver.go:100: Done generating TLS certificates
1603240138 [WRN] BTCW rpcserver.go:238: Can't listen on [::1]:64763: 1.1.1-0993fc3c listen tcp6 [::1]:64763: bind: cannot assign requested address

  btcutil/er/er.go:359 +0x188     btcutil/er.E()
  pktwallet/rpcserver.go:133 +0x85        main.startRPCServers.func1()
  pktwallet/rpcserver.go:236 +0x423       main.makeListeners()
  pktwallet/rpcserver.go:160 +0x268       main.startRPCServers()
  pktwallet/btcwallet.go:135 +0x2a7       main.walletMain()
  pktwallet/btcwallet.go:41 +0xa5         main.main()

1603240138 [INF] BTCW btcwallet.go:336: Attempting RPC client connection to localhost:64765
1603240138 [INF] RPCS server.go:198: Listening on 127.0.0.1:64763
1603240138 [INF] CHNS infrastructure.go:1388: Established connection to RPC server localhost:64765
1603240139 [INF] WLLT wallet.go:3335: Opened wallet
1603240139 [INF] WLLT chainntfns.go:100: Notification chain.ClientConnected

# ./pktctl --wallet getnewaddress
pkt1[redacted]

These binaries will run anywhere - no libc linkage required - note that on Linux glibc "static" binaries are not actually static anymore, because they will dynamically dlload/dlopen stuff, etc. Linux is terrible.

bin/pktctl:    ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
bin/pktd:      ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
bin/pktwallet: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped

newDo is a "hermetic" build system for building for redistribution, intended to on at least Linux (as far back as 1.2 kernels!), Mac OS X, MINGW 32/64, Win WSL, Free/Net/Open/DragonFlyBSD, Solaris, and even others, and is tested with with zsh, bash, ksh93, dash, ash, mksh, lksh, and OpenKSH [and is 100% sub-POSIX shell] - thousands and thousands of eye-cancer-causing lines of it, full of crazyness and insanity and stuff to ensure you get a working build - and tries it's best to not let you use incompatible flags.

newDo supports the following options, as of this writing:

# With BE_VERBOSE enabled, the script itself and any external
# commands it calls will be configured for extra verbosity. 
#BE_VERBOSE=0                           # OFF by default   

# When DEBUG_OUTPUT is enabled, the script itself as well    
# as some called commands will be configured to produce   
# extra output, which may be helpful for debugging issues.    
#DEBUG_OUTPUT=0                         # OFF by default  

# If PKT_FAIL_DIRTY is enabled, the build process will fail 
# if any changes have been made to the the sources or the
# build scripts, to ensure a stable and reliable build.      
#PKT_FAIL_DIRTY=1                       # ON by default     

# If using a build that utilizes Cgo, this option will add
# "-march=native" to the compiler flags, which will produce  
# binary output that is possibly faster, but not-portable.
#PKT_BUILD_NATIVE=0                     # OFF by default      

# When enabled, no tests are run when the build is done.    
#PKT_SKIP_TESTS=0                       # OFF by default   

# When enabled, use a separate GOPATH for the PKT build.     
#PKT_OWN_GOPATH=1                       # ON by default   

# If PKT_OWN_GOPATH is enabled, then the PKT_OWN_GOPATH_DIR
# variable will be used to provide a specific directory to  
# be used for the GOPATH. If undefined, the default "go-pkt"
# directory will be created in the current directory.    
#PKT_OWN_GOPATH_DIR="$HOME/.go-pkt"     # SET by default     

# When this option is enabled, special steps are taken to  
# ensure a *fully* fresh build is produced, without using 
# any previous build artifacts. At a minimum, this calls     
# use git clean and go clean -cache. Additionally, if     
# PKT_OWN_GOPATH is enabled, the entire contents of the       
# PKT_OWN_GOPATH_DIR will be cleared. If ccache is is found,
# the ccache cache directory is also completely cleared.    
# During the build, -a to rebuild all will be enabled also. 
#PKT_FROM_SCRATCH=0                     # OFF by default 

# When enabled, builds the (unsupported) wallettool utility.
#PKT_BUILD_WALLETTOOL=0         # OFF by default           

# When enbabld, builds the (unsupported) dropwtxmgr utility. 
#PKT_BUILD_DROPWTXMGR=0         # OFF by default          

# This option will make best efforts for a "pure Go" build, 
# avoiding any unnecessary use of Cgo functionality or Gccgo,
# and will force the selection of the pure Go DNS resolver. 
# This outputs a standalone static binary on most platforms.
#PKT_BUILD_PUREGO=1                     # ON by default      

# This option, in combination with the required toolchain, 
# such as that which is available on Fedora 26 or higher, 
# will build PIE (Position Independent Executable) binaries  
# which use the hardened compiler and linker flags developed
# and tested by Red Hat. These options are are the defaults   
# when no overriding options are specified using the Red Hat
# distributed releases of Golang. This option could be useful
# to users of other distributions, or for Red Hat users who 
# are utilizing the mainline Google Golang distribution.    
#PKT_BUILD_HARDRH=0                     # OFF by default     

# This option attempts to create a reproducible binary, where
# the same source will produce the the same binary each time,
# at the the expense of extended features, such as support   
# for GNU_HASH, Build-ID, and similar ELF functionality.    
#PKT_BUILD_REPROD=1             # ON by default               

# This option will make use of the "UPX" compression utiltity.
# if available, to reduce the size of the compiled binaries.
#PKT_UPX_COMPRESS=0                             # OFF by default

I have a newer alpha version that can do cross-compiling successfully as well as producing automated opmsg and/or gnupg signed releases. Again, this is most going to be ported to Go and hopefully would be able to quality for inclusion in a release.

I'll put a copy of it here in a PR, in case it might help someone build redist binaries or what not - but I recommend NOT using it unless have the specific NEED for it, such as produce a ABI/API agnostic distributable releases, etc.

pkt-cash / pktd

Building with newDo on w/frolvlad/alpine-gcc [PureGO example] #72