Reproducible build

[Pesa]

We should make the build process reproducible.

The first problem I've noticed is that app/version/make-version.sh embeds a timestamp in the generated version.go. We should get rid of the timestamp. There may be other sources of non-reproducibility in the build system.

[yoursunny]

Reproducible build is now available for make cmds target, under the following conditions:

• same version of Ubuntu and apt packages
• same version/patches of DPDK and SPDK with same configuration
• same version of Go dependencies: this can be achieved by running make goget at the same time, or copying $GOPATH/src folder
• same commit of ndn-dpdk codebase, with no "dirty" files
• same compiler and release mode choices when invoking make cmds

[yoursunny]

A recent test on commit bdb376136f2645f78144dd17aa76ac48b30756e8 indicates that the builds are still unreproducible, even on the same machine.

I used the following sequence of commands:

1. npm install
2. make
3. find build -type f | xargs openssl sha256 | sort >1.txt
4. make clean
5. make
6. find build -type f | xargs openssl sha256 | sort >2.txt
7. diff 1.txt 2.txt

If the builds were reproducible, the last command should have empty output. Instead, it reports that build/bin/ndnfw-dpdk has a different digest.

To diagnose this issue, I decompiled both programs with objdump -D build/bin/ndnfw-dpdk: https://pastebin.com/Ux0GPqZD The assembly has difference in the following sections:

• .note.go.buildid and .note.gnu.build-id: some instructions are different; the symbol per_lcore_gPccDebugString (a global variable) appears, but a large offset like 0x400321 suggests that it may be unrelated
• ndn-dpdk/strategy/strategy_elf.fastrouteO (and all other strategy ELF objects): one instruction movabs $0xed654aafd,%rcx has a different number in it

buildid is stored in a Go package or binary, and it's supposed to "accurately record a content hash of the file". Therefore, I believe the difference in buildid sections is caused by the difference in strategy sections.

Strategy ELF objects, as produced by clang-8, are the same. Therefore, go-bindata is likely the troublemaker here. I'll see what adjustment can be made to resolve this issue.

[yoursunny]

Fixed in a0172a047d2ecf9dc7e9f6528efc814e36e031b8.

It turns out that go-bindata is including metadata for the binary assets, which contains file modification time. Since the strategy ELF objects are freshly compiled, their modification times will differ. Setting go-bindata -nometadata flag solves this problem.

Due to recent changes in build procedure, find build -type f | xargs openssl sha256 | sort would capture the digest of meson and ninja logs. Ignoring the difference on these log files, the binary files compiled in two different virtual machines are identical.