Document how to investigate reproducibility issues

LATEST UPDATE IS THAT THE FOLLOWING MODULES ARE NOT REPRODUCIBLE (02/24/2022):

busybox
cryptsetup
cryptsetup2
fbwhiptail
hidapi
hotp-verification
json-c
libgcrypt
lvm2
qrencode
tpmtotp
util-linux

Expend from https://github.com/osresearch/heads/issues/927#issuecomment-744075679

Heads is not reproducible from same debian-10 host base (CircleCI docker vs QubesOS debian-10 base). ~/QubesIncoming/x230-master/* is the output of CircleCI build's artifacts:

user@x230-test:~/heads$ diff build/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1039-g9339394.rom ~/QubesIncoming/x230-master/heads-x230-hotp-maximized-v0.2.0-1039-g9339394.rom 
Binary files build/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1039-g9339394.rom and /home/user/QubesIncoming/x230-master/heads-x230-hotp-maximized-v0.2.0-1039-g9339394.rom differ
user@x230-test:~/heads$ diff build/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1039-g9339394-top.rom ~/QubesIncoming/x230-master/heads-x230-hotp-maximized-v0.2.0-1039-g9339394-top.rom 
user@x230-test:~/heads$ diff build/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1039-g9339394-bottom.rom ~/QubesIncoming/x230-master/heads-x230-hotp-maximized-v0.2.0-1039-g9339394-bottom.rom 
Binary files build/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1039-g9339394-bottom.rom and /home/user/QubesIncoming/x230-master/heads-x230-hotp-maximized-v0.2.0-1039-g9339394-bottom.rom differ

So the full 12mb rom and bottom SPI rom are different.

Here is where hashes.txt shows build binaries differences between the builds:

user@x230-test:~/heads$ egrep '^[0-9a-f]{64}' build/x230-hotp-maximized/hashes.txt | while read line; do HASH_REF=$(echo $line|awk -F " " {'print $1'}); FILE_REF=$(echo $line|awk -F "/" {'print $NF'}); if ! grep -q "$HASH_REF" ~/QubesIncoming/x230-master/hashes.txt; then echo "$FILE_REF doesn't match";fi; done
tools.cpio doesn't match
veritysetup doesn't match
cryptsetup doesn't match
cryptsetup-reencrypt doesn't match
busybox doesn't match
libblkid.so.1 doesn't match
gawk doesn't match

Where we don't really care about gawk being different, and where tools.cpio is the output prior of compression of other tools not being reproducible.

Then to compare local to CircleCI artifacts, you need to download and extract what was packaged (initrd.cpio.xz) and compare it with local build in case of mismatch of rom images signatures.

Origin

We are interested here into commit https://github.com/osresearch/heads/commit/9339394b699fb1790eea3623d6ad5acf7334297a (9339394 commit id. Note: appended into rom name).

CircleCI builds happens on https://app.circleci.com/pipelines/github/osresearch/heads
Look for build for 9339394. Look at the jobs for your board. Click on it (here x230-htop-maximized for sake of example).
Click on the artifacts tab, which brings you here
This is the content of buid/x230-htop-maximized directory. For the sake of the reproducibility issues that happens in binaries we build, it will always be encompassed into the initrd.cpio.xz file (compressed bundled tools.cpio, modules.cpio, heads.cpio).

Now lets compare:

Remote:

mkdir -p ~/Downloads/new-test
wget https://361-64810881-gh.circle-artifacts.com/0/build/x230-hotp-maximized/initrd.cpio.xz -O ~/Downloads/new-test/initrd.cpio.xz
cd ~/Downloads/new-test
xz -d initrd.cpio.xz ; for i in initrd.cpio modules.cpio tools.cpio heads.cpio; do cpio -i < $i; done

Local:

mkdir -p new-test
cp initrd.cpio.xz new-test/
cd new-test
xz -d initrd.cpio.xz ; for i in initrd.cpio modules.cpio tools.cpio heads.cpio; do cpio -i < $i; done

Will do one diffoscope per binaries

veritysetup difference between 2 debian-10 hosts with different paths of builds:

user@x230-test:~/heads$ diffoscope  ~/Downloads/new-test/bin/veritysetup build/x230-hotp-maximized/new-test/bin/veritysetup 
--- /home/user/Downloads/new-test/bin/veritysetup
+++ build/x230-hotp-maximized/new-test/bin/veritysetup
├── readelf --wide --sections {}
│ @@ -3,16 +3,16 @@
│  Section Headers:
│    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
│    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
│    [ 1] .interp           PROGBITS        0000000000400200 000200 000019 00   A  0   0  1
│    [ 2] .hash             HASH            0000000000400220 000220 000254 04   A  4   0  8
│    [ 3] .gnu.hash         GNU_HASH        0000000000400478 000478 000038 00   A  4   0  8
│    [ 4] .dynsym           DYNSYM          00000000004004b0 0004b0 000780 18   A  5   1  8
│ -  [ 5] .dynstr           STRTAB          0000000000400c30 000c30 0003f9 00   A  0   0  1
│ -  [ 6] .gnu.version      VERSYM          000000000040102a 00102a 0000a0 02   A  4   0  2
│ +  [ 5] .dynstr           STRTAB          0000000000400c30 000c30 0003fc 00   A  0   0  1
│ +  [ 6] .gnu.version      VERSYM          000000000040102c 00102c 0000a0 02   A  4   0  2
│    [ 7] .gnu.version_r    VERNEED         00000000004010d0 0010d0 000040 00   A  5   2  8
│    [ 8] .rela.dyn         RELA            0000000000401110 001110 000048 18   A  4   0  8
│    [ 9] .rela.plt         RELA            0000000000401158 001158 0006f0 18  AI  4  21  8
│    [10] .init             PROGBITS        0000000000401848 001848 00000d 00  AX  0   0  1
│    [11] .plt              PROGBITS        0000000000401860 001860 0004b0 10  AX  0   0 16
│    [12] .text             PROGBITS        0000000000401d10 001d10 002d71 00  AX  0   0 16
│    [13] .fini             PROGBITS        0000000000404a81 004a81 000008 00  AX  0   0  1
├── readelf --wide --dynamic {}
│ @@ -2,28 +2,28 @@
│  Dynamic section at offset 0x6e08 contains 26 entries:
│    Tag        Type                         Name/Value
│   0x0000000000000001 (NEEDED)             Shared library: [libcryptsetup.so.4]
│   0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
│   0x0000000000000001 (NEEDED)             Shared library: [libdevmapper.so.1.02]
│   0x0000000000000001 (NEEDED)             Shared library: [libpopt.so.0]
│   0x0000000000000001 (NEEDED)             Shared library: [libc.so]
│ - 0x000000000000000f (RPATH)              Library rpath: [//lib:/root/project/install/lib]
│ + 0x000000000000000f (RPATH)              Library rpath: [//lib:/home/user/heads/install/lib]
│   0x000000000000000c (INIT)               0x401848
│   0x000000000000000d (FINI)               0x404a81
│   0x0000000000000004 (HASH)               0x400220
│   0x000000006ffffef5 (GNU_HASH)           0x400478
│   0x0000000000000005 (STRTAB)             0x400c30
│   0x0000000000000006 (SYMTAB)             0x4004b0
│ - 0x000000000000000a (STRSZ)              1017 (bytes)
│ + 0x000000000000000a (STRSZ)              1020 (bytes)
│   0x000000000000000b (SYMENT)             24 (bytes)
│   0x0000000000000015 (DEBUG)              0x0
│   0x0000000000000003 (PLTGOT)             0x607000
│   0x0000000000000002 (PLTRELSZ)           1776 (bytes)
│   0x0000000000000014 (PLTREL)             RELA
│   0x0000000000000017 (JMPREL)             0x401158
│   0x0000000000000007 (RELA)               0x401110
│   0x0000000000000008 (RELASZ)             72 (bytes)
│   0x0000000000000009 (RELAENT)            24 (bytes)
│   0x000000006ffffffe (VERNEED)            0x4010d0
│   0x000000006fffffff (VERNEEDNUM)         2
│ - 0x000000006ffffff0 (VERSYM)             0x40102a
│ + 0x000000006ffffff0 (VERSYM)             0x40102c
│   0x0000000000000000 (NULL)               0x0
├── readelf --wide --version-info {}
│ @@ -1,10 +1,10 @@
│  
│  Version symbols section '.gnu.version' contains 80 entries:
│ - Addr: 000000000040102a  Offset: 0x00102a  Link: 4 (.dynsym)
│ + Addr: 000000000040102c  Offset: 0x00102c  Link: 4 (.dynsym)
│    000:   0 (*local*)       2 (CRYPTSETUP_1.0)    0 (*local*)       0 (*local*)    
│    004:   0 (*local*)       0 (*local*)       0 (*local*)       0 (*local*)    
│    008:   0 (*local*)       2 (CRYPTSETUP_1.0)    3 (LIBPOPT_0)     0 (*local*)    
│    00c:   3 (LIBPOPT_0)     0 (*local*)       0 (*local*)       0 (*local*)    
│    010:   3 (LIBPOPT_0)     3 (LIBPOPT_0)     0 (*local*)       2 (CRYPTSETUP_1.0) 
│    014:   0 (*local*)       2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0) 
│    018:   2 (CRYPTSETUP_1.0)    0 (*local*)       2 (CRYPTSETUP_1.0)    0 (*local*)
├── readelf --wide --decompress --hex-dump=.dynstr {}
│ @@ -58,10 +58,10 @@
│    0x00400fa0 61646473 65740073 6967656d 70747973 addset.sigemptys
│    0x00400fb0 65740073 69677072 6f636d61 736b0073 et.sigprocmask.s
│    0x00400fc0 69676163 74696f6e 00737472 6572726f igaction.strerro
│    0x00400fd0 725f7200 7365746c 6f63616c 65005f5f r_r.setlocale.__
│    0x00400fe0 6c696263 5f737461 72745f6d 61696e00 libc_start_main.
│    0x00400ff0 4c494250 4f50545f 30004352 59505453 LIBPOPT_0.CRYPTS
│    0x00401000 45545550 5f312e30 002f2f6c 69623a2f ETUP_1.0.//lib:/
│ -  0x00401010 726f6f74 2f70726f 6a656374 2f696e73 root/project/ins
│ -  0x00401020 74616c6c 2f6c6962 00                tall/lib.
│ +  0x00401010 686f6d65 2f757365 722f6865 6164732f home/user/heads/
│ +  0x00401020 696e7374 616c6c2f 6c696200          install/lib.

cryptsetup difference between 2 debian-10 hosts with different paths of builds:

user@x230-test:~/heads$ diffoscope  ~/Downloads/new-test/bin/cryptsetup build/x230-hotp-maximized/new-test/bin/cryptsetup
--- /home/user/Downloads/new-test/bin/cryptsetup
+++ build/x230-hotp-maximized/new-test/bin/cryptsetup
├── readelf --wide --sections {}
│ @@ -3,16 +3,16 @@
│  Section Headers:
│    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
│    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
│    [ 1] .interp           PROGBITS        0000000000400200 000200 000019 00   A  0   0  1
│    [ 2] .hash             HASH            0000000000400220 000220 000358 04   A  4   0  8
│    [ 3] .gnu.hash         GNU_HASH        0000000000400578 000578 000038 00   A  4   0  8
│    [ 4] .dynsym           DYNSYM          00000000004005b0 0005b0 000ac8 18   A  5   1  8
│ -  [ 5] .dynstr           STRTAB          0000000000401078 001078 0006e5 00   A  0   0  1
│ -  [ 6] .gnu.version      VERSYM          000000000040175e 00175e 0000e6 02   A  4   0  2
│ +  [ 5] .dynstr           STRTAB          0000000000401078 001078 0006e8 00   A  0   0  1
│ +  [ 6] .gnu.version      VERSYM          0000000000401760 001760 0000e6 02   A  4   0  2
│    [ 7] .gnu.version_r    VERNEED         0000000000401848 001848 000040 00   A  5   2  8
│    [ 8] .rela.dyn         RELA            0000000000401888 001888 000048 18   A  4   0  8
│    [ 9] .rela.plt         RELA            00000000004018d0 0018d0 000a38 18  AI  4  21  8
│    [10] .init             PROGBITS        0000000000402308 002308 00000d 00  AX  0   0  1
│    [11] .plt              PROGBITS        0000000000402320 002320 0006e0 10  AX  0   0 16
│    [12] .text             PROGBITS        0000000000402a00 002a00 005e81 00  AX  0   0 16
│    [13] .fini             PROGBITS        0000000000408881 008881 000008 00  AX  0   0  1
├── readelf --wide --dynamic {}
│ @@ -2,28 +2,28 @@
│  Dynamic section at offset 0xce08 contains 26 entries:
│    Tag        Type                         Name/Value
│   0x0000000000000001 (NEEDED)             Shared library: [libcryptsetup.so.4]
│   0x0000000000000001 (NEEDED)             Shared library: [libpopt.so.0]
│   0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
│   0x0000000000000001 (NEEDED)             Shared library: [libdevmapper.so.1.02]
│   0x0000000000000001 (NEEDED)             Shared library: [libc.so]
│ - 0x000000000000000f (RPATH)              Library rpath: [//lib:/root/project/install/lib]
│ + 0x000000000000000f (RPATH)              Library rpath: [//lib:/home/user/heads/install/lib]
│   0x000000000000000c (INIT)               0x402308
│   0x000000000000000d (FINI)               0x408881
│   0x0000000000000004 (HASH)               0x400220
│   0x000000006ffffef5 (GNU_HASH)           0x400578
│   0x0000000000000005 (STRTAB)             0x401078
│   0x0000000000000006 (SYMTAB)             0x4005b0
│ - 0x000000000000000a (STRSZ)              1765 (bytes)
│ + 0x000000000000000a (STRSZ)              1768 (bytes)
│   0x000000000000000b (SYMENT)             24 (bytes)
│   0x0000000000000015 (DEBUG)              0x0
│   0x0000000000000003 (PLTGOT)             0x60d000
│   0x0000000000000002 (PLTRELSZ)           2616 (bytes)
│   0x0000000000000014 (PLTREL)             RELA
│   0x0000000000000017 (JMPREL)             0x4018d0
│   0x0000000000000007 (RELA)               0x401888
│   0x0000000000000008 (RELASZ)             72 (bytes)
│   0x0000000000000009 (RELAENT)            24 (bytes)
│   0x000000006ffffffe (VERNEED)            0x401848
│   0x000000006fffffff (VERNEEDNUM)         2
│ - 0x000000006ffffff0 (VERSYM)             0x40175e
│ + 0x000000006ffffff0 (VERSYM)             0x401760
│   0x0000000000000000 (NULL)               0x0
├── readelf --wide --version-info {}
│ @@ -1,10 +1,10 @@
│  
│  Version symbols section '.gnu.version' contains 115 entries:
│ - Addr: 000000000040175e  Offset: 0x00175e  Link: 4 (.dynsym)
│ + Addr: 0000000000401760  Offset: 0x001760  Link: 4 (.dynsym)
│    000:   0 (*local*)       2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0)    0 (*local*)    
│    004:   0 (*local*)       0 (*local*)       0 (*local*)       2 (CRYPTSETUP_1.0) 
│    008:   0 (*local*)       0 (*local*)       0 (*local*)       2 (CRYPTSETUP_1.0) 
│    00c:   3 (LIBPOPT_0)     0 (*local*)       3 (LIBPOPT_0)     0 (*local*)    
│    010:   0 (*local*)       0 (*local*)       3 (LIBPOPT_0)     2 (CRYPTSETUP_1.0) 
│    014:   2 (CRYPTSETUP_1.0)    3 (LIBPOPT_0)     0 (*local*)       2 (CRYPTSETUP_1.0) 
│    018:   0 (*local*)       2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0)    0 (*local*)
├── readelf --wide --decompress --hex-dump=.dynstr {}
│ @@ -104,11 +104,11 @@
│    0x004016c8 736f0073 7464696e 00736967 61646473 so.stdin.sigadds
│    0x004016d8 65740073 6967656d 70747973 65740073 et.sigemptyset.s
│    0x004016e8 69677072 6f636d61 736b0073 69676163 igprocmask.sigac
│    0x004016f8 74696f6e 00737472 6572726f 725f7200 tion.strerror_r.
│    0x00401708 7365746c 6f63616c 65005f5f 6c696263 setlocale.__libc
│    0x00401718 5f737461 72745f6d 61696e00 4c494250 _start_main.LIBP
│    0x00401728 4f50545f 30004352 59505453 45545550 OPT_0.CRYPTSETUP
│ -  0x00401738 5f312e30 002f2f6c 69623a2f 726f6f74 _1.0.//lib:/root
│ -  0x00401748 2f70726f 6a656374 2f696e73 74616c6c /project/install
│ -  0x00401758 2f6c6962 00                         /lib.
│ +  0x00401738 5f312e30 002f2f6c 69623a2f 686f6d65 _1.0.//lib:/home
│ +  0x00401748 2f757365 722f6865 6164732f 696e7374 /user/heads/inst
│ +  0x00401758 616c6c2f 6c696200                   all/lib.

cryptsetup-reencrypt difference between 2 debian-10 hosts with different paths of builds:

user@x230-test:~/heads$ diffoscope  ~/Downloads/new-test/bin/cryptsetup-reencrypt build/x230-hotp-maximized/new-test/bin/cryptsetup-reencrypt
--- /home/user/Downloads/new-test/bin/cryptsetup-reencrypt
+++ build/x230-hotp-maximized/new-test/bin/cryptsetup-reencrypt
├── readelf --wide --sections {}
│ @@ -3,20 +3,20 @@
│  Section Headers:
│    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
│    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
│    [ 1] .interp           PROGBITS        0000000000400200 000200 000019 00   A  0   0  1
│    [ 2] .hash             HASH            0000000000400220 000220 000294 04   A  4   0  8
│    [ 3] .gnu.hash         GNU_HASH        00000000004004b8 0004b8 000038 00   A  4   0  8
│    [ 4] .dynsym           DYNSYM          00000000004004f0 0004f0 000900 18   A  5   1  8
│ -  [ 5] .dynstr           STRTAB          0000000000400df0 000df0 0004f0 00   A  0   0  1
│ -  [ 6] .gnu.version      VERSYM          00000000004012e0 0012e0 0000c0 02   A  4   0  2
│ -  [ 7] .gnu.version_r    VERNEED         00000000004013a0 0013a0 000060 00   A  5   3  8
│ -  [ 8] .rela.dyn         RELA            0000000000401400 001400 000048 18   A  4   0  8
│ -  [ 9] .rela.plt         RELA            0000000000401448 001448 000870 18  AI  4  21  8
│ -  [10] .init             PROGBITS        0000000000401cb8 001cb8 00000d 00  AX  0   0  1
│ +  [ 5] .dynstr           STRTAB          0000000000400df0 000df0 0004f3 00   A  0   0  1
│ +  [ 6] .gnu.version      VERSYM          00000000004012e4 0012e4 0000c0 02   A  4   0  2
│ +  [ 7] .gnu.version_r    VERNEED         00000000004013a8 0013a8 000060 00   A  5   3  8
│ +  [ 8] .rela.dyn         RELA            0000000000401408 001408 000048 18   A  4   0  8
│ +  [ 9] .rela.plt         RELA            0000000000401450 001450 000870 18  AI  4  21  8
│ +  [10] .init             PROGBITS        0000000000401cc0 001cc0 00000d 00  AX  0   0  1
│    [11] .plt              PROGBITS        0000000000401cd0 001cd0 0005b0 10  AX  0   0 16
│    [12] .text             PROGBITS        0000000000402280 002280 0044f1 00  AX  0   0 16
│    [13] .fini             PROGBITS        0000000000406771 006771 000008 00  AX  0   0  1
│    [14] .rodata           PROGBITS        0000000000406780 006780 001990 00   A  0   0 32
│    [15] .eh_frame_hdr     PROGBITS        0000000000408110 008110 00013c 00   A  0   0  4
│    [16] .eh_frame         PROGBITS        0000000000408250 008250 0008cc 00   A  0   0  8
│    [17] .ctors            PROGBITS        0000000000608dd0 008dd0 000010 00  WA  0   0  8
├── readelf --wide --symbols {}
│ @@ -90,10 +90,10 @@
│      86: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND sigaction
│      87: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND memcpy
│      88: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND strlen
│      89: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND crypt_get_dir@CRYPTSETUP_1.0 (2)
│      90: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getline
│      91: 0000000000608df8     8 OBJECT  GLOBAL DEFAULT   19 stdout
│      92: 0000000000608df0     8 OBJECT  GLOBAL DEFAULT   19 stderr
│ -    93: 0000000000401cb8     1 FUNC    GLOBAL DEFAULT   10 _init
│ +    93: 0000000000401cc0     1 FUNC    GLOBAL DEFAULT   10 _init
│      94: 0000000000608e00     8 OBJECT  GLOBAL DEFAULT   19 stdin
│      95: 0000000000406771     1 FUNC    GLOBAL DEFAULT   13 _fini
├── readelf --wide --relocs {}
│ @@ -1,15 +1,15 @@
│  
│ -Relocation section '.rela.dyn' at offset 0x1400 contains 3 entries:
│ +Relocation section '.rela.dyn' at offset 0x1408 contains 3 entries:
│      Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
│  0000000000608df0  0000005c00000005 R_X86_64_COPY          0000000000608df0 stderr + 0
│  0000000000608df8  0000005b00000005 R_X86_64_COPY          0000000000608df8 stdout + 0
│  0000000000608e00  0000005e00000005 R_X86_64_COPY          0000000000608e00 stdin + 0
│  
│ -Relocation section '.rela.plt' at offset 0x1448 contains 90 entries:
│ +Relocation section '.rela.plt' at offset 0x1450 contains 90 entries:
│      Offset             Info             Type               Symbol's Value  Symbol's Name + Addend
│  0000000000609018  0000000100000007 R_X86_64_JUMP_SLOT     0000000000000000 crypt_keyslot_status@CRYPTSETUP_1.0 + 0
│  0000000000609020  0000000200000007 R_X86_64_JUMP_SLOT     0000000000000000 memcmp + 0
│  0000000000609028  0000000300000007 R_X86_64_JUMP_SLOT     0000000000000000 ioctl + 0
│  0000000000609030  0000000400000007 R_X86_64_JUMP_SLOT     0000000000000000 write + 0
│  0000000000609038  0000000500000007 R_X86_64_JUMP_SLOT     0000000000000000 tcsetattr + 0
│  0000000000609040  0000000600000007 R_X86_64_JUMP_SLOT     0000000000000000 sscanf + 0
├── readelf --wide --dynamic {}
│ @@ -2,28 +2,28 @@
│  Dynamic section at offset 0x8e08 contains 26 entries:
│    Tag        Type                         Name/Value
│   0x0000000000000001 (NEEDED)             Shared library: [libcryptsetup.so.4]
│   0x0000000000000001 (NEEDED)             Shared library: [libpopt.so.0]
│   0x0000000000000001 (NEEDED)             Shared library: [libdevmapper.so.1.02]
│   0x0000000000000001 (NEEDED)             Shared library: [libuuid.so.1]
│   0x0000000000000001 (NEEDED)             Shared library: [libc.so]
│ - 0x000000000000000f (RPATH)              Library rpath: [//lib:/root/project/install/lib]
│ - 0x000000000000000c (INIT)               0x401cb8
│ + 0x000000000000000f (RPATH)              Library rpath: [//lib:/home/user/heads/install/lib]
│ + 0x000000000000000c (INIT)               0x401cc0
│   0x000000000000000d (FINI)               0x406771
│   0x0000000000000004 (HASH)               0x400220
│   0x000000006ffffef5 (GNU_HASH)           0x4004b8
│   0x0000000000000005 (STRTAB)             0x400df0
│   0x0000000000000006 (SYMTAB)             0x4004f0
│ - 0x000000000000000a (STRSZ)              1264 (bytes)
│ + 0x000000000000000a (STRSZ)              1267 (bytes)
│   0x000000000000000b (SYMENT)             24 (bytes)
│   0x0000000000000015 (DEBUG)              0x0
│   0x0000000000000003 (PLTGOT)             0x609000
│   0x0000000000000002 (PLTRELSZ)           2160 (bytes)
│   0x0000000000000014 (PLTREL)             RELA
│ - 0x0000000000000017 (JMPREL)             0x401448
│ - 0x0000000000000007 (RELA)               0x401400
│ + 0x0000000000000017 (JMPREL)             0x401450
│ + 0x0000000000000007 (RELA)               0x401408
│   0x0000000000000008 (RELASZ)             72 (bytes)
│   0x0000000000000009 (RELAENT)            24 (bytes)
│ - 0x000000006ffffffe (VERNEED)            0x4013a0
│ + 0x000000006ffffffe (VERNEED)            0x4013a8
│   0x000000006fffffff (VERNEEDNUM)         3
│ - 0x000000006ffffff0 (VERSYM)             0x4012e0
│ + 0x000000006ffffff0 (VERSYM)             0x4012e4
│   0x0000000000000000 (NULL)               0x0
├── readelf --wide --version-info {}
│ @@ -1,10 +1,10 @@
│  
│  Version symbols section '.gnu.version' contains 96 entries:
│ - Addr: 00000000004012e0  Offset: 0x0012e0  Link: 4 (.dynsym)
│ + Addr: 00000000004012e4  Offset: 0x0012e4  Link: 4 (.dynsym)
│    000:   0 (*local*)       2 (CRYPTSETUP_1.0)    0 (*local*)       0 (*local*)    
│    004:   0 (*local*)       0 (*local*)       0 (*local*)       0 (*local*)    
│    008:   0 (*local*)       2 (CRYPTSETUP_1.0)    0 (*local*)       0 (*local*)    
│    00c:   0 (*local*)       2 (CRYPTSETUP_1.0)    0 (*local*)       0 (*local*)    
│    010:   3 (LIBPOPT_0)     0 (*local*)       0 (*local*)       0 (*local*)    
│    014:   3 (LIBPOPT_0)     0 (*local*)       0 (*local*)       0 (*local*)    
│    018:   3 (LIBPOPT_0)     2 (CRYPTSETUP_1.0)    3 (LIBPOPT_0)     0 (*local*)    
│ @@ -23,14 +23,14 @@
│    04c:   3 (LIBPOPT_0)     2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0) 
│    050:   0 (*local*)       0 (*local*)       0 (*local*)       0 (*local*)    
│    054:   2 (CRYPTSETUP_1.0)    2 (CRYPTSETUP_1.0)    0 (*local*)       0 (*local*)    
│    058:   0 (*local*)       2 (CRYPTSETUP_1.0)    0 (*local*)       0 (*local*)    
│    05c:   0 (*local*)       1 (*global*)      0 (*local*)       1 (*global*)   
│  
│  Version needs section '.gnu.version_r' contains 3 entries:
│ - Addr: 0x00000000004013a0  Offset: 0x0013a0  Link: 5 (.dynstr)
│ + Addr: 0x00000000004013a8  Offset: 0x0013a8  Link: 5 (.dynstr)
│    000000: Version: 1  File: libuuid.so.1  Cnt: 1
│    0x0010:   Name: UUID_1.0  Flags: none  Version: 4
│    0x0020: Version: 1  File: libpopt.so.0  Cnt: 1
│    0x0030:   Name: LIBPOPT_0  Flags: none  Version: 3
│    0x0040: Version: 1  File: libcryptsetup.so.4  Cnt: 1
│    0x0050:   Name: CRYPTSETUP_1.0  Flags: none  Version: 2
├── readelf --wide --decompress --hex-dump=.dynstr {}
│ @@ -73,10 +73,11 @@
│    0x00401250 64736574 00736967 656d7074 79736574 dset.sigemptyset
│    0x00401260 00736967 70726f63 6d61736b 00736967 .sigprocmask.sig
│    0x00401270 61637469 6f6e0073 74726572 726f725f action.strerror_
│    0x00401280 72007365 746c6f63 616c6500 5f5f6c69 r.setlocale.__li
│    0x00401290 62635f73 74617274 5f6d6169 6e005555 bc_start_main.UU
│    0x004012a0 49445f31 2e30004c 4942504f 50545f30 ID_1.0.LIBPOPT_0
│    0x004012b0 00435259 50545345 5455505f 312e3000 .CRYPTSETUP_1.0.
│ -  0x004012c0 2f2f6c69 623a2f72 6f6f742f 70726f6a //lib:/root/proj
│ -  0x004012d0 6563742f 696e7374 616c6c2f 6c696200 ect/install/lib.
│ +  0x004012c0 2f2f6c69 623a2f68 6f6d652f 75736572 //lib:/home/user
│ +  0x004012d0 2f686561 64732f69 6e737461 6c6c2f6c /heads/install/l
│ +  0x004012e0 696200                              ib.
├── objdump --line-numbers --disassemble --demangle --reloc --section=.init {}
│ @@ -1,11 +1,11 @@
│  
│  
│  
│  Disassembly of section .init:
│  
│ -0000000000401cb8 <_init@@Base>:
│ -  401cb8:    50                      push   %rax
│ -  401cb9:    e8 82 22 00 00          callq  403f40 <getline@plt+0x1cd0>
│ -  401cbe:    e8 6d 4a 00 00          callq  406730 <getline@plt+0x44c0>
│ -  401cc3:    58                      pop    %rax
│ -  401cc4:    c3                      retq   
│ +0000000000401cc0 <_init@@Base>:
│ +  401cc0:    50                      push   %rax
│ +  401cc1:    e8 7a 22 00 00          callq  403f40 <getline@plt+0x1cd0>
│ +  401cc6:    e8 65 4a 00 00          callq  406730 <getline@plt+0x44c0>
│ +  401ccb:    58                      pop    %rax
│ +  401ccc:    c3                      retq   
├── objdump --line-numbers --disassemble --demangle --reloc --section=.plt {}
│ @@ -7,453 +7,453 @@
│    401cd0:    ff 35 32 73 20 00       pushq  0x207332(%rip)        
│    401cd6:    ff 25 34 73 20 00       jmpq   *0x207334(%rip)        
│    401cdc:    0f 1f 40 00             nopl   0x0(%rax)
│  
│  0000000000401ce0 <crypt_keyslot_status@plt>:
│    401ce0:    ff 25 32 73 20 00       jmpq   *0x207332(%rip)        
│    401ce6:    68 00 00 00 00          pushq  $0x0
│ -  401ceb:    e9 e0 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401ceb:    e9 e0 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401cf0 <memcmp@plt>:
│    401cf0:    ff 25 2a 73 20 00       jmpq   *0x20732a(%rip)        
│    401cf6:    68 01 00 00 00          pushq  $0x1
│ -  401cfb:    e9 d0 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401cfb:    e9 d0 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d00 <ioctl@plt>:
│    401d00:    ff 25 22 73 20 00       jmpq   *0x207322(%rip)        
│    401d06:    68 02 00 00 00          pushq  $0x2
│ -  401d0b:    e9 c0 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d0b:    e9 c0 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d10 <write@plt>:
│    401d10:    ff 25 1a 73 20 00       jmpq   *0x20731a(%rip)        
│    401d16:    68 03 00 00 00          pushq  $0x3
│ -  401d1b:    e9 b0 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d1b:    e9 b0 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d20 <tcsetattr@plt>:
│    401d20:    ff 25 12 73 20 00       jmpq   *0x207312(%rip)        
│    401d26:    68 04 00 00 00          pushq  $0x4
│ -  401d2b:    e9 a0 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d2b:    e9 a0 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d30 <sscanf@plt>:
│    401d30:    ff 25 0a 73 20 00       jmpq   *0x20730a(%rip)        
│    401d36:    68 05 00 00 00          pushq  $0x5
│ -  401d3b:    e9 90 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d3b:    e9 90 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d40 <fsync@plt>:
│    401d40:    ff 25 02 73 20 00       jmpq   *0x207302(%rip)        
│    401d46:    68 06 00 00 00          pushq  $0x6
│ -  401d4b:    e9 80 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d4b:    e9 80 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d50 <gettimeofday@plt>:
│    401d50:    ff 25 fa 72 20 00       jmpq   *0x2072fa(%rip)        
│    401d56:    68 07 00 00 00          pushq  $0x7
│ -  401d5b:    e9 70 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d5b:    e9 70 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d60 <crypt_header_backup@plt>:
│    401d60:    ff 25 f2 72 20 00       jmpq   *0x2072f2(%rip)        
│    401d66:    68 08 00 00 00          pushq  $0x8
│ -  401d6b:    e9 60 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d6b:    e9 60 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d70 <snprintf@plt>:
│    401d70:    ff 25 ea 72 20 00       jmpq   *0x2072ea(%rip)        
│    401d76:    68 09 00 00 00          pushq  $0x9
│ -  401d7b:    e9 50 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d7b:    e9 50 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d80 <lseek@plt>:
│    401d80:    ff 25 e2 72 20 00       jmpq   *0x2072e2(%rip)        
│    401d86:    68 0a 00 00 00          pushq  $0xa
│ -  401d8b:    e9 40 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d8b:    e9 40 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401d90 <strcmp@plt>:
│    401d90:    ff 25 da 72 20 00       jmpq   *0x2072da(%rip)        
│    401d96:    68 0b 00 00 00          pushq  $0xb
│ -  401d9b:    e9 30 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401d9b:    e9 30 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401da0 <crypt_set_log_callback@plt>:
│    401da0:    ff 25 d2 72 20 00       jmpq   *0x2072d2(%rip)        
│    401da6:    68 0c 00 00 00          pushq  $0xc
│ -  401dab:    e9 20 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401dab:    e9 20 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401db0 <creat@plt>:
│    401db0:    ff 25 ca 72 20 00       jmpq   *0x2072ca(%rip)        
│    401db6:    68 0d 00 00 00          pushq  $0xd
│ -  401dbb:    e9 10 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401dbb:    e9 10 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401dc0 <strndup@plt>:
│    401dc0:    ff 25 c2 72 20 00       jmpq   *0x2072c2(%rip)        
│    401dc6:    68 0e 00 00 00          pushq  $0xe
│ -  401dcb:    e9 00 ff ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401dcb:    e9 00 ff ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401dd0 <poptSetOtherOptionHelp@plt>:
│    401dd0:    ff 25 ba 72 20 00       jmpq   *0x2072ba(%rip)        
│    401dd6:    68 0f 00 00 00          pushq  $0xf
│ -  401ddb:    e9 f0 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401ddb:    e9 f0 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401de0 <memset@plt>:
│    401de0:    ff 25 b2 72 20 00       jmpq   *0x2072b2(%rip)        
│    401de6:    68 10 00 00 00          pushq  $0x10
│ -  401deb:    e9 e0 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401deb:    e9 e0 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401df0 <unlink@plt>:
│    401df0:    ff 25 aa 72 20 00       jmpq   *0x2072aa(%rip)        
│    401df6:    68 11 00 00 00          pushq  $0x11
│ -  401dfb:    e9 d0 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401dfb:    e9 d0 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e00 <malloc@plt>:
│    401e00:    ff 25 a2 72 20 00       jmpq   *0x2072a2(%rip)        
│    401e06:    68 12 00 00 00          pushq  $0x12
│ -  401e0b:    e9 c0 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e0b:    e9 c0 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e10 <poptGetContext@plt>:
│    401e10:    ff 25 9a 72 20 00       jmpq   *0x20729a(%rip)        
│    401e16:    68 13 00 00 00          pushq  $0x13
│ -  401e1b:    e9 b0 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e1b:    e9 b0 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e20 <fflush@plt>:
│    401e20:    ff 25 92 72 20 00       jmpq   *0x207292(%rip)        
│    401e26:    68 14 00 00 00          pushq  $0x14
│ -  401e2b:    e9 a0 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e2b:    e9 a0 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e30 <strncmp@plt>:
│    401e30:    ff 25 8a 72 20 00       jmpq   *0x20728a(%rip)        
│    401e36:    68 15 00 00 00          pushq  $0x15
│ -  401e3b:    e9 90 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e3b:    e9 90 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e40 <isxdigit@plt>:
│    401e40:    ff 25 82 72 20 00       jmpq   *0x207282(%rip)        
│    401e46:    68 16 00 00 00          pushq  $0x16
│ -  401e4b:    e9 80 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e4b:    e9 80 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e50 <poptGetArgs@plt>:
│    401e50:    ff 25 7a 72 20 00       jmpq   *0x20727a(%rip)        
│    401e56:    68 17 00 00 00          pushq  $0x17
│ -  401e5b:    e9 70 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e5b:    e9 70 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e60 <crypt_keyslot_add_by_volume_key@plt>:
│    401e60:    ff 25 72 72 20 00       jmpq   *0x207272(%rip)        
│    401e66:    68 18 00 00 00          pushq  $0x18
│ -  401e6b:    e9 60 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e6b:    e9 60 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e70 <poptPrintHelp@plt>:
│    401e70:    ff 25 6a 72 20 00       jmpq   *0x20726a(%rip)        
│    401e76:    68 19 00 00 00          pushq  $0x19
│ -  401e7b:    e9 50 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e7b:    e9 50 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e80 <strdup@plt>:
│    401e80:    ff 25 62 72 20 00       jmpq   *0x207262(%rip)        
│    401e86:    68 1a 00 00 00          pushq  $0x1a
│ -  401e8b:    e9 40 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e8b:    e9 40 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401e90 <crypt_free@plt>:
│    401e90:    ff 25 5a 72 20 00       jmpq   *0x20725a(%rip)        
│    401e96:    68 1b 00 00 00          pushq  $0x1b
│ -  401e9b:    e9 30 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401e9b:    e9 30 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401ea0 <open@plt>:
│    401ea0:    ff 25 52 72 20 00       jmpq   *0x207252(%rip)        
│    401ea6:    68 1c 00 00 00          pushq  $0x1c
│ -  401eab:    e9 20 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401eab:    e9 20 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401eb0 <crypt_init@plt>:
│    401eb0:    ff 25 4a 72 20 00       jmpq   *0x20724a(%rip)        
│    401eb6:    68 1d 00 00 00          pushq  $0x1d
│ -  401ebb:    e9 10 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401ebb:    e9 10 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401ec0 <crypt_get_cipher@plt>:
│    401ec0:    ff 25 42 72 20 00       jmpq   *0x207242(%rip)        
│    401ec6:    68 1e 00 00 00          pushq  $0x1e
│ -  401ecb:    e9 00 fe ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401ecb:    e9 00 fe ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401ed0 <crypt_header_restore@plt>:
│    401ed0:    ff 25 3a 72 20 00       jmpq   *0x20723a(%rip)        
│    401ed6:    68 1f 00 00 00          pushq  $0x1f
│ -  401edb:    e9 f0 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401edb:    e9 f0 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401ee0 <crypt_log@plt>:
│    401ee0:    ff 25 32 72 20 00       jmpq   *0x207232(%rip)        
│    401ee6:    68 20 00 00 00          pushq  $0x20
│ -  401eeb:    e9 e0 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401eeb:    e9 e0 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401ef0 <strchr@plt>:
│    401ef0:    ff 25 2a 72 20 00       jmpq   *0x20722a(%rip)        
│    401ef6:    68 21 00 00 00          pushq  $0x21
│ -  401efb:    e9 d0 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401efb:    e9 d0 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f00 <crypt_set_data_device@plt>:
│    401f00:    ff 25 22 72 20 00       jmpq   *0x207222(%rip)        
│    401f06:    68 22 00 00 00          pushq  $0x22
│ -  401f0b:    e9 c0 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f0b:    e9 c0 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f10 <uuid_parse@plt>:
│    401f10:    ff 25 1a 72 20 00       jmpq   *0x20721a(%rip)        
│    401f16:    68 23 00 00 00          pushq  $0x23
│ -  401f1b:    e9 b0 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f1b:    e9 b0 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f20 <printf@plt>:
│    401f20:    ff 25 12 72 20 00       jmpq   *0x207212(%rip)        
│    401f26:    68 24 00 00 00          pushq  $0x24
│ -  401f2b:    e9 a0 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f2b:    e9 a0 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f30 <__libc_start_main@plt>:
│    401f30:    ff 25 0a 72 20 00       jmpq   *0x20720a(%rip)        
│    401f36:    68 25 00 00 00          pushq  $0x25
│ -  401f3b:    e9 90 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f3b:    e9 90 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f40 <poptGetInvocationName@plt>:
│    401f40:    ff 25 02 72 20 00       jmpq   *0x207202(%rip)        
│    401f46:    68 26 00 00 00          pushq  $0x26
│ -  401f4b:    e9 80 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f4b:    e9 80 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f50 <crypt_format@plt>:
│    401f50:    ff 25 fa 71 20 00       jmpq   *0x2071fa(%rip)        
│    401f56:    68 27 00 00 00          pushq  $0x27
│ -  401f5b:    e9 70 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f5b:    e9 70 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f60 <sigemptyset@plt>:
│    401f60:    ff 25 f2 71 20 00       jmpq   *0x2071f2(%rip)        
│    401f66:    68 28 00 00 00          pushq  $0x28
│ -  401f6b:    e9 60 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f6b:    e9 60 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f70 <poptGetNextOpt@plt>:
│    401f70:    ff 25 ea 71 20 00       jmpq   *0x2071ea(%rip)        
│    401f76:    68 29 00 00 00          pushq  $0x29
│ -  401f7b:    e9 50 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f7b:    e9 50 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f80 <exit@plt>:
│    401f80:    ff 25 e2 71 20 00       jmpq   *0x2071e2(%rip)        
│    401f86:    68 2a 00 00 00          pushq  $0x2a
│ -  401f8b:    e9 40 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f8b:    e9 40 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401f90 <fprintf@plt>:
│    401f90:    ff 25 da 71 20 00       jmpq   *0x2071da(%rip)        
│    401f96:    68 2b 00 00 00          pushq  $0x2b
│ -  401f9b:    e9 30 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401f9b:    e9 30 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401fa0 <posix_memalign@plt>:
│    401fa0:    ff 25 d2 71 20 00       jmpq   *0x2071d2(%rip)        
│    401fa6:    68 2c 00 00 00          pushq  $0x2c
│ -  401fab:    e9 20 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401fab:    e9 20 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401fb0 <poptBadOption@plt>:
│    401fb0:    ff 25 ca 71 20 00       jmpq   *0x2071ca(%rip)        
│    401fb6:    68 2d 00 00 00          pushq  $0x2d
│ -  401fbb:    e9 10 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401fbb:    e9 10 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401fc0 <tcgetattr@plt>:
│    401fc0:    ff 25 c2 71 20 00       jmpq   *0x2071c2(%rip)        
│    401fc6:    68 2e 00 00 00          pushq  $0x2e
│ -  401fcb:    e9 00 fd ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401fcb:    e9 00 fd ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401fd0 <crypt_get_volume_key_size@plt>:
│    401fd0:    ff 25 ba 71 20 00       jmpq   *0x2071ba(%rip)        
│    401fd6:    68 2f 00 00 00          pushq  $0x2f
│ -  401fdb:    e9 f0 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401fdb:    e9 f0 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401fe0 <strerror_r@plt>:
│    401fe0:    ff 25 b2 71 20 00       jmpq   *0x2071b2(%rip)        
│    401fe6:    68 30 00 00 00          pushq  $0x30
│ -  401feb:    e9 e0 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401feb:    e9 e0 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000401ff0 <crypt_get_error@plt>:
│    401ff0:    ff 25 aa 71 20 00       jmpq   *0x2071aa(%rip)        
│    401ff6:    68 31 00 00 00          pushq  $0x31
│ -  401ffb:    e9 d0 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  401ffb:    e9 d0 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402000 <strtoul@plt>:
│    402000:    ff 25 a2 71 20 00       jmpq   *0x2071a2(%rip)        
│    402006:    68 32 00 00 00          pushq  $0x32
│ -  40200b:    e9 c0 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40200b:    e9 c0 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402010 <close@plt>:
│    402010:    ff 25 9a 71 20 00       jmpq   *0x20719a(%rip)        
│    402016:    68 33 00 00 00          pushq  $0x33
│ -  40201b:    e9 b0 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40201b:    e9 b0 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402020 <crypt_deactivate@plt>:
│    402020:    ff 25 92 71 20 00       jmpq   *0x207192(%rip)        
│    402026:    68 34 00 00 00          pushq  $0x34
│ -  40202b:    e9 a0 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40202b:    e9 a0 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402030 <sysconf@plt>:
│    402030:    ff 25 8a 71 20 00       jmpq   *0x20718a(%rip)        
│    402036:    68 35 00 00 00          pushq  $0x35
│ -  40203b:    e9 90 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40203b:    e9 90 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402040 <fpathconf@plt>:
│    402040:    ff 25 82 71 20 00       jmpq   *0x207182(%rip)        
│    402046:    68 36 00 00 00          pushq  $0x36
│ -  40204b:    e9 80 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40204b:    e9 80 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402050 <crypt_activate_by_passphrase@plt>:
│    402050:    ff 25 7a 71 20 00       jmpq   *0x20717a(%rip)        
│    402056:    68 37 00 00 00          pushq  $0x37
│ -  40205b:    e9 70 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40205b:    e9 70 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402060 <setlocale@plt>:
│    402060:    ff 25 72 71 20 00       jmpq   *0x207172(%rip)        
│    402066:    68 38 00 00 00          pushq  $0x38
│ -  40206b:    e9 60 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40206b:    e9 60 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402070 <calloc@plt>:
│    402070:    ff 25 6a 71 20 00       jmpq   *0x20716a(%rip)        
│    402076:    68 39 00 00 00          pushq  $0x39
│ -  40207b:    e9 50 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40207b:    e9 50 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402080 <poptPrintUsage@plt>:
│    402080:    ff 25 62 71 20 00       jmpq   *0x207162(%rip)        
│    402086:    68 3a 00 00 00          pushq  $0x3a
│ -  40208b:    e9 40 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40208b:    e9 40 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402090 <crypt_set_debug_level@plt>:
│    402090:    ff 25 5a 71 20 00       jmpq   *0x20715a(%rip)        
│    402096:    68 3b 00 00 00          pushq  $0x3b
│ -  40209b:    e9 30 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40209b:    e9 30 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004020a0 <crypt_keyslot_max@plt>:
│    4020a0:    ff 25 52 71 20 00       jmpq   *0x207152(%rip)        
│    4020a6:    68 3c 00 00 00          pushq  $0x3c
│ -  4020ab:    e9 20 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4020ab:    e9 20 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004020b0 <read@plt>:
│    4020b0:    ff 25 4a 71 20 00       jmpq   *0x20714a(%rip)        
│    4020b6:    68 3d 00 00 00          pushq  $0x3d
│ -  4020bb:    e9 10 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4020bb:    e9 10 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004020c0 <sigprocmask@plt>:
│    4020c0:    ff 25 42 71 20 00       jmpq   *0x207142(%rip)        
│    4020c6:    68 3e 00 00 00          pushq  $0x3e
│ -  4020cb:    e9 00 fc ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4020cb:    e9 00 fc ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004020d0 <crypt_volume_key_get@plt>:
│    4020d0:    ff 25 3a 71 20 00       jmpq   *0x20713a(%rip)        
│    4020d6:    68 3f 00 00 00          pushq  $0x3f
│ -  4020db:    e9 f0 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4020db:    e9 f0 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004020e0 <crypt_set_confirm_callback@plt>:
│    4020e0:    ff 25 32 71 20 00       jmpq   *0x207132(%rip)        
│    4020e6:    68 40 00 00 00          pushq  $0x40
│ -  4020eb:    e9 e0 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4020eb:    e9 e0 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004020f0 <strtoull@plt>:
│    4020f0:    ff 25 2a 71 20 00       jmpq   *0x20712a(%rip)        
│    4020f6:    68 41 00 00 00          pushq  $0x41
│ -  4020fb:    e9 d0 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4020fb:    e9 d0 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402100 <__errno_location@plt>:
│    402100:    ff 25 22 71 20 00       jmpq   *0x207122(%rip)        
│    402106:    68 42 00 00 00          pushq  $0x42
│ -  40210b:    e9 c0 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40210b:    e9 c0 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402110 <fputs@plt>:
│    402110:    ff 25 1a 71 20 00       jmpq   *0x20711a(%rip)        
│    402116:    68 43 00 00 00          pushq  $0x43
│ -  40211b:    e9 b0 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40211b:    e9 b0 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402120 <crypt_set_iteration_time@plt>:
│    402120:    ff 25 12 71 20 00       jmpq   *0x207112(%rip)        
│    402126:    68 44 00 00 00          pushq  $0x44
│ -  40212b:    e9 a0 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40212b:    e9 a0 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402130 <poptFreeContext@plt>:
│    402130:    ff 25 0a 71 20 00       jmpq   *0x20710a(%rip)        
│    402136:    68 45 00 00 00          pushq  $0x45
│ -  40213b:    e9 90 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40213b:    e9 90 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402140 <tolower@plt>:
│    402140:    ff 25 02 71 20 00       jmpq   *0x207102(%rip)        
│    402146:    68 46 00 00 00          pushq  $0x46
│ -  40214b:    e9 80 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40214b:    e9 80 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402150 <free@plt>:
│    402150:    ff 25 fa 70 20 00       jmpq   *0x2070fa(%rip)        
│    402156:    68 47 00 00 00          pushq  $0x47
│ -  40215b:    e9 70 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40215b:    e9 70 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402160 <atoi@plt>:
│    402160:    ff 25 f2 70 20 00       jmpq   *0x2070f2(%rip)        
│    402166:    68 48 00 00 00          pushq  $0x48
│ -  40216b:    e9 60 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40216b:    e9 60 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402170 <stat@plt>:
│    402170:    ff 25 ea 70 20 00       jmpq   *0x2070ea(%rip)        
│    402176:    68 49 00 00 00          pushq  $0x49
│ -  40217b:    e9 50 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40217b:    e9 50 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402180 <isatty@plt>:
│    402180:    ff 25 e2 70 20 00       jmpq   *0x2070e2(%rip)        
│    402186:    68 4a 00 00 00          pushq  $0x4a
│ -  40218b:    e9 40 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40218b:    e9 40 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402190 <poptStrerror@plt>:
│    402190:    ff 25 da 70 20 00       jmpq   *0x2070da(%rip)        
│    402196:    68 4b 00 00 00          pushq  $0x4b
│ -  40219b:    e9 30 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40219b:    e9 30 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004021a0 <crypt_get_data_offset@plt>:
│    4021a0:    ff 25 d2 70 20 00       jmpq   *0x2070d2(%rip)        
│    4021a6:    68 4c 00 00 00          pushq  $0x4c
│ -  4021ab:    e9 20 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4021ab:    e9 20 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004021b0 <crypt_get_cipher_mode@plt>:
│    4021b0:    ff 25 ca 70 20 00       jmpq   *0x2070ca(%rip)        
│    4021b6:    68 4d 00 00 00          pushq  $0x4d
│ -  4021bb:    e9 10 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4021bb:    e9 10 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004021c0 <crypt_get_uuid@plt>:
│    4021c0:    ff 25 c2 70 20 00       jmpq   *0x2070c2(%rip)        
│    4021c6:    68 4e 00 00 00          pushq  $0x4e
│ -  4021cb:    e9 00 fb ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4021cb:    e9 00 fb ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004021d0 <vasprintf@plt>:
│    4021d0:    ff 25 ba 70 20 00       jmpq   *0x2070ba(%rip)        
│    4021d6:    68 4f 00 00 00          pushq  $0x4f
│ -  4021db:    e9 f0 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4021db:    e9 f0 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004021e0 <select@plt>:
│    4021e0:    ff 25 b2 70 20 00       jmpq   *0x2070b2(%rip)        
│    4021e6:    68 50 00 00 00          pushq  $0x50
│ -  4021eb:    e9 e0 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4021eb:    e9 e0 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  00000000004021f0 <ntohs@plt>:
│    4021f0:    ff 25 aa 70 20 00       jmpq   *0x2070aa(%rip)        
│    4021f6:    68 51 00 00 00          pushq  $0x51
│ -  4021fb:    e9 d0 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  4021fb:    e9 d0 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402200 <sigaddset@plt>:
│    402200:    ff 25 a2 70 20 00       jmpq   *0x2070a2(%rip)        
│    402206:    68 52 00 00 00          pushq  $0x52
│ -  40220b:    e9 c0 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40220b:    e9 c0 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402210 <crypt_load@plt>:
│    402210:    ff 25 9a 70 20 00       jmpq   *0x20709a(%rip)        
│    402216:    68 53 00 00 00          pushq  $0x53
│ -  40221b:    e9 b0 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40221b:    e9 b0 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402220 <crypt_set_rng_type@plt>:
│    402220:    ff 25 92 70 20 00       jmpq   *0x207092(%rip)        
│    402226:    68 54 00 00 00          pushq  $0x54
│ -  40222b:    e9 a0 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40222b:    e9 a0 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402230 <sigaction@plt>:
│    402230:    ff 25 8a 70 20 00       jmpq   *0x20708a(%rip)        
│    402236:    68 55 00 00 00          pushq  $0x55
│ -  40223b:    e9 90 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40223b:    e9 90 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402240 <memcpy@plt>:
│    402240:    ff 25 82 70 20 00       jmpq   *0x207082(%rip)        
│    402246:    68 56 00 00 00          pushq  $0x56
│ -  40224b:    e9 80 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40224b:    e9 80 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402250 <strlen@plt>:
│    402250:    ff 25 7a 70 20 00       jmpq   *0x20707a(%rip)        
│    402256:    68 57 00 00 00          pushq  $0x57
│ -  40225b:    e9 70 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40225b:    e9 70 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402260 <crypt_get_dir@plt>:
│    402260:    ff 25 72 70 20 00       jmpq   *0x207072(%rip)        
│    402266:    68 58 00 00 00          pushq  $0x58
│ -  40226b:    e9 60 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40226b:    e9 60 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
│  
│  0000000000402270 <getline@plt>:
│    402270:    ff 25 6a 70 20 00       jmpq   *0x20706a(%rip)        
│    402276:    68 59 00 00 00          pushq  $0x59
│ -  40227b:    e9 50 fa ff ff          jmpq   401cd0 <_init@@Base+0x18>
│ +  40227b:    e9 50 fa ff ff          jmpq   401cd0 <_init@@Base+0x10>
├── objdump --line-numbers --disassemble --demangle --reloc --section=.text {}
│ @@ -1625,15 +1625,15 @@
│    403e04:    48 8d 35 fd 4f 20 00    lea    0x204ffd(%rip),%rsi        
│    403e0b:    48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
│    403e0f:    e8 00 00 00 00          callq  403e14 <getline@plt+0x1ba4>
│    403e14:    48 8b 37                mov    (%rdi),%rsi
│    403e17:    48 8d 57 08             lea    0x8(%rdi),%rdx
│    403e1b:    45 31 c9                xor    %r9d,%r9d
│    403e1e:    41 b8 71 67 40 00       mov    $0x406771,%r8d
│ -  403e24:    b9 b8 1c 40 00          mov    $0x401cb8,%ecx
│ +  403e24:    b9 c0 1c 40 00          mov    $0x401cc0,%ecx
│    403e29:    bf 80 22 40 00          mov    $0x402280,%edi
│    403e2e:    e9 fd e0 ff ff          jmpq   401f30 <__libc_start_main@plt>
│    403e33:    66 2e 0f 1f 84 00 00    nopw   %cs:0x0(%rax,%rax,1)
│    403e3a:    00 00 00 
│    403e3d:    0f 1f 00                nopl   (%rax)
│    403e40:    b8 10 ec 60 00          mov    $0x60ec10,%eax
│    403e45:    48 3d 10 ec 60 00       cmp    $0x60ec10,%rax
│ @@ -1688,15 +1688,15 @@
│    403f06:    48 39 d8                cmp    %rbx,%rax
│    403f09:    72 e5                   jb     403ef0 <getline@plt+0x1c80>
│    403f0b:    e8 30 ff ff ff          callq  403e40 <getline@plt+0x1bd0>
│    403f10:    b8 00 00 00 00          mov    $0x0,%eax
│    403f15:    48 85 c0                test   %rax,%rax
│    403f18:    74 0a                   je     403f24 <getline@plt+0x1cb4>
│    403f1a:    bf 90 82 40 00          mov    $0x408290,%edi
│ -  403f1f:    e8 dc c0 bf ff          callq  0 <_init@@Base-0x401cb8>
│ +  403f1f:    e8 dc c0 bf ff          callq  0 <_init@@Base-0x401cc0>
│    403f24:    5b                      pop    %rbx
│    403f25:    41 5c                   pop    %r12
│    403f27:    c6 05 f2 ac 20 00 01    movb   $0x1,0x20acf2(%rip)        
│    403f2e:    5d                      pop    %rbp
│    403f2f:    c3                      retq   
│    403f30:    c3                      retq   
│    403f31:    66 66 2e 0f 1f 84 00    data16 nopw %cs:0x0(%rax,%rax,1)
│ @@ -1705,15 +1705,15 @@
│    403f40:    b8 00 00 00 00          mov    $0x0,%eax
│    403f45:    48 85 c0                test   %rax,%rax
│    403f48:    74 1e                   je     403f68 <getline@plt+0x1cf8>
│    403f4a:    55                      push   %rbp
│    403f4b:    be 40 ec 60 00          mov    $0x60ec40,%esi
│    403f50:    bf 90 82 40 00          mov    $0x408290,%edi
│    403f55:    48 89 e5                mov    %rsp,%rbp
│ -  403f58:    e8 a3 c0 bf ff          callq  0 <_init@@Base-0x401cb8>
│ +  403f58:    e8 a3 c0 bf ff          callq  0 <_init@@Base-0x401cc0>
│    403f5d:    5d                      pop    %rbp
│    403f5e:    e9 0d ff ff ff          jmpq   403e70 <getline@plt+0x1c00>
│    403f63:    0f 1f 44 00 00          nopl   0x0(%rax,%rax,1)
│    403f68:    e9 03 ff ff ff          jmpq   403e70 <getline@plt+0x1c00>
│    403f6d:    0f 1f 00                nopl   (%rax)
│    403f70:    41 57                   push   %r15
│    403f72:    31 c0                   xor    %eax,%eax

libblkid.so.1 difference between 2 debian-10 hosts with different paths of builds is too big to output here.

busybox difference between 2 debian-10 hosts with different paths of builds is too big to output here.

mainly, time of compilation is injected, all address of code is shifted, including entry point.

@osresearch @flammit : any troubleshooting insights would be amazing for the community. Otherwise I'll try to fix when I have time...

@tlaurion I don't know how precisely CI looks like, but you know my rambling about reproducible builds. The only feasible paths are Nix/Guix related and having reproducible build environment. This moves very slow because everyone is short on time. Anyway Dasharo has this on roadmap and have to address the issue.I gathered some thoughts to have one point to which I can reference people when reproducibility problem arise: https://docs.dasharo.com/osf-trolling-list/build_process/

@tlaurion I don't know how precisely CI looks like, but you know my rambling about reproducible builds. The only feasible paths are Nix/Guix related and having reproducible build environment. This moves very slow because everyone is short on time. Anyway Dasharo has this on roadmap and have to address the issue.I gathered some thoughts to have one point to which I can reference people when reproducibility problem arise: https://docs.dasharo.com/osf-trolling-list/build_process/

@pietrushnic This seems to exactly be the problem here, with cryptsetup and busybox being reproducible in the past but having broke reproducibility recently, where I do not quite understand without further digging. What happened and why RPATHs are now introduced in the cryptsetup binaries and libs, where they weren't before :

│ - 0x000000000000000f (RPATH)              Library rpath: [//lib:/root/project/install/lib]
│ + 0x000000000000000f (RPATH)              Library rpath: [//lib:/home/user/heads/install/lib]

As for coreboot layer, those are now again reproducible (was not in the past, relaying on musl-cross-make, now relaying on coreboot buildstack per coreboot version) which is now "pinned" per version of coreboot; specified in the board configuration.

So yeah. We have all libraries and binaries outside of coreboot being built against musl-cross-make and local host toolchain. Seems like host toolchain introduced changes.... again.

CI is over here based on current CircleCI config:

Pulls debian-10 docker
installs dedian-10 host tools: build-essential zlib1g-dev uuid-dev libdigest-sha-perl libelf-dev bc bzip2 bison flex git gnupg iasl m4 nasm patch python python2 python3 wget gnat cpio ccache pkg-config cmake libusb-1.0-0-dev autoconf texinfo ncurses-dev doxygen graphviz udev libudev1 libudev-dev automake libtool rsync innoextract
Starts heads build process per board (while reusing cache if existing, out of scope here)
- builds make and gawk if version discreteness, then musl-cross-make. All other modules depends on CROSS_COMPILE which definitions are under global Makefile
- Starts building coreboot, including its buildstack first from config in board config, where coreboot depends on its own, version specific, buildstack
- Builds linux headers (headers installs) with musl-cross-make (which configure depends on host tools)
- Builds board specific requirements (totp-tools, cryptsetup, gpg2, etc) from muls-cross-make and leaked configure dependencies
- Wraps previous dev.cpio, tools.cpio, heads.cpio into initrd.cpio.xz; builds coreboot rom.

Notes on resolving reproducibility issues, just because some questions were asked over slack and wanted to keep track

rpath patches/buildtweaks that were applied to other libs and binaries in the past

Yes, I know @felixsinger works on that. Please note there will be stable toolchains of coreboot, based on recent decisions of coreboot leadership. More to that there would be stability checks also. So things going in very good direction. Please check notes: https://docs.google.com/document/d/1NRXqXcLBp5pFkHiJbrLdv3Spqh1Hu086HYkKrgKjeDQ/edit?usp=sharing

Yes, I know @felixsinger works on that. Please note there will be stable toolchains of coreboot, based on recent decisions of coreboot leadership. More to that there would be stability checks also. So things going in very good direction. Please check notes: https://docs.google.com/document/d/1NRXqXcLBp5pFkHiJbrLdv3Spqh1Hu086HYkKrgKjeDQ/edit?usp=sharing

For anyone else interested: see notes from 30 June 2021 forward

Redid the test as part of passing O2->Os (in the goal of reducing modules footprint at the cost of performance)

The list of non-reproducible binaries grows.

Here is the small script I came up with, having a local board build under ~/heads while having dumped CircleCI hashes.txt for the same board under a local directory (/tmp/circleci_cryptsetup_for_all here): rm -f ./output_non_reproducible ./identical_hashes ./bad_hashes; grep / hashes.txt | awk -F " " '{print $1}'| while read hash; do grep -q $hash ~/heads/build/x230-hotp-maximized/hashes.txt; if [ $? == 0 ]; then echo $hash >> ./identical_hashes; else echo $hash >> ./bad_hashes;fi;done; cat ./bad_hashes| while read bad; do grep $bad hashes.txt >> ./output_non_reproducible; done;

cat ./output_non_reproducible shows:

b52b5c56ddd870a88c5fd58cc78259afa85b5a9dfad3d86f5df935d1dd668c3c  /root/project/build/x230-hotp-maximized/tools.cpio
0926dde50a4ded3286f445ed515f3c9e27771022b153fe63cb8c3be597cd0391  ./lib/libcryptsetup.so.12
dcb42777d0ab5ee22be9abe9b834c20ea12cfd5ee1d493b664e7c3d1a56308d6  ./lib/libjson-c.so.5
4e27c1681daf9882c05d7a8f1c617593ee0e100cf4aad1e55f1833dd91b44545  ./lib/libgcrypt.so.20
5cee77d4ae3ee96e6a6b6856302449ffad4ff6691414946b53f71fa614e93345  ./lib/libtpm.so
d648d4b6710774bc16b8d50183614061d4bb22eace15c97b99853e4747006a13  ./lib/libuuid.so.1
f521557a2616a2fccd8738827526bdc77aade773a3b24ba627543ef0e4c8af1d  ./lib/libblkid.so.1
d634a36cc7cf24d7b201c385c1fecf0d0faa39761944c3398e671fc208bc06fd  ./bin/busybox
d9858fcbebd3ceb34473f3d05dbd4be4f745f36276516434ece24f1fccc5a387  ./bin/totp
e6d46749fe3bd9af195d614f867f4bade0d9a32505635e06cdf574a1ae635fe1  ./bin/hotp
1e285a68eb9e978bea867e87fabbc1e1288d78495cd8e0f938a16330c81c9f2e  ./bin/base32
d1204ecf1df92d7feadfc47f13be1f6c9d8a8a78dd22cda3f8668c145121bf2b  ./bin/qrenc
ca39155972ec33a70cf0cbaca56486a003a0e01be22595d1bbf9aff6aeeb22af  ./bin/tpm
1e117685faf7b1b28dc43b538d45af3f1e95fe8dbaaa41fbff23b9a84b264cbc  ./bin/cryptsetup
7f9b14e92370c3a26688e7372cdd59e8dba9c6db460f9486c45cdcd6c291c445  ./bin/cryptsetup-reencrypt
86feb33089789978b41a58007b603c7aef7309a64986cac38a84e648fdd81875  ./bin/veritysetup
efc02e76ba8fabbe47dbfd7ae86a7f4026b3f12cbdd44272fbe5241823311602  ./bin/dmsetup
d6eb80e207a1397f0d7185692942e9e3e12ddd91ae2b6f35e3fb469869ec24fa  ./bin/lvm
da4f0c6eea7861f3c5709a9eb066a34db2d2114f993f43276731601d753d2f29  ./bin/fbwhiptail
5178f615a6a5e1604061a4d7a599a1590b1ed51c151fec8dfca004293d08bb7c  build/x230-hotp-maximized/initrd.cpio.xz

Where the following pinpoints to the possible modules being responsible to produce such binaries: cat output_non_reproducible | awk -F "/" '{print $NF}'| while read bin; do grep $bin -R ~/heads/modules|awk -F ":" '{print $1}'; done|sort|uniq

Which produces the following (not completely valid) list:

/home/user/heads/modules/busybox
/home/user/heads/modules/coreboot
/home/user/heads/modules/cryptsetup
/home/user/heads/modules/cryptsetup2
/home/user/heads/modules/fbwhiptail
/home/user/heads/modules/gawk
/home/user/heads/modules/hidapi
/home/user/heads/modules/hotp-verification
/home/user/heads/modules/json-c
/home/user/heads/modules/libgcrypt
/home/user/heads/modules/linux
/home/user/heads/modules/linuxboot
/home/user/heads/modules/lvm2
/home/user/heads/modules/qrencode
/home/user/heads/modules/tpmtotp
/home/user/heads/modules/util-linux

The curated version of output_non_reproducible file produces following modules now being non-reproducible modules: user@localhost:/tmp/circleci_cryptsetup_for_all$ cat output_non_reproducible | awk -F "/" '{print $NF}'| while read bin; do grep $bin -R ~/heads/modules|awk -F ":" '{print $1}'; done|sort|uniq|awk -F "/" '{print $NF}'

busybox
cryptsetup
cryptsetup2
fbwhiptail
hidapi
hotp-verification
json-c
libgcrypt
lvm2
qrencode
tpmtotp
util-linux

@jans23 @mrchromebox : do we do plan together to resolve this sharing knowledge we have in the area to bring back reproducible builds?

~I may have access to grant money to fix this and would love to know who is the most knowledgeable to fix those~

And would also gladly and proactively document such process in the contribute section and also learn my way with such paid mentorship.

Maybe we could look at hotp-verification but currently no time for others, unfortunately. :-(

One technique I've found useful is to have a unique string in the build path and then grep through the various outputs for it. This found that json-c was including some paths for the musl headers, and that in turn causes cryptsetup2 to be irreproducible.

cryptsetup1 ignores the --disable-rpath flag to configure, so it burns rpaths into the binaries. cryptsetup2's patch forcibly removes that code, so it would need to be back ported.

busybox should be fixed by #74 or by adding -DBB_EXTRA_VERSION='"none"' to the build flags so that it doesn't use the autoconf timestamp.

haven't gotten to the others yet

One of my thoughts for ensuring reproducibility is to have two CI runners with different base OS images and require that they produce the same firmware images for any commits. That will double the CPU required, although it is important to catch these early rather than after things have bit-rotted or when someone asks "hey, when did this break?".

A few other important things:

Linux must have an initrd provided, or else it will generate a irreproducible one from the host system's /dev/console
Linux 4.x seems to include some full paths, so it is necessary to pass -ffile-prefix-map=.... to filter them out.
The cross compilers also need to have their paths rewritten; this was causing issues with cairo
zlib tries to detect cross compilers and ends up breaking some of the tools; only built the libraries to avoid it
coreboot's .config file can end up with absolute paths for the bzimage and initrd, which will then mess up the hashes if CONFIG_INCLUDE_CONFIG_FILE=y is set since the .config will be included in the ROM.

Anything using wild cards is suspect... it looks like lvm2 might have a problem in its makefile that globs the exported symbols: https://github.com/osresearch/linux-builder/issues/6

gnu make has sorted, then unsorted, and then sorted again: https://savannah.gnu.org/bugs/index.php?52076

As of today, https://github.com/osresearch/heads/pull/1367 fixed json-c.

Here are the current reproducibility issues between local and CircleCI builds (both debian-11 based) at the moment of merging that PR (when using fbwhiptail):

user@heads-tests:/tmp$ HASHES1=~/heads/build/x86/x230-hotp-maximized/hashes.txt
user@heads-tests:/tmp$ HASHES2=/tmp/hashes.txt
user@heads-tests:/tmp$ egrep '^[0-9a-f]{64}' $HASHES1 | while read line; do HASH_REF=$(echo $line|awk -F " " {'print $1'}); FILE_REF=$(echo $line|awk -F "/" {'print $NF'}); if ! grep -q "$HASH_REF" $HASHES2; then echo "$FILE_REF doesn't match";fi; done
tools.cpio doesn't match
libmbedcrypto.so.0 doesn't match
libgcrypt.so.20 doesn't match
fbwhiptail doesn't match
kexec doesn't match
busybox doesn't match
bash doesn't match
initrd.cpio.xz doesn't match
heads-x230-hotp-maximized-v0.2.0-1508-g46aa253-bottom.rom doesn't match
heads-x230-hotp-maximized-v0.2.0-1508-g46aa253.rom doesn't match

So the following heads modules needs work/patching:

mbedtls
libgcrypt
fbwhiptail
kexec
busybox
bash

Which causes tools.cpio (where linux libaries and binaries are packed) and initrd.cpio.xz (where tools.cpio, modules.cpio and heads.cpio are compressed) to be non-reproducible and consequently the final rom.

Thanks for excellent writeup @tlaurion !

Is libgcrypt any better after https://github.com/osresearch/heads/pull/1350 ? I tried to test it myself, but it seems I am not even able to install Debian 11 due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983357 and https://bugzilla.kernel.org/show_bug.cgi?id=207695 - too much yak shaving for me at the moment...

So, I have managed to overcome my debian 11 issues and it seems that I was able to reproduce libgcrypt 1.10.1 build from CI on a freshly deployed Debian 11 amd64 virtual machine:

$ find . -name libgcrypt* | while read X; do sha256sum "$X"; done
5cabab0f0523fae5a6e50adbf217bc3d67ade4efde0f4d6a828189d1942f19d4  ./build-unpack/lib/libgcrypt.so.20
5cabab0f0523fae5a6e50adbf217bc3d67ade4efde0f4d6a828189d1942f19d4  ./ci-unpack/lib/libgcrypt.so.20

linuxboot / heads-wiki

Document how to investigate reproducibility issues #70