search

MyHush / hush

Hush is a fork of Zcash focused on secure communications
https://myhush.org
Other
67 stars 37 forks source link

Reproducible/Deterministic Builds #60

Closed leto closed 6 years ago

leto commented 6 years ago

Describe the issue

No deterministic builds

Can you reliably reproduce the issue?

Yes

If so, please list the steps to reproduce below:

  1. Compile and observe,
  2. No
  3. Reproducibility

Bounty

500HUSH

Expected behaviour

Since Bitcoin Core and Zcash have reproducible builds, we should as well. It increases the security of our entire ecosystem.

Actual behaviour + errors

We do not have Gitian setup

The version of Hush you were using:

All

Any extra information that might be useful in the debugging process.

I like tuhtles

oDinZu commented 6 years ago

@leto, @madbuda, @radix42 and peoples of the world I have been working on some personal projects for Hush. I am not ready to share quite yet. I am excited, but it is simply not ready and takes tedious patience. In the mean time, I felt this should be shared to help others understand more or what not.

I recommend using vagrant and gitian =) Also works nicely with zcash. Zcash also uses Ansible @leto, should we consider using this too? I am not quite sure if it is needed or not.

image

Also -- this image may not be 200% correct, once the entire idea is ready to be reviewed I will let the staff check it out first to make corrections or w/e is best for the team and Hush.

Note: The massive in your face copyright is because this was submitted to public already via the instructor for college individual project and the individual doesn't like me much since I am open source with my approach or 'odd' approach -- I will leave it at that. I don't want history to repeat itself. I put a lot of hard work into this to help me understand better and also share with others to catch up to speed a bit quicker to develop for hush <3.

Note 2: Gitian and git are a bit backwards, but overall this image should help give a good map in our noggin to help catch up to speed. The project and info I share is still in beta so to speak.

leto commented 6 years ago

To be more clear, this bounty is about porting repro builds from BTC/ZEC to Hush, there is prior art to follow: https://github.com/zcash/zcash/blob/4ee9d712b573bccd36e29134d96eaad54182ebab/contrib/gitian-descriptors/README.md

kentsommer commented 6 years ago

@csharpee

Are you currently working on this (setting up reproducible builds)? If you are, I won't work on this. However, if you aren't currently working on this task, I'm willing to work on it.

oDinZu commented 6 years ago

Okay - beautiful, yeah I am not working on this at the moment.

kentsommer commented 6 years ago

I've got builds up and running! If someone could test with the current configuration (uses my fork of hush) that would be great!

The instructions for how to setup your host can be found in the hush-gitian repo. Follow them exactly and you should be good to go! Once this is tested, I'll go ahead and PR my hush fork and someone else will have to pull in the two other repos.

Main repos:

  1. https://github.com/kentsommer/hush-gitian
  2. https://github.com/kentsommer/hush
  3. https://github.com/kentsommer/gitian.sigs
radix42 commented 6 years ago

Great! I'll give it a whirl soon, I may have to update some stuff on my Ubuntu box tho (haven't done zcash deterministic builds on it for example in months and months)

oDinZu commented 6 years ago

I have tested it on Ubuntu 16.04. It built successfully! I started to do this on Amazon EC2 instance and ran into the VT-X error. So I installed via locally on my ol' System76.

vagrant@hush-build:~$ ./gitian-build.sh --verify
v1.0.12-dev
~/hush ~
HEAD is now at 28fb18f... More reference fixes
~
~/gitian-builder ~

Verifying v1.0.12-dev Linux

gpg: Signature made Fri 03 Nov 2017 01:01:22 AM GMT
gpg:                using RSA key #####
gpg: Good signature from "Kent Sommer <#####>"
###6D: OK
~
vagrant@hush-build:~$
leto commented 6 years ago

This looks really exciting, looking forward to testing it out!

leto commented 6 years ago

@kentsommer I forked the 2 repos into MyHush org

I am currently testing with ansible 2.2.1.0 and vagrant 1.9.1 on OS X and I successfully created the Gitian VM, it's currently building :+1:

kentsommer commented 6 years ago

@radix42, @leto

I've submitted #65 with a few notes on final things that need to be changed to get this fully up and running.

leto commented 6 years ago

I got really close:

Generating report
5ad058059f14549aca764b9000f5cd6eba53ed98ab3eeb136c87ddbac7606a81  hush-1.0.12-linux64-debug.tar.gz
b1e4af6d9af4119884214f99cf622d2da802c2ff368c75074282236e34c4d1b2  hush-1.0.12-linux64.tar.gz
e5645066615fd9532797870555638cdddd49dc8fe47bc9b716d7ff66c32d3090  src/hush-1.0.12.tar.gz
8d5dd4418b254a919dc4ddc3a9533fc1992ac9d418802729485d638e3687d861  hush-1.0.12-res.yml
Done.
gpg: skipped "F16219F4C23F91112E9C734A8DFCBF8E5A4D8019": No secret key
gpg: signing failed: No secret key
./bin/gsign:11:in `system!': failed to run gpg2 --detach-sign -u "F16219F4C23F91112E9C734A8DFCBF8E5A4D8019" "../gitian.sigs/v1.0.12-dev/F16219F4C23F91112E9C734A8DFCBF8E5A4D8019/hush-1.0.12-build.assert" (RuntimeError)
    from ./bin/gsign:93:in `<main>'
~

Committing v1.0.12-dev Signatures

~/gitian.sigs ~
[master b862538] Add v1.0.12-dev signatures for F16219F4C23F91112E9C734A8DFCBF8E5A4D8019
 1 file changed, 385 insertions(+)
 create mode 100644 v1.0.12-dev/F16219F4C23F91112E9C734A8DFCBF8E5A4D8019/hush-1.0.12-build.assert
~

I think this is related to the fact that I merely specified my keybase pukey ID, but it could not be found locally, perhaps because I am supposed to import it first? @kentsommer

kentsommer commented 6 years ago

@leto

If you did not specify the key ID here then you are correct, you would have to manually import your GPG key to the VM.

leto commented 6 years ago

I ran curl https://keybase.io/dukeleto/key.asc | gpg --import to import the correct key onto the machine that is the host for the hush-build VM, and now things are chugging along again...

leto commented 6 years ago

@kentsommer thanks, I was using the wrong key, gpg_key_name instead of gpg_key_id, I think that will hopefully fix it and then I will push my assert files. Thanks for the help!

radix42 commented 6 years ago

do we have a baby steps, one thing at t time doc for setting up gitian builds? I dunno if the system I was using for it is up right now, and it needs updating in any case from when i did zcash gitian builds on it

On Sun, Nov 5, 2017 at 7:58 PM, Duke Leto notifications@github.com wrote:

I ran curl https://keybase.io/dukeleto/key.asc | gpg --import to import the correct key onto the machine that is the host for the hush-build VM, and now things are chugging along again...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342033936, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0Ba_Mr9LRhDOwVjyYJ6FhykxXFOmks5sznXGgaJpZM4QEvn4 .

kentsommer commented 6 years ago

@radix42

If I understand you correctly, what you are looking for can be found in the readme here: https://github.com/MyHush/hush-gitian

radix42 commented 6 years ago

awesome thank you, was wondering if that existed :-)

On Sun, Nov 5, 2017 at 8:04 PM, Kent Sommer notifications@github.com wrote:

@radix42 https://github.com/radix42

If I understand you correctly, what you are looking for can be found in the readme here: https://github.com/MyHush/hush-gitian

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342034756, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0ELAHFrso2SMCoSVx4YftVtmzqd1ks5sznc_gaJpZM4QEvn4 .

radix42 commented 6 years ago

I will work on getting that box up tomorrow, I have not enough light to work on it at night (yes, I know, I am like a hermit in a cave here, lol), and I'll update all the versions of stuff on it.

On Sun, Nov 5, 2017 at 8:05 PM, David Mercer radix42@gmail.com wrote:

awesome thank you, was wondering if that existed :-)

On Sun, Nov 5, 2017 at 8:04 PM, Kent Sommer notifications@github.com wrote:

@radix42 https://github.com/radix42

If I understand you correctly, what you are looking for can be found in the readme here: https://github.com/MyHush/hush-gitian

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342034756, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0ELAHFrso2SMCoSVx4YftVtmzqd1ks5sznc_gaJpZM4QEvn4 .

leto commented 6 years ago

This could be something related to my host GPG being gpg (GnuPG/MacGPG2) 2.0.14 while the hush-build VM has gpg (GnuPG) 2.0.26. I still get gpg: signing failed: No secret key. I am using gpg_key_id: '5A4D8019' in my gitian.yml

radix42 commented 6 years ago

ugh you need to actually import your pgp private key i think to be able to sign stuff, into...which layer of VM, Kent? did you get it to work i can't recall what I did with zcash gitian it was a PITA with perms

On Sun, Nov 5, 2017 at 9:10 PM, Duke Leto notifications@github.com wrote:

This could be something related to my host GPG being gpg (GnuPG/MacGPG2) 2.0.14 while the hush-build VM has gpg (GnuPG) 2.0.26. I still get gpg: signing failed: No secret key. I am using gpg_key_id: '5A4D8019' in my gitian.yml

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342041939, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0ORJF6rgN2MSnPGGjeA2aj1FK5ksks5szobPgaJpZM4QEvn4 .

kentsommer commented 6 years ago

@leto @radix42

Yes I've got importing and signing to work. I doubt it is an issue with GPG version mismatch @leto. Did you rerun the vagrant up --provision hush-build command after adding the gpg_key_id field and verify that Ansible is showing the key was imported correctly?

Unfortunately I don't have an OSX machine available otherwise I would test it for you.

Correct output should be this (you should be prompted for your GPG key passphrase):

 __________________________________________________________
/ TASK [gitian : Import the GPG private key to the Vagrant \
\ user.]                                                   /
 ----------------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

ok: [hush-build] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
 ___________________________________________________
< TASK [gitian : Clean up secret key file in /tmp.] >
 ---------------------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

changed: [hush-build] => {"changed": true, "failed": false, "path": "/tmp/04C73A7FD927976D.sec", "state": "absent"}
leto commented 6 years ago

when I run vagrant up --provision hush-build i finally get:

fatal: [hush-build -> localhost]: FAILED! => {"changed": true, "cmd": ["gpg2", "--list-secret-keys", "|", "grep", "5A4D8019"], "delta": "0:00:00.140504", "end": "2017-11-05 19:53:40.453874", "failed": true, "rc": 2, "start": "2017-11-05 19:53:40.313370", "stderr": "gpg: error reading key: No secret key", "stdout": "", "stdout_lines": [], "warnings": []}
 ____________
< PLAY RECAP >
 ------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

hush-build                 : ok=53   changed=21   unreachable=0    failed=1****

with this gitian.yml:

---
- name: Apply the Hush Gitian builder role.
  become: yes
  hosts: localhost:hush-build
  vars:
    #hush_git_repo_url: https://github.com/MyHush/hush
    hush_git_repo_url: https://github.com/kentsommer/hush
    hush_version: v1.0.12-dev
    gpg_key_name: '5A4D8019'
    #gpg_key_name: 'F16219F4C23F91112E9C734A8DFCBF8E5A4D8019'
    git_name: 'Duke Leto'
    git_email: 'duke@leto.net'
    # optional - auto imports private key necessary for signing
    gpg_key_id: '5A4D8019'
    # optional
    ssh_key_name: ''
  roles:
    - role: common
      tags: common
    - role: gitian
      tags: gitian

It seems that only my public key was imported and it does not even try to prompt for a password, because it can't find the secret key

kentsommer commented 6 years ago

@leto

Do you have any output when you run gpg2 --list-secret-keys on your host machine?

kent@fenix:~$ gpg2 --list-secret-keys
/home/kent/.gnupg/pubring.gpg
-----------------------------
sec   rsa4096/XXXXXXXX 2017-11-01 [SC]
uid         [ultimate] Kent Sommer <kent.sommer13@gmail.com>
ssb   rsa4096/XXXXXXXX 2017-11-01 [E]

If so, run again with gpg2 --list-secret-keys --keyid-format LONG and copy the ID after the slash on the line starting with sec and use that in the gpg_key_id and gpg_key_name fields in the gitian.yml file.

The entry in gpg_key_id should be of length 16.

If the above works then I will edit the instructions and ansible scripts appropriately.

leto commented 6 years ago

@kentsommer thanks for your help, I finally got my Keybase secret key imported to my host gpg correctly, my provision was all green and happy, and now I am running a plain ./gitian-build.sh again

kentsommer commented 6 years ago

@leto

Awesome! Let me know if there are anymore issues :+1:

leto commented 6 years ago

@kentsommer please look this over and let me know if it seems OK: https://github.com/MyHush/gitian.sigs/commit/52a1bbe3f7257d4cefa0214a3d430c7b521290bc

Thanks again, this is super awesome stuff 💯

kentsommer commented 6 years ago

@leto

Those hashes match up with mine. Everything else looks good as well!

radix42 commented 6 years ago

W00t, so awesome, your assets match

peoples commented 6 years ago

Can yinz please remove me from this thread? It all sounds impressive, but I have no idea what it means. Yet it's filling up my inbox.

Please and thank you.

dngbarnes

From: David Mercer notifications@github.com Sent: Monday, November 6, 2017 1:58 AM To: MyHush/hush Cc: peoples; Mention Subject: Re: [MyHush/hush] Reproducible/Deterministic Builds (#60)

W00t, so awesome, your assets match

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/MyHush/hush/issues/60#issuecomment-342072201, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AHDCqzYT2gMhSk1spV06YYZ6bnpe3e_Yks5szrxBgaJpZM4QEvn4.

128514 commented 6 years ago

Lol, from @csharpee peoples of the world comment. Someone will remove you.

lludlow commented 6 years ago

@peoples github notifications are managed by the users, if you are getting notices then you either subscribed or have been mentioned. You would need to unsubscribe

leto commented 6 years ago

@radix42 my vote is that @kentsommer has earned his 500HUSH, his documentation made it very easy to get things setup locally and he was extremely responsive in helping this newb get stuff working

radix42 commented 6 years ago

I agree, your assets all matched.....i was gonna wait for a third set that matched, but I am too sick to crawl around under my desk and fix my vagrant box, so @kentsommer what HUSH addy does the bounty go to? email, PM me or post a taddr here if you'd like!

when i get my vagrant machine up i wanna setup deterministic build for the Windows version too, as it is cross compiled on linux and just has some additional dependencies for the build

On Mon, Nov 6, 2017 at 10:05 AM, Duke Leto notifications@github.com wrote:

@radix42 https://github.com/radix42 my vote is that @kentsommer https://github.com/kentsommer has earned his 500HUSH, his documentation made it very easy to get things setup locally and he was extremely responsive in helping this newb get stuff working

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342215737, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0GvFcloZqcKKiVjQfV2THfMbX1rEks5szzxMgaJpZM4QEvn4 .

radix42 commented 6 years ago

also @leto I have been trying to discord, slack, signal, call or txt you the last 30+ minutes and you seem offline aside from the above github reply

On Mon, Nov 6, 2017 at 10:08 AM, David Mercer radix42@gmail.com wrote:

I agree, your assets all matched.....i was gonna wait for a third set that matched, but I am too sick to crawl around under my desk and fix my vagrant box, so @kentsommer what HUSH addy does the bounty go to? email, PM me or post a taddr here if you'd like!

when i get my vagrant machine up i wanna setup deterministic build for the Windows version too, as it is cross compiled on linux and just has some additional dependencies for the build

On Mon, Nov 6, 2017 at 10:05 AM, Duke Leto notifications@github.com wrote:

@radix42 https://github.com/radix42 my vote is that @kentsommer https://github.com/kentsommer has earned his 500HUSH, his documentation made it very easy to get things setup locally and he was extremely responsive in helping this newb get stuff working

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342215737, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0GvFcloZqcKKiVjQfV2THfMbX1rEks5szzxMgaJpZM4QEvn4 .

kentsommer commented 6 years ago

@radix42

I'm happy to wait for a third set if you would like? My address is here, but feel free to hold off sending until you verify a third set. Would prefer you are confident before sending :+1:

hush-addr: t1JjmKWun4jn61JgVSK1fQKifVToqhKsov3

radix42 commented 6 years ago

cool, my vagrant box needs a bit more attention than I thought so its not up yet (ETOOMUCHCATHAIR in its vents)

On Mon, Nov 6, 2017 at 4:52 PM, Kent Sommer notifications@github.com wrote:

@radix42 https://github.com/radix42

I'm happy to wait for a third set if you would like? My address is here, but feel free to hold off sending until you verify a third set. Would prefer you are confident before sending 👍

hush-addr: t1JjmKWun4jn61JgVSK1fQKifVToqhKsov3

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MyHush/hush/issues/60#issuecomment-342327724, or mute the thread https://github.com/notifications/unsubscribe-auth/AF9e0GJ0FyfZLqOCJjJyJHr3Vw9-NzZkks5sz5vRgaJpZM4QEvn4 .

radix42 commented 6 years ago

Bounty happily paid! https://explorer.myhush.org/tx/0e086a21558875b25db2eb60e39e9754126b4d867426fe9f2063d64458f41678