Document supply chain

[alokmenghrajani]

The docs don't explain what kinds of supply chain attacks we are worried about, what we are doing to mitigate them, what risks we need to assume, etc.

There's also many ways to approach this problem, and it might be worth clarifying our take / vision. For example, we believe that being able to reproduce builds is more useful than having a few people perform a clean build.

[kousu]

Thank you for thinking carefully through your risks. What are your thoughts on the elephant that's in the room?

[mcpherrinm]

Subzero is designed to be run on machines which are airgapped: The only connection in and out of the system is via QR codes. Even if the machine contained a bloomberg-proposed implant, it would not have a way of getting data out.

Additionally, Subzero is designed to minimally trust its host computer: The private key, policy checking, and transaction signing all happens in an HSM, loading code which is signed and cannot be modified by the computer the HSM is in.

[alokmenghrajani]

(Ignoring the fact that the companies impacted by these recent news events have acknowledged not being affected, this is a very good question and exactly the kind of things I would like to discuss in more details.)

Generally, the HSMs provides a good defense against the host machine being compromised. The wallet private keys are created inside the HSMs and are never exported. We trust our HSMs because they are subject to various certifications; some of those cover the security of the supply chain.

We have to trust the host machine to do things like display accurate information. This information is also available on other machines (e.g. the QR codes can be decoded on any laptop or mobile device).

Finally, the whole setup is offline.

There's also nothing preventing people from running their cold wallet on a mix of different hardware.

[alokmenghrajani]

I have a draft discussing the supply chain aspect of things. Next step is to clean it up and add it to our docs.

[bosswissam]

@alokmenghrajani looking forward to this - happy to serve as a guinea pig.

Looking at current docs - I'm assuming the iso resulting from https://github.com/square/subzero/tree/master/live-usb-creator#building-image is what you want on the DVD, but the "Writing image to USB drive" section is there just as another option?

Also noticed the use of md5 (https://github.com/square/subzero/tree/master/live-usb-creator#build-process) - knowing it's weaknesses, do you think it's sufficient for the purposes?

[bosswissam]

Ah also based on https://github.com/square/subzero/tree/master/live-usb-creator#building-image - the DVD/USB used should be at least 13gb in size? Might be worth adding to https://subzero.readthedocs.io/en/master/physical_components/#physical-components.

Hopefully my comments are helpful guys! This is an awesome project!

[alokmenghrajani]

I never got around to discussing the supply chain issue in more depth. Now that we have nearly 2 years of experience running this project, we can probably think about supply chain from various different angles. Here are some thoughts:

• The wallet being completely offline, with only QR codes as the communication channel, gives a ton of operational safety. My only concern at this point is covert signature channels, which we know we can't 100% fix but we also know how to reduce risks there.
• The HSM's ability to split admin credentials among multiple people and expose keys encrypted under these admin credentials gives us confidence that we can recover our sites in the event of a disaster.
• It is easier to discuss the supply chain security of the code running on the HSM and the code running on the server separately. This separation is not technically correct since we can never be 100% certain that the outer layer isn't lying to us and the code is actually running on the physical device. The HSM unfortunately neither provides a secure input nor a secure output.
• For the code running on the HSM:
  • The build is reproducible. Our latest release was successfully built by multiple engineers, using different hardware. We all converged on the exact same binary, which one of us signed.
  • The firmware needs to be trusted. It is a vendor supplied opaque blob.
  • The vendor software plays a significant role, since that's where the cross compiler comes from. We can verify the integrity of a download by having two or more people request the software and compare hashes. We currently don't verify the content of the vendor software. Given more time/resources we could either force the vendor to disclose the original source (for GPL binaries, such as the gcc cross compiler). An alternative would be to explore custom toolchains (e.g. cross-compile with our own gcc or llvm).
• For the code running on the server:
  • The build process is not reproducible. This is something I would really like us to achieve, but we currently don't have the resources to get there.
  • The build process requires trusted machines. We use fresh laptops, but it would be nicer to have each version be able to build the next iteration. Or have some kind of reproducible build image, and then build on that image.
  • The build process requires trusting Vagrant, Virtualbox, CentOS, Java, the contents of this repo, etc.
  • During a signing ceremony, we need to trust the hardware itself: the bios, keyboard, DVD drive, QR code scanners, HSM, etc.

Some other things I had in my notes:

• Most commits to this repo are GPG signed by the author. We however don't sign the review. We also trust Github at code review time.
• We don't verify the GPG signatures at build time.

From an attacker's point of view, there are a few scenarios:

• The attacker is an insider (or coerces an insider) and is able to covertly sign malicious transactions during a normal signing ceremony. The attacker then recovers the malicious signatures at a later point in time. This scenario is unlikely to happen (since the code is compiled with a single gateway wallet), unless the attacker also controls said gateway wallet.
• The attacker is able to circumvent physical security and sign malicious transactions without being an insider or interacting with an insider. Again, this scenario is unlikely to happen unless the attacker also controls the gateway wallet.
• The attacker is able to recover a quorum of admin credentials and encrypted wallets. The attacker can then recreate a clone of the entire wallet on their own hardware. This is probably the most likely scenario. The mitigations here are physical controls around admin credentials and ensuring admins can verify the state of things when they use their credentials. E.g. every admin gets to validate the bootable media before booting a server.
• The attacker leverages a supply chain attack to leak wallet keys or covert transactions. The main mitigating factors are the fact that the setup is offline (making such an attack harder) and the offline-online components have to converge on the same data prior to broadcasting the data.
• The attacker is able to partially perform a combination of the above scenarios and the combination is sufficient to steal funds.

[alokmenghrajani]

Looking at current docs - I'm assuming the iso resulting from https://github.com/square/subzero/tree/master/live-usb-creator#building-image is what you want on the DVD, but the "Writing image to USB drive" section is there just as another option?

Writing image to USB drive is useful for debugging / development purpose since USB drives are ~10x faster.

Also noticed the use of md5 (https://github.com/square/subzero/tree/master/live-usb-creator#build-process) - knowing it's weaknesses, do you think it's sufficient for the purposes?

The checksum is only used to detect corrupted/scratched optical media. It is just a hash of all the data and is not cryptographically tied to anything.

Ah also based on https://github.com/square/subzero/tree/master/live-usb-creator#building-image - the DVD/USB used should be at least 13gb in size? Might be worth adding to https://subzero.readthedocs.io/en/master/physical_components/#physical-components.

You need a lot more than 13GB of free space to build everything until the final iso (I usually keep ~50GB free). The final ISO is fairly small (around 1GB) and fits on any type of DVD.

[bosswissam]

@alokmenghrajani unclear what the centos root username and password are, is it just root with one of the passwords in https://github.com/square/subzero/blob/master/live-usb-creator/rhel7-livemedia.ks?

Also noticed the top of that file says "DEVEL" - outside of changing username/password, anything you'd recommend to be changed for production?

[alokmenghrajani]

Root is password-less: https://github.com/square/subzero/blob/master/live-usb-creator/rhel7-livemedia.ks#L306 There's also another user added here: https://github.com/square/subzero/blob/master/live-usb-creator/rhel7-livemedia.ks#L165

A root password isn't useful here (ideally we would make the system auto-login). One thing we don't do is encrypt the boot media, which would add a layer of defense (in addition to the existing password on the HSM's smart card) against physical access/tampering.

The DEVEL comment is probably incorrect, this is the only file we have and we use it for dev/staging/production. I believe this file was based on one of the files in https://github.com/weldr/lorax/blob/master/docs/.

[bosswissam]

Hm interesting.

I'm trying to boot a VMware Fusion machine from the iso, and it won't take root as username (immediately re-prompts) (screencapture: https://app.box.com/s/p0wl8km16iica8dwtzqqnsyj8xlh5ucy).

I can't boot-up the VM from the USB stick because Mac can no longer read the device after I ran the dd command to copy over the iso - is that normal? The device was in FAT32 to start. Actually seems even if I don't use dd, VMware fusion can't boot from a USB anyway.

Using this for testing before I go ahead and buy dedicated hardware.

[alokmenghrajani]

Moving build issues to dedicated issue.