How can I verify the integrity of DnsControl?

[jamieweb]

Before implementing DnsControl in a production environment, I need a way to verify the integrity of the software download.

Is there anywhere that I can get GPG signatures of the releases? Or even just plain hashes? Some commits are also not signed, so I cannot verify this way.

It'd be most ideal if this integrity verification was somehow built into the distribution method - perhaps a packaged release (e.g. Snap, Flatpak) could be a helper for this?

[tlimoncelli]

Hi Jamie,

The releases on the github release page (https://github.com/StackExchange/dnscontrol/releases) include the githash of the release. It is not a cryptographic hash, nor are the individual commits GPG signed.

Many competing projects provide more formal release processes. Or you might consider a commercial product with enterprise support. I hear good things about OctoDNS, Terraform (which has an enterprise version), and others.

[jamieweb]

It's unfortunate that this was closed without further discussion. I appreciate that the with nature of open-source, release procedures can be a challenge in some cases, especially to do securely. When done properly though, the release/update procedures are definitely better than with proprietary software, due to the inherent transparency and auditability.

One of the main points of DnsControl is to improve DNS deployment for production systems, by introducing usable change management, testing, etc. I really would urge you to consider this issue further if you can, as currently the lack of release/build integrity verification is a significant barrier to me using this.

I would normally suggest that you open this up for PRs, but in this case I don't think that'd be beneficial, as the changes would have to be to your release procedure, rather than to the code itself.

Thank you for your consideration.

[tlimoncelli]

Hi Jamie,

Sorry for closing this issue prematurely. That was a bit rude. I apologize.

What would you like to see for "release/build integrity"? That is, could you state in technical terms what you would consider satisfying your need?

[jamieweb]

Hi @tlimoncelli,

No problem, thanks for re-opening this.

Essentially what I'm looking for is some form of protection against the distribution channel being compromised, as well as a way to verify cryptographically that the code I downloaded is actually the same as the code you wrote and distributed. Preferably this should be possible to do automatically, instead of having to perform complex manual checks every time there is a new release/update.

At present, the integrity of the DnsControl software that users download directly depends on the security of this GitHub repository, as well as all developers who have access to it. Running go get or downloading the executable from the 'Releases' page is essentially volunteering to run untrusted and unverified code from the internet, which is always a bad idea.

There are many ways that you could provide verifiable downloads for DnsControl - I've listed a few below in a very rough order of least to most security:

• Share hashes of the binary releases through a different distribution channel (e.g. SE website)
• Sign all commits with your developer GPG keys
• Sign releases with your developer GPG keys and provide the signature along with the download
• Integrate integrity protection into the distribution method, e.g. a Snap/Flatpak package with enforced signature/hash checking, etc

Unfortunately, most users probably won't perform the manual checks, so an option where they are built in and fool-proof would be the best approach (and also the most work).

Thanks for your help, and I'm happy to provide any further information you need.

Kind regards, Jamie

[captncraig]

Downloads from the github releases do depend on having a trusted https connection to github. An active network level attacker replacing the binaries in-flight seems much less likely than any of:

1. Malicious "contributor" providing a feature PR with hidden malicious code disguised as a vendored dependency update.
2. Compromised maintainer github credentials allow a malicious actor to push malicious code into the repository. Future releases get this bad code if unnoticed.
3. Compromised or rogue maintainer uploads a bad release built from source code that does not match the advertised tag from the release.

We can mitigate number 1 by enforcing a strict code-review policy for pull-requests (even for maintainers), but a very skilled attacker could foreseeably sneak something past us.

Github credential security is paramount to numbers 2 and 3. I'm skeptical any proposed solution would be particularly immune if our github accounts are "hacked" so thoroughly. Any hashes or keys published in the repo would be easily replaced in such a case. We do enforce a 2FA policy for anybody with write-access.

It seems very hard to verifiably prove that a go binary was built from a particular set of unmodified source. We can show you the build scripts and so forth, but if you absolutely must trust the entire chain of custody, you'd be better off building from source yourself.

Regarding commit signing, I have read varying opinions about their effectiveness. We have a fair number of external contributors, and I'd be very hesitant to add policies that would discourage contributions. In general, I don't particularly care who wrote a commit. I care that it is good code without bugs or malicious intent. We achieve this by carefully checking prs before we merge them into master. Git itself verifies the integrity of the files once admitted to master.

Publishing signatures of release binaries seems like the most obvious, feasible solution to some of this, but it really just says "yes, this binary was built by me". That protects against some possible attack vectors, but the public keys would need to be published elsewhere, and it seems unlikely anybody would really ever verify those sigs. And if my github credentials are compromised, it is entirely possible the public keys are stolen as well through a compromised developer machine.

All this really leads me around to: if you don't trust the github downloads, by all means build it from source yourself. This eliminates any attack vector downstream from actually getting malicious code into the repository.

[tlimoncelli]

Let's not try to boil the ocean, folks.

I feel like you're cataloging everything that could be done, but not answering my original question: What would satisfy your need?

To be more clear: What business policy or compliance rules are you trying to meet?

[jamieweb]

The policy is that all software installed must pass a robust form of integrity checking, to give reasonable assurance that the software downloaded matches the software that was produced and released by the official, legitimate developers.

Examples of 'robust integrity checking' in this policy include: APT packages that are GPG signed, Binary releases that are GPG signed, hashes distributed through a different channel and Snap packages that are GPG signed.

Automatic verification methods are preferred (e.g. APT/Snap), rather than manual ones, due to the extra work required to install updates, possible human-error, etc.

HTTPS/TLS doesn't count significantly towards this in most cases. For example, malware hosted on a compromised GitHub account would still be served with valid TLS, so in that case TLS provides little protection against what I'm talking about.

[tlimoncelli]

I was hoping that you'd say something along those lines. When you put it into those terms, it seems more tractable.

It would be great if dnscontrol was available as an APT package (and YUM, and etc.) for both the integrity reasons, and because it would make it easier for people to get.

I'm not too connected to the Linux developer community, so I don't know what the process is to get into APT. How difficult is it to find a volunteer?

Tom

[jamieweb]

I think it can be quite challenging for new developers to get their packages into the Debian/Ubuntu APT repositories, especially if you don't have existing contacts. However, for Snap, anybody can release packages as long as they meet certain criteria (open source, not malicious/adware, etc), so I'd suggest having a look at Snap as a start.

Above all, having DnsControl available as a Snap would make it much more accessible. Lots of people click away from projects quickly when they see that they have to manually download/compile stuff, so having a simple snap install command would be great.

Thanks

[tlimoncelli]

We don't use that distro where DnsControl is developed (Stack Overflow) so it would be rather difficult for us to do it. However, we'd gladly have a volunteer take ownership and set it up. You, perhaps?

[jamieweb]

Apologies if I've misinterpreted what you said, but just to confirm, by 'Snap packages', I am talking about the universal packaging format for Linux, which runs on most modern distros including Debian, Ubuntu, Mint, Arch, Fedora, etc. See snapcraft.io for more details.

In regards to helping with packaging DnsControl for the Snap store, that is something I would be interested in helping with. However, for me to do that, I would still need a way to verify the integrity of the downloaded DnsControl software that I am packaging, as it would be irresponsible of me to package and redistribute DnsControl without having adequate assurance of its integrity. I think it would be more secure for the Snap package to be produced by yourself/Stack Overflow, however if you don't have any plans to release a Snap package for DnsControl, then I'd be happy to take that responsibility onboard.

In regards to the integrity checks for if I publish the Snap package, I would probably be compiling the source code rather than using the binary releases, which makes things easier. One possible way would be for you to ensure that each release tagged commit is signed with the GPG key of a trusted DnsControl developer, so that I have reasonable assurance that a compromised GitHub account or rogue contributor hasn't modified the source code.

Also, the owner/manager of a Snap package can be transferred, so if I were to release DnsControl on the Snap store, Stack Overflow would be able to take over if needed, as package names are given out to the project owners as a priority, rather than on a first-come first-served basis.

Thank you.

[jamieweb]

@tlimoncelli If you have an update on this, please let me know. I'm happy to take onboard the responsibility of the Snap package if I can verify DnsControl properly before packaging it.

[tlimoncelli]

Hi Jamie,

Sadly, I'm in the middle of a number of big projects right now and I can't give this the bandwidth it deserves.

If you want do it, that'd be great. We can transfer it to Stack in the future. I can gpg sign the releases. I'm not entirely sure what that involves (I use gpg, but haven't used it in combination with Git) so let me know what you need.

Tom

[jamieweb]

@tlimoncelli,

Thanks, I'll have a look at packaging this and let you know what I need security/integrity wise.

I'll hopefully get this done within around 2 weeks.

Thanks!

[jamieweb]

@tlimoncelli

Hi, to give a quick progress update for you:

I've set up a development repo on my GitLab account, so the Snap build environment will be public/open source. I've also registered the Snap with the Snap store, so it can be published when it's ready to go.

My build/release process will be something like this:

1. DnsControl update is released by StackOverflow
2. I receive notice of update via RSS feed on GitHub releases page
3. I pull down the new code, and merge it into my Snap package staging repo, checking for a signed commit by a trusted developer at HEAD, and performing a basic code review where possible
4. I package the app using Snapcraft, and test that it works using a test machine
5. I merge my staging repo into production
6. I publish the Snap package to the Snap store
7. Users can install it using snap install dnscontrol

In order to meet the integrity requirement, it would massively help if you could try to ensure that the HEAD commit of a major release is always GPG signed. This at least gives me some assurance that the code at the time of the release is safe.

Thank you for your help, Jamie

[tlimoncelli]

Sounds good! I'm traveling this week but when I get back I plan on doing a new release, and I'll GPG sign the release. To be clear, you mean GPG sign the commit that increments the release number, etc. using the process documented here: https://help.github.com/en/articles/signing-commits

[jamieweb]

Yes, what you said there is correct. The tagged commit for the release needs to be GPG signed using Git (like the link you gave shows).

Enjoy the rest of your week, and I look forward to the new release! :)

[jamieweb]

@tlimoncelli

Another status update for you.

I've successfully built a Snap packaged version of DnsControl on a test machine. It is working fine, including been able to make network connections to providers.

My next step is to configure and test the Snap confinement, as Snap packages are heavily sandboxed with access only to the services/features that they need (e.g. network access).

After that I will set everything up in my local build environment (rather than on a test machine), and check the proper package info into my dnscontrol-snap Git repository so that it's all public/auditable. Then it should be ready to publish!

[jamieweb]

@tlimoncelli

I've got everything working in my local build environment, including the Snap confinement, so I'm pretty much ready to release it on the Snap store.

My public Git repository containing the working build configuration is: https://gitlab.com/jamieweb/dnscontrol-snap The important file in the repo is snapcraft.yaml.

Once the next DnsControl version is released (with a GPG signed release tag), I'm happy to release the Snap to the Snap store.

Do you want to do any of your own internal testing/review of my Snap first?

[captncraig]

I'm afraid we don't have a good environment to test your snap. We trust you. Is the packaging something that can be merged back into this repo? A dist folder would make sense for this kind of snap packaging, potentially rpm scripts or choco packages and things in the future.

What servers are actually involved in building a package? Is it something you run and upload, or is there some more automatic process?

[jamieweb]

I can look at merging in the Snap config if you'd like. The file ideally needs to be at /snapcraft.yaml or /snap/snapcraft.yaml within the repo.

The actual build process is performed using the Snapcraft client on my local machine. It compiles the application and packages it along with all dependencies into a .snap file, which can then be pushed automatically to the Snap store.

DnsControl failed to build as a Snap with the /integrationTest directory present, as the Go compiler fails since there are only test files within this directory. This is something that you might be able to fix within this repo.

I noticed that the version in master was just bumped to 0.3.0 - are things almost ready for the v0.3 release?

[tlimoncelli]

Hi! Quick update. My crappy week has become a crappy month. I'm dealing with 2 emergencies right now. I'm not going to be able to work on this this month, June doesn't look good, and I'm on vacation July 1-Aug 8.

[jamieweb]

@tlimoncelli Thanks for the update - I understand your need to prioritise and appreciate you letting me know!

If any of the other developers want to take this on while you're unavailable, I will be happy to work with them. Also, if the next release tagged commit can be GPG signed, I can merge all the changes into my repo and actually publish the package. Once you're back, we can look at merging it into upstream so that Stack Overflow can manage the packaging going forward.

I hope everything is OK with the emergencies, and enjoy your vacation when it comes.

[tlimoncelli]

Well here's my first try. I did a release (v2.9) with "git commit -S" and it requested my gpg passphrase.

I don't see any signatures when I do "git log" so I'm not sure if I did it right. Let me know.

[jamieweb]

Hi Tom,

Thanks for giving this a go.

In order for the GitHub releases page to have the 'Verified' emblem, you'll need to sign the tag, which you should be able to do with git tag -s. You'll also need to make sure that your GPG public key is properly added to your GitHub account.

For example, the releases page for KeePassXC has green 'Verified' emblems for each release: https://github.com/keepassxreboot/keepassxc/releases

I'm just going to merge in the changes to my local dnscontrol-snap repo and test the Snap package.

Thanks for your help, Jamie

[jamieweb]

@tlimoncelli @captncraig So you're aware, I'm still keeping an eye on this. As soon as the next release comes around, I'll package and test it, then publish the Snap if that's all OK with you. After that I'll submit a PR to merge the Snap config in, and then hopefully SE can take over the Snap packaging responsibilities.

[tlimoncelli]

Friendly ping?

[tlimoncelli]

This is stale. I'm closing this. Please re-open if needed. Thanks!