Unacceptable release process, missing transparancy, privacy not guaranteed

[frenck]

I love your product guys, and I maintain the AdGuard Home integration and add-on for Home Assistant for that reason.

However, in the recent release (0.103.x) the binaries are no longer distributed on GitHub. Furthermore, Travis CI has been removed.

Looking at the general process, the builds are not public, and even worse, the binaries are hosted on a static private site. This means there is no transparency and impossible to know if the binary downloaded originates from the source code or is tempered with.

Making matters worse, there is no signing or verification process for the binaries.

In conclusion, IMHO this is unacceptable for the, privacy aware, open-source project this project claims to be.

I hereby propose to move the release process, build process and binaries fully into the public space to create transparency and provide proof to the privacy protection this project claims to offer.

../Frenck

[ameshkov]

Hm, never thought this may raise such questions.

Let me please explain the changes and motives.

1. Travis CI was not removed, it was replaced with Github Actions.

2. Keeping release scripts on Travis or Github Actions means sharing secret keys with third-parties, and I don't really like the idea of sharing this with anyone.

3. Moreover, at some point we will start code-sign executables and/or notarize macOS binaries, the situation will be even worse, as under no circumstances I will share our code signing certs with any third party.

4. Please note that the builds are reproducible and "CHANNEL=release make release" will produce the same binaries.

Knowing this, what do you think can be done to prove authenticity of the binaries properly?

[frenck]

Not sharing certificates and other secrets with third parties is acceptable, but that isn't a reason why the builds cannot be publicly.

For example, providing public read access to the build process of a private CI.

Great examples are for example the builds of Alpine Linux or any other Linux distribution. They generally use private builds server visible for the public, for the reasons you've just stated.

[c0b41]

well they don't even share old version just stable version, see https://github.com/AdguardTeam/AdGuardHome/wiki/Getting-Started#installation

[ameshkov]

Not sharing certificates and other secrets with third parties is acceptable, but that isn't a reason why the builds cannot be publicly.

For example, providing public read access to the build process of a private CI.

Could you please explain your threat model, I don't fully understand what this protects users from?

My thinking when moving release process to private space was that removing all third-parties will actually make things more secure.

Granted, I thought that having reproducible build is enough and didn't realize that not everyone will agree. However, I believe that we can find a solution that would solve both problems - my trust issues with third-parties and providing maximum transparency.

Am I right that at any point you would like to be sure or to be able to check that binaries aren't meddled with?

But neither hosting on Github nor making private CI logs public helps with this. Binary can still be silently replaced at any time on GH and on a server.

What if we use Github Actions to publish binaries checksums to Github releases. In this case checksums will still be kept&visible in CI logs that we don't control.

[ameshkov]

well they don't even share old version just stable version, see https://github.com/AdguardTeam/AdGuardHome/wiki/Getting-Started#installation

Well, the build is reproducible and at any point of time you can checkout any version code and simply compile it.

Or do you mean that this is inconvenient? Do you often need to downgrade?

[frenck]

Those checksums will not match after signing binaries.

While your approach sounds apparently more secure, it only provides security to you. As a user that security is fake and unverifiable, and thus a useless claim.

Even if this would rely on trust (which it probably boils down to), the binaries should be verifiable against the trusted source (which right now, is impossible as well). This can be achieved with e.g., GPG-signing. For example, I cannot verify some random dude/hacker got access to your static server and replaced the binaries with some nasty stuff.

However, IMHO this is still not ideal to a project that is about privacy.

[c0b41]

@ameshkov mostly i'am using specific version but latest release just latest release not version.

[frenck]

Just wanted to add to my last comment; I put a higher trust in GitHub, than your private static server. Which, in my opinion, is a fair and reasonable trust level to have.

[temporallyaccount]

Just wanted to add to my last comment; I put a higher trust in GitHub, than your private static server.

Dislike with that anyway.

[temporallyaccount]

Wait. how about firefox?

[frenck]

@Rru247 Firefox is publicly documented and verifiable, for more information see: https://firefox-source-docs.mozilla.org/taskcluster/signing.html

Or more general on Mozilla's wiki: https://wiki.mozilla.org/Security/Binary_Transparency

[ameshkov]

Let me please take some time and do a small research on this topic.

I agree with most of your points save for trust in Github. There are dozens of known cases with compromised accounts&repos on Github (the latest I guess is Microsoft), and I am pretty sure our static server hosted on our own hardware with private VPN-restricted access is safer. But yeah, unfortunately, my claim is unverifiable.

Anyways, I'll be back in a couple of hours.

Also, thanks for raising this topic and shame on me for not looking into verification best practices before making the CI change.

[temporallyaccount]

It'll be good if there solution for keep current changes and otherhand verifiable for rest of them. I'm not talking for anyone's side. Lol. but i concern about certain things pointed by @ameshkov above.

[frenck]

@ameshkov Take your time, these are not easy subjects or problems to resolve. Please, don't get me wrong, as this is not an attack of any kind.

I honestly believe there is no way to resolve this ever fully. The answer or acceptable solution probably is somewhere in the middle. Even GPG relies on a web of trust. The main point is; right now there isn't anything in place.

As for the Home Assistant add-on, I'll look into moving that into builds from source (as those builds are fully public, and I need a version number of some kind to be able to leverage Docker layer caching, which the current "latest" binaries do not provide as well).

[ameshkov]

Please, don't get me wrong, as this is not an attack of any kind.

No worries, being straightforward helps get to the point sooner, I am all for it.

As for the Home Assistant add-on, I'll look into moving that into builds from source (as those builds are fully public

Makefile is documented so this shouldn't be a problem, but let me know if you need any help with this.

and I need a version number of some kind to be able to leverage Docker layer caching, which the current "latest" binaries do not provide as well).

Hmm, you mean AG Home binaries or is it something missing in the Dockerfile?

[frenck]

Docker layer caching works based on changes/hashes of the individual Docker commands in the dockerfile.

Previously, my cache would bust, as the download url contains a version number, which is no longer the case with the current way the binaries are released.

(Which is in-line with the comments by @c0b41)

Additionally with just having latest available, mean upstream packages can no longer make reproducible builds based on your binaries as well. (Which is another problem for me as well)

[ameshkov]

Additionally with just having latest available, mean upstream packages can no longer make reproducible builds based on your binaries as well

You mean that there's nothing to reproduce, do I understand it right?

Hm, if it's useful to people then why not, publishing all versions is not much of a problem. We can do that to multiple mirrors (static server, Github releases, somewhere else?).

[frenck]

You mean that there's nothing to reproduce, do I understand it right?

Yes?

Let's say I build some sort of upstream package for AdGuard. I made that package a few days ago, based on AdGuard 0.103.1, and used: https://static.adguard.com/adguardhome/release/AdGuardHome_linux_armv6.tar.gz

Now a couple of days later, if I rebuild my package, all of a sudden, my package does not contain the expected 0.103.1 anymore, but a newer version (that surprisingly might break my package all of a sudden, without changing a single line of code to my package).

Being able to get a specific version of the software is crucial, not only for upstream packaging but also for end-users (e.g., rollback or any other provisioned environments).

Hm, if it's useful to people then why not, publishing all versions is not much of a problem.

That would be awesome.

We can do that to multiple mirrors (static server, Github releases, somewhere else?).

I think it doesn't matter that much, as long as it is available. Please note, that this does not solve the original reasoning behind my issue creation and left this part out of the original issue intentionally.

[szolin]

I'm not an expert in this domain, but here's what I think:

1. It's better to add a version to AGH releases, so they become e.g. AdGuardHome_windows_v0.104.0_amd64.zip

2. It's better to package into .rpm and .deb and sign them with AdGuard key, so the packages are verifiable by anyone. It's OK to continue publishing .tar.gz and .zip packages for those who like them more.

3. Binaries hosted on a private server owned by AdGuard are actually safer for users than hosted on GitHub. Users trust AdGuard only, not AdGuard + GitHub. And p.2 solves the issue with a publicly verifiable binaries. In case someone doesn't trust AdGuard in the first place - he should understand that there's no possible way to solve his issue, but to write the software himself.

These are just my thoughts.

[frenck]

Binaries hosted on a private server owned by AdGuard are actually safer for users than hosted on GitHub

The is no way to verify that statement. I don't know about their security policies. A rogue employee, script kiddie, or some kind of hacker exploiting a security flaw in that beautiful private server.

At the time I download it, that binary might have been replaced already. I cannot tell. So please sign all binaries a provide a way to verify the binary hasn't been tampered with by non-authorized people.

In case someone doesn't trust AdGuard in the first place

It is not AdGuard I don't trust. It is the rest of the world, which you seem to miss in this case.

[szolin]

A rogue employee, script kiddie, or some kind of hacker exploiting a security flaw in that beautiful private server. At the time I download it, that binary might have been replaced already.

The same applies to GitHub, doesn't it?

It is the rest of the world, which you seem to miss in this case.

No, I haven't missed it - my p.2 covers such issues. If someone silently replaces a binary file you would know as it won't be signed by AdGuard. If someone steals a private key for signing (which I think is far more safely stored in a private place) - well, is there anything anyone can do?

[frenck]

The same applies to GitHub, doesn't it?

Yes it does, absolutely.

well, is there anything anyone can do?

Just cause a lock on your front door could be picked, doesn't mean we should not use a lock at all.

[ameshkov]

Quick upd: I am still on vacations and being on a road trip makes it hard to formulate thoughts so please excuse me.

Briefly, my thoughts are:

1. We should sign the binaries (detached GPG?).
2. We should publish checksums as well.
3. We should provide a document explaining how to verify binaries and how to build the very same binaries from source.
4. We will be publishing all AGH versions, not just keeping the latest one (I admit this was a particularly bad idea).
5. We will be publishing release binaries to GH releases as well, people seem to be used to it.

I'll get to it when I am back next week. If I am missing anything, please comment.

[ameshkov]

Done.

Beta and edge channel releases are now signed with GPG (the next stable release will be signed as well). Beta binaries are published to Github releases as well.

Binary verification: https://github.com/AdguardTeam/AdGuardHome/wiki/Verify-Releases

[frenck]

Amazing! ❤️