Un-auditable dependencies in GRR code

[darrenbilby]

We are acquiring a number of dependencies on external, unauditable blobs that are required for using GRR. While its true that we'll always depend on blobs of code, at least if they are tracked in a code repo users have an option to track the changes and updates.

I don't think its a necessary state for the things we control, so would be great to start removing some of them, or at least justifying why they are there, from a brief look I found:

Server debs on docs: https://github.com/google/grr/blob/master/scripts/install_script_ubuntu.sh#L21

Rekall component on apache server: http://images.rekall-forensic.com/share/rekall-core-1.5.0.tar.gz#egg=rekall-core-1.5

Chipsec component (chipsec fork tarball in docs): (relies on link in docs, fix in progress)

mrgcastle Vagrant images vs upstream: https://github.com/google/grr/blob/master/vagrant/Vagrantfile#L14

[destijl]

Re the vagrant images, I would have loved to not have to make my own, but there isn't an official build for that version of redhat. I think the way forward for building linux clients is to use docker (for a number of reasons), one of which is better official image support (including for centos 5.11). https://hub.docker.com/_/centos/

[scudette]

The Rekall code is pre-release. Alas PyPi do not allow re-uploading of binaries so every time we push a release to PyPi we need to up the version number and this means rebuilding all the distributions at the same time. The intention is for this code to be officially released when GRR is also released, but we need to test the integration somehow.

BTW we are moving more towards binary releases and less source code building. The next client will be completely buildable on a virgin windows machine (obviously with python though) without installing any compilers at all. It takes about a minute to go from git clone to functioning client templates.

On 10 February 2016 at 03:03, Greg notifications@github.com wrote:

Re the vagrant images, I would have loved to not have to make my own, but there isn't an official build for that version of redhat. I think the way forward for building linux clients is to use docker (for a number of reasons), one of which is better official image support (including for centos 5.11). https://hub.docker.com/_/centos/

— Reply to this email directly or view it on GitHub https://github.com/google/grr/issues/331#issuecomment-182166869.

[darrenbilby]

Moving to binary releases is fine, that makes sense, and its how most software is released. The point is that those releases should be signed and be released on an official site that can be tracked, PPA seems to be the normal method for that?

I don't like the idea that there is no way I can tell if you modify the server deb file on google drive, and have no guarantees about the integrity of that file other than "I hope they don't lose the password for that account". Given we expect people to run this code as root everywhere, we should be setting the bar pretty high for release processes.

As for the source side, its awesome that we're working to get everything to build from scratch, thanks for all your work on that, lets just try and keep un-auditable binary blobs from ending up as part of that process.

[scudette]

PPA is only appropriate for ubuntu packages which GRR is not.

Python packages can be signed (https://pypi.python.org/security) but I dont know if other packages we depend on are signed in the same way.

It is possible to host all the dependencies on a google storage bucket (even for third party ones) by simply doing:

pip wheel grr -d some_directory/

will drop wheels for all dependencies in that dir.

The you can just publish the directory over HTTPS (eg. in a cloud storage bucket) and install them all:

pip install grr --find-links URL of index of release.

This way it will not even install from PyPi (in case PyPi got hacked) but it will instead download the same wheels from GRR storage bucket (over ssl).

At the end of the day you have to trust the person making the release obviously. Losing the password for a PPA account, or a github account is just as bad a losing the password to the releases PPA.

I guess the question is - who do we trust, and how do we do it? Do we trust PyPi? if so, then publish everything on PyPi, if not then host our own wheels. How do we affect the trust? Keep the ability to make a GRR release to a small group of well audited people and trust the bucket security to enforce non-modification - is that enough? Should we also add GPG signatures for individual uploaders?

These questions must be answered in any release system, so thanks for bringing them up for discussion.

Thanks Michael.

On 22 February 2016 at 04:19, Darren Bilby notifications@github.com wrote:

Moving to binary releases is fine, that makes sense, and its how most software is released. The point is that those releases should be signed and be released on an official site that can be tracked, PPA seems to be the normal method for that?

I don't like the idea that there is no way I can tell if you modify the server deb file on google drive, and have no guarantees about the integrity of that file other than "I hope they don't lose the password for that account". Given we expect people to run this code as root everywhere, we should be setting the bar pretty high for release processes.

As for the source side, its awesome that we're working to get everything to build from scratch, thanks for all your work on that, lets just try and keep un-auditable binary blobs from ending up as part of that process.

— Reply to this email directly or view it on GitHub https://github.com/google/grr/issues/331#issuecomment-186981614.

[clausing]

No, if you are distributing/using a deb for install on Debian/Ubuntu systems then it is perfectly appropriate to use a PPA so that it can be installed (and automatically updated, or at least automatic notification of new versions) via apt. There is plenty of stuff in PPAs that isn't, strictly speaking, Ubuntu

[joachimmetz]

There is plenty of stuff in PPAs that isn't, strictly speaking, Ubuntu

There is also a lot in PPAs that is not verified either. That a dpkg is is signed for upload does not mean someone actually bothered to verify it.

I don't think its a necessary state for the things we control,

Agree, to make a long bug short ;)

For plaso we have been working on getting dependencies into GIFT. We've been building l2tdevtools to make the build and distribution process easier. I opt to get GRR and rekall into this this same deployment process, also see: https://github.com/log2timeline/l2tdevtools/issues/12

@scudette and I killed a major distribution hurdle with pytsk recently: https://github.com/py4n6/pytsk/commit/7dc523be8eb1b4348ddaa9ef5c890045211a14c9

rekall deployment should be something easlier now as well with the separation of core and UI

some dependencies are hard to package especially of conservative release cycles for linux distros versus the speed at which GRR and its dependencies evolves. If we can get rid of those in favor of other more friendly dependencies that could be useful.

[scudette]

Jim, I agree that a ppa is a good method for distribution of dpkgs but grr is more than just a Debian package. It is a python project which can actually run on many platforms. It's OK to also distribute for Ubuntu but we should mainly distribute as a python package. Am 26.02.2016 10:32 nachm. schrieb "Joachim Metz" <notifications@github.com

:

There is plenty of stuff in PPAs that isn't, strictly speaking, Ubuntu

There is also a lot in PPAs that is not verified either. That a dpkg is is signed for upload does not mean someone actually bothered to verify it.

I don't think its a necessary state for the things we control,

Agree, to make a long bug short ;)

For plaso we have been working on getting dependencies into GIFT https://launchpad.net/%7Egift. We've been building l2tdevtools https://github.com/log2timeline/l2tdevtools to make the build and distribution process easier. I opt to get GRR and rekall into this this same deployment process, also see: log2timeline/l2tdevtools#12 https://github.com/log2timeline/l2tdevtools/issues/12

@scudette https://github.com/scudette and I killed a major distribution hurdle with pytsk recently: py4n6/pytsk@7dc523b https://github.com/py4n6/pytsk/commit/7dc523be8eb1b4348ddaa9ef5c890045211a14c9

rekall deployment should be something easlier now as well with the separation of core and UI

some dependencies are hard to package especially of conservative release cycles for linux distros versus the speed at which GRR and its dependencies evolves.

— Reply to this email directly or view it on GitHub https://github.com/google/grr/issues/331#issuecomment-189490075.