search

actions / runner-images

GitHub Actions runner images
MIT License
10.1k stars 3.04k forks source link

Should Chocolatey be used to build Windows Images? #2693

Closed jachin84 closed 3 years ago

jachin84 commented 3 years ago

Description
Looking at https://github.com/actions/virtual-environments/blob/main/images/win/scripts/Installers/Initialize-VM.ps1 it seems as though the community repository is being used which is not recommended. See https://docs.chocolatey.org/en-us/information/security.

This makes it harder to justify the use of the hosted build agents when having a discussion with security folk.

Area for Triage:
Servers

Question, Bug, or Feature?:
Question

Virtual environments affected

al-cheb commented 3 years ago

@gep13, @pauby, Hey, folks. Could you please help us to clarify this moment?

gep13 commented 3 years ago

The section of that document that @jachin84 is referring to is likely this one:

When you use Chocolatey in an organizational sense, do so in a manner that requires no internet access. Chocolatey doesn't require internet access at all. The default source (https://chocolatey.org/packages, aka the community package repository) that is available on installed is typically the first thing to be removed when organizations are using Chocolatey. This provides the utmost in security for organizations.

and:

It goes without stating that if you are a business and you are using Chocolatey, you should think long and hard before trusting an external source you have no control over (chocolatey.org packages, in addition to all of the binaries that download from official distribution channels over the internet). It is both free and easy to set up your own private feed where you can vet packages and have complete control over the binaries and what gets installed. This also provides a complete offline solution that is reliable and trustworthy. This is what we recommend for businesses that use Chocolatey in production scenarios (and what many of them do). There is a great article written up on the reasoning and options for hosting your own server.

Now, what I will say is that any package that is published to chocolatey.org does go through a rigorous package moderation process, and all binaries which are used within a package (where possible) are sent to VirusTotal for scanning, so we, Chocolatey, attempt to make the usage of the packages on chocolatey.org as safe as possible.

In an organizational context, the acceptance of risk is completely different, and we definitely encourage Organizations to completely own all areas of their package installation automation, including bringing all packages in-house and vetting all the packages and what they are doing prior to installing them.

This is not an issue that is specific to Chocolatey on Windows. The same could be said about using brew on the Mac images, and the Linux Package Managers on the Linux Images.

ferventcoder commented 3 years ago

I believe the title should be updated from "Should Chocolatey be used to build Windows Images?" to "Should Chocolatey Community Repository be used to build Windows Images?" as that is what the discussion is about here.

ferventcoder commented 3 years ago

I believe Gary said it all very well especially the point about "This is not an issue that is specific to Chocolatey on Windows. The same could be said about using brew on the Mac images, and the Linux Package Managers on the Linux Images." - we're just more open about it than other groups. I don't have much more to add to what Gary said as it was quite well written and addresses the points of concern here.

As a side note (a little off-topic): there is a distinction that needs to be assessed over organizational use and what GitHub is doing here - there is a nuance here where an organization is providing a service to the Windows community and software developers that is free, that feels like an intended use of the Community Repository. Where a company/organization is managing an internal infrastructure, that is definitely not an intended use of the community repository.

maxim-lobanov commented 3 years ago

As I understand, the alternatives of using community repository are hosting own choco repository or refuse using Chocolatey at all.

Hosting own choco repository will bring much complexity to image generation since we will have to follow all tools updates and upgrade tool versions in our repository according to it. I am not sure that it will be safer because we definitely won't be able to moderate every tool version and every update from our side as Choco team has already been doing. Also it won't work smoothly for customers who uses our repository for generation own images (self-hosted agents). As Choco team mentioned above, they already have defined process of moderation:

Every version of every package submitted must pass through a rigorous moderation review process before they become publicly available (includes checks for quality, consistency, installation, and validations against VirusTotal).

If we get rid of Choco completely, it will lead us to maintenance hell where we have to maintain installation process for hundreds of tools manually. Adjust image-generation scripts for tool updates (where installation process is changed) and etc.

Chocolatey (and other package managers, like brew on macOS and Ubuntu work very similar) provides great experience where community and tool owners maintain installation process for tools and installation process is pretty simple for us from imagegen side.

In our understanding, we should trust choco and other packages managers as long as they have own verification process (moderation process) to make sure that packages are valid and not compromised. cc: @alepauly for any thoughts

jachin84 commented 3 years ago

Some great discussion here, thanks.

Perhaps for a bit more context. Due to organisational security policy I cannot use the Github hosted agent. So I need to build my own image. I found this repo originally when searching for how to make a my own agent image. However I cannot use the script from this repo to build my image either as this process is potentially open to supply chain attack. See https://docs.microsoft.com/en-us/windows/security/threat-protection/intelligence/supply-chain-malware.

While I think there would be some effort involved in removing dependencies on community package repositories, I think doing so is worth it in this case. It would also demonstrate to the community that GitHub takes security very seriously. The recent SolarWinds breach is an excellent example of a supply chain attack and shows what can happen when an attacker gains access to a build system.

I have already started work on a private projected based on this repository. My current approach is to download installers from a blob storage account and verify the hash before attempting installation. This does mean I need a process to download and hash binaries before adding them to the storage account. Maybe not ideal, but certainly very secure and I'm expecting more reliable. Ie. the image build process won't fail because of an unexpected new version of a package that has a different install behavior. Version updates become very deliberate.

gep13 commented 3 years ago

@jachin84 for the process that you are describing, i.e. building your own copy of the build image, there would be nothing to stop you continuing to use Chocolatey, but you would configure during the initial Packer execution, to point Chocolatey at a different source.

This would be your internal package feed, that would have your own privately vetted versions of the packages. In addition, you would configure Chocolatey to NOT use the Chocolatey Community Repository.

This gives you the benefit of directly controlling all the packages that are installed on your base images, without having to re-write all the installation scripts.

As part of the installation process for a Chocolatey package, the validation of the installer hash is already being performed, and Chocolatey would error out if these hashes did not match as part of the installation.

gep13 commented 3 years ago

@jachin84 said... the image build process won't fail because of an unexpected new version of a package that has a different install behavior. Version updates become very deliberate.\

The process that I have described above also helps with this problem. You simply wouldn't add a package into your own internal feed until you have verified that it does exactly as you expect it to.

We, Chocolatey, has long advocated that any script that uses Chocolatey to perform the installation of the package should always pin the installation to a specific version of a package, unless you are 100% certain about upgrading of a package.

You would want to do:

choco upgrade packageName --version 1.1.1

Rather than

choco upgrade packageName

Unless you completely control all the package updates that make it onto your internal source/feed.

ferventcoder commented 3 years ago

However I cannot use the script from this repo to build my image either as this process is potentially open to supply chain attack. See https://docs.microsoft.com/en-us/windows/security/threat-protection/intelligence/supply-chain-malware.

@jachin84 I get that you introduce risk for this anytime you decide to use another tool. So it's worth noting that to avoid supply chain malware, you likely can't use any products from the internet (python, make, Microsoft's own build tools) without going back and verifying with each of them that they are using secure practices.

Chocolatey's community repository documents our processes - what you get when you download a package from the community repository is the same software you get from the official distribution points and we have the data all open to show that to anyone on the internet.

So yeah, potentially there is still a case for supply chain attacks, but the packages themselves already have a lot in the way of security as @gep13 described to lessen the supply chain attack against what might need to be designed elsewhere. Between Chocolatey itself and the moderation processes for the community, a package approved already:

ferventcoder commented 3 years ago

While I think there would be some effort involved in removing dependencies on community package repositories, I think doing so is worth it in this case. It would also demonstrate to the community that GitHub takes security very seriously.

@jachin84 This is not meant to be snarky at all, more of a genuine curiosity - There are real organizations who have 10+ years experience in taking security quite seriously - in offerings such as these and real lessons learned and this is shown in the offerings to the community. Given that actual, hard-fought experience is there with the folks who offer these community-based resources and they take security very seriously already, what is it that GitHub folks could offer here above and beyond this and not make mistakes along the way?

jachin84 commented 3 years ago

Sorry I did not, in any way, mean to sound ungrateful and do recognize the large effort that goes into security.

I was hesitant to even log this as an issue as I had a pretty good idea it wouldn't accomplish much. My only objective was to make life easier for those that are in similar shoes to me. Let me explain. I work at a company that makes a SaaS produce in the Finance industry. On a regular basis we have to have conversations with our customers about how we handle security. There are some topics that are coming up more often where we get a bit bogged down in the detail. One of these topics is the specifics of how we build our software and how we run our software. In all cases so far where I have had to explain Chocolatey I've lost the debate (As an aside I use both Chocolatey and more recently winget and personally have no issue with community package managers). In the end it was just easier to remove it from our setup which made these security conversations much easier.

gep13 commented 3 years ago

@jachin84 said... In the end it was just easier to remove it from our setup which made these security conversations much easier.

Genuine question here... In these discussions, was the idea of owning the packages being consumed internally come up as an option? There are folks using Chocolatey in completely air-gapped networks, with no ability to call out to the internet, and they are doing so successfully.

pauby commented 3 years ago

@jachin84 I haven't added anything to this conversation as yet as I didn't want it to come across as everybody piling in with questions, but I did have one.

In all cases so far where I have had to explain Chocolatey I've lost the debate

Do you have the same debate about Linux package managers / Homebrew (assuming one or both are applicable to what you're doing)? I'm curious to know if this is a security concern around package managers in general or just Chocolatey.

jachin84 commented 3 years ago

Genuine question here... In these discussions, was the idea of owning the packages being consumed internally come up as an option? There are folks using Chocolatey in completely air-gapped networks, with no ability to call out to the internet, and they are doing so successfully.

Granted, I never looked into a private Chocolatey feed. I have more than a few years experience in large scale enterprise desktop deployment so was happy to stick with the tools I know.

jachin84 commented 3 years ago

Do you have the same debate about Linux package managers / Homebrew (assuming one or both are applicable to what you're doing)? I'm curious to know if this is a security concern around package managers in general or just Chocolatey.

Windows only.

pauby commented 3 years ago

@jachin84

Windows only.

Apologies. Just want to be clear, you only have the debate about Windows package managers, or you're only using Windows package managers?

ferventcoder commented 3 years ago

In all cases so far where I have had to explain Chocolatey I've lost the debate

This seems like we (Chocolatey Software) are not arming you with the proper information (documentation) in order to have a comprehensive talk on the security of Chocolatey with folks. This seems to happen from time to time - I'm hopeful that this year with the changes and documentation we are planning, that will definitely help in those conversations. One such area where it is explained really well is how it is used in a secure environment for a particular company - https://youtu.be/hZapqJImt1w

naikrovek commented 3 years ago

I'm OK with Chocolatey, as long as it can be proven to be reasonably secure to security people in the enterprise.

What I'm not OK with is the builder script continually failing in the Chocolatey stages somewhere, about 80% of the time. I haven't been able to build an image in weeks without manually removing all Chocolatey steps, because one of them, somewhere, always seems to fail. Checksum won't match, or an installer will exit with a non-zero exit code... Wait a day, try again, checksum failure somewhere else or a different installer gives a non-zero exit code.

Use whatever you want to make the installer steps simpler. I don't care at all, but use something that works. Please.

miketimofeev commented 3 years ago

@naikrovek we're constantly improving our build scripts, please report any problems you see by creating an issue. The latest problem with checksum was fixed here https://github.com/actions/virtual-environments/pull/3035

naikrovek commented 3 years ago

I'm not sure what you'll be able to do to counter a Choco installer returning something non-zero but ok. I'll file an issue next time I encounter this. It's already failed once today so I'll try again tomorrow.

miketimofeev commented 3 years ago

@naikrovek we can consider getting rid of choco for some tools, thanks!

naikrovek commented 3 years ago

Before I forget, let me express a strong feeling of gratitude for the existence of this repo and it's ongoing maintenance. You are saving a lot of people a lot of time maintaining their own solution, and even though most of us are silent, we are all thankful and appreciate the time and effort that goes into this.

maxim-lobanov commented 3 years ago

Closing the issue since initial question was answered above: https://github.com/actions/virtual-environments/issues/2693#issuecomment-779114218 Please let us know if you have any additional thoughts or questions.