Provide linux installation without homebrew

[SkeLLLa]

Homebrew is not common tool for linux systems. And it (homebrew) not only consume space and interfere with other tools, but it also works rather bad. Also SAM cli is only one tool that requires homebrew to be installed.

There are a lot of better tools that provide linux system installation. Consider using them instead of uncommon homebrew and will make this tool linux-friendly.

[TiS]

For starters, I suggest snapcraft (highly), or flatpak (somehow reluctantly), if you want to have one install for multiple environments.

[SkeLLLa]

@TiS I don't think that this actually need functionality that snapcraft provides. I think it would just some installation script like nvm or even homebrew does curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash.

[rapofran]

please this

[rapofran]

@SkeLLLa did you find some workaround?

[TiS]

I suggested these tools, because they bundle dependencies. This is good for tools like SAM, if we want them to be independent from OS details and packages.

Coming from Ubuntu, Snapcraft is the most natural choice for me, but I heard many people choose flatpak as better alternative (I don't know why, though, as my personal flatpak experience is quite bad; maybe it's just my luck).

Using just installation script can be tricky, as one has to somehow take into account all of package version differences between systems. Also, installation script makes things like uninstall / upgrade harder to do, while snapcraft / flatpak, being package managers, suport them out of the box.

[rapofran]

I think the docs are bad written

running pip install aws-sam-cli installs everything OK

[siwyd]

It is a bit peculiar to suggest Linux users to install Homebrew when a simple pip3 install --user aws-sam-cli works.

[dwhitla]

Plenty of Mac users refuse to install Homebrew also because it interferes with their alternative distribution preference. Please get rid of Homebrew completely as a dependency for aws-sam-cli. It is completely unnecessary.

[dwhitla]

apt get install

[darrenweiner]

Agreed, brew is not the recommended approach for linux. Pip makes a lot more sense, and will perform much better in a ci/cd pipeline. Brew is incredibly slow.

[SpyderDave]

Agree completely. Homebrew is terrible, and since no other AWS tools (awscli, codedeploy agent, ssm-agent etc) require homebrew it seems rather strange to be an outlier. I am so happy I found this post as I was struggling to get the sam-cli installed properly in a build-agent which uses virtenv's . I switched the install to a simple pip3 install --user and all is well now.

[oonisim]

It is suggested that pip does not install up-to-date SAM CLI.

• https://stackoverflow.com/questions/60718834

[jfuss]

@oonisim That is incorrect. The person suggesting that is probably trying to install into an unsupported version of python and pip is selecting an old version that was 'allowed'. You must be using Python3.6+.

[jfuss]

Just an update note on this thread. We are aware of parts of the community not happy with installing Homebrew to install SAM CLI. To provide some context here: when we initially went down the path of installers, Homebrew was a good way to reach a lot of the community. Mac was initially the target but then we learned that it can also support Linux. This allowed us to reach both platforms with a 'single' installation option. We still do not recommend PIP, as it does not provide a reproducible installation and is very easy to screw up your python environment (among other things).

We are looking at improving this experience, most likely targeting Linux first but none of this planned yet, as the team is still discussing things. The two likely options are providing an installation script that could install SAM CLI in an isolated and reproducible fashion (most likely still require a supported version of python on the system) or follow the footsteps on the AWS CLI and vend a single binary that has python bundled with it (similar to how we are currently doing MSI's for windows).

[dwhitla]

@jfuss - That sounds reasonable. However ...

I strongly recommend pip. It has provided a perfectly reproducible installation every time I have used it. For the record, pip does what you tell it to do via your requirements manifest. If you restrict its dependencies to exact versions - including the transitive dependencies - then you get an identical installation every time (unless a dependency is no longer available - which is desirable behaviour in this case). If you restrict dependency versions instead to ranges, you may get different versions - but if you have good reason for the ranges then you get what you are expecting for the expected reasons.

As for it screwing up your python environment - if you're using python you should already understand how to configure its environment for you particular operating system. If you don't, then you already have a bigger problem to resolve. Rather than aim for the lowest common denominator at additional inconvenience for people who know how to use their tools, I would recommend following accepted best practice for the environment tool set you are targeting - in this case Python. Understanding how to use that tool set is a separate issue for the user to deal with that is just part of being a developer. No-one is expecting you to handle that too.

If you use or develop with Python on a nix OS and you aren't sandboxing your tools using virtual envs you are begging for a broken system. If you use and develop with Python on a nix OS and you don't already know this you should probably stop now and find another vocation.

pip is actually a better tool for this purpose even than the various Linux package managers, because it allows this kind of per-environment sandboxing that is totally within the developer's control. It is also the cross-platform "'single' installation option" you were expecting - incorrectly - in Homebrew.

FWIW I $%^&ing hate Homebrew on Mac. It is a mess, makes a mess of your machine, and should be avoided completely. Pushing it on Linux, where you already have the much better native package managers that Homebrew tried to imitate for Mac users, is just a really poor choice.

[dwhitla]

@jfuss - Please don't build an installer. The existing ubiquitous python tools are already available in CI/CD systems. Installers are also notorious for leaving crap all over your system when you stop using the tools they installed. At least the existing tools are well known, well understood and well documented. Rather than reinvent another wheel I suggest your time would be better spent, for those served by an installer, documenting best installation practice to help the noobs avoid the pitfalls you are trying to code around for them. Just my $0.02

[jfuss]

@dwhitla Thanks for your input however pip does not meet out needs for distribution. It is great for libraries not so great for tools. The language SAM CLI is written in is an implementation of the tool and while some sets of customers are ok with this, others are not. No one should care about the language and we should not be leaking those details back to customers.

Now pip is not reproducible in the way we would need to support it. Sure, we could pin everything but that causes other problems for some customers. Sure, we could document more steps for customers to 'follow' but we do not think that is right either. More steps, restrictions such as python versions supported, etc that we impose have to be followed by all customers. We want to provide a simple installation experience (download this binary), not make it more complex with additional overhead.

Homebrew is cross-platform, as it support linux and Mac. Not everyone hates Homebrew, but we understand that some of our customers are not happy with it (including you).

[justin8]

I’ve got to say I agree with @dwhitla here. The AWS CLI v2 install process that you’re praising I have to google every time, I need to download a random zip, it’s outside of any standardized package installation tool, it’s not updated automatically, if I want to verify the integrity of the download (which is built in to pip and most other package managers) then I need to follow another 5 step process that I have no chance of memorising on top of the 4 step installation process.

Home brew provides some of these and works well for Mac, but it’s incredibly slow and hacky at best under Linux. It’s also commonly installed on a total of zero Linux distributions.

All of this makes it very cumbersome compared to pretty much all the competitor tooling.

[dwhitla]

@jfuss - I have been in the situation of deploying end user tools implemented in python and understand that problem. I agree it is not the easy single click experience of say installing an MSI. But the important point for me is that it works in all my CI/CD environments which are virtualised and instantiated on-demand. That is to say all tools required for a build are installed during the build by an automated process - in this case pip.

If you absolutely insist on building your own installer please ensure this use case is considered and supported or you will break about half of your existing users' CI pipelines.

I also don't have Homebrew available on these virtual machines and don't have the option of installing it. Python (and pip) on the other hand, is part of the base Linux image and is the ubiquitous scripting Swiss army knife on most modern *nix platforms. If I have to instead curl some self unpacking zip file from AWS, while that doesn't tick my box for integrity verification which I get for free with existing tools, it is still preferable to hacking up some Homebrew installation with all of its kitchen-sink dependencies for each and every CI build every few minutes.

With pip I can also control the installation such that it doesn't overwrite or mask system libraries or in any other way threaten to brick my development machines. I contain everything in a virtual env and when I stop using AWS CLI I will just delete the virtual env. I have developers on various Linux distros, Mac, BSD and Windows. How do you plan to support BSD and Windows users with Homebrew?

[dwhitla]

Not everyone hates Homebrew, but we understand that some of our customers are not happy with it (including you).

Not everyone hates Windows, but the fact that it was a completely unnecessary dependency for managing Cisco routers using their supplied management tools made me stop buying Cisco. The point here is you don't need it but have made it a dependency.

[gcrum-bc]

@jfuss homebrew is a mac specific package manager. It was conceived because the only way to install opensource packages was to compile them from source if a binary didn't exist before homebrew. Linux distributions which are the dominant server platform already have pre-existing package management that dates back decades before homebrew's conception. That said pip install "is" cross platform because it supports more than 2 (mac, linux) OS's. So to say that homebrew is cross platform is incorrect. Also packages in homebrew are compiled for mac osx and linux as a secondary thought. That said I would never install homebrew on any of the servers I manage for 1 package or any package I can install directly from the opensource community. I prefer to keep my systems lean and use packages compiled specifically for the linux kernel. BTW when was the last time you encountered a Mac OSX server? I find this requirement compelling to convince the developers I work with to switch to terraform instead.

[mckalexee]

@jfuss can you explain in more detail on why the aws-sam-cli is unique enough from other cli tools that it needs Homebrew for Linux?

I have about 10 years of Linux sysadmin experience, and this is the only tool I've seen where the installation instructions require you to install Homebrew. IMO the justification feels weak.

Not everyone hates Homebrew, but we understand that some of our customers are not happy with it (including you).

This is understating the issue and I feel like it's a bit dismissive. I don't see a single person in this thread agreeing with you and if you look at the reactions on your comments and look at the reactions on @siwyd's comment you can see a stark difference in what users want.

The consensus is clear that requiring homebrew is not desired.

[jfuss]

So again, we know the Linux community is not happy with Homebrew and we are working on adjusting that (no timelines to share at the moment). Though I will share, we are looking at following what AWS CLI has done with AWS CLI V2. It's more useful to talk about that then the back and forth about the hate for Homebrew.

@mckalexee I am not claiming we are unique. What I am claiming is that we added it because it was easy at the time (no overhead for us to support LinuxBrew) and gave us what we were after, that is a deterministic way to install the CLI. Pip only give us that if we pin everything, and while I would actually love to do that it's is also not reasonable, as it requires everyone to follow Python best practices, understand which pip they are using, ensure they have a supported Python Version, etc. I am not claiming PIP or LinuxBrew is a prefect solution but its what we have right now. We recommend LinuxBrew in our docs because it gives us the deterministic installs (through brew bottles). Right now we do not have a plan to remove PIP but that might change in the future. So use as you will.

The points I was attempting to make through all of this is:

• We hear the Linux community is unhappy with the current recommendations and have plans to address that.
• We originally choose to support Linux brew because it was a no-op to support and gave us a deterministic way to vend the CLI.

@mckalexee My point on not everyone hates homebrew is that we do have users (on Linux) using it. This isn't meant to be dismissive and I sorry if it came across that way. I know people here want a different solution but that doesn't mean there isn't a population that is ok with it.

edit: missing a key word "not" in "this isn't meant to be dismissive"

[gcrum-bc]

@jfuss I'm not sure if you may have interpreted my words as having a tone that reverberated hatred towards homebrew. That is not the case and I apologize for that. My concern is that when I build servers I want them as stable as possible using what has been designed for them to make them work. This is known as "Best Practices" in building servers. I think homebrew for the Mac OSX ecosystem is actually a good thing. Just not for Linux. I would not choose homebrew to install apache2, curl or netcat on a Linux server because this would cause a conflict between package managers (bad practice) so why would I install it at all. I have never run into another DevOPS/Systems Administrator that uses homebrew on their Linux servers in the 20+ years I have been in this field. I wonder what the response would be if the homebrew team requested that their package manager be included in a distro's ISO? Why?

Also if one is using Python then they are following best practices to begin with. Otherwise they do not know what they are doing. Python 3 is now installed by default on multiple Linux distributions. So that leaves the package maintainers to make sure that requirement is simply stated.

Keep in mind that what works and feels right for you on your Mac OSX workstation isn't going to be the same on Linux servers. I started out as a Mac Classic user. I think Mac OSX is a beautiful OS however the shell it provides does not meet my expectations so I use Linux instead. That difference alone has taught me this lesson.

Now enough of my opinion. I now ask you a question.

Why was Python chosen if Python best practices for installing and maintaining it doesn't meet the project's needs? You can't have one foot on the boat and the other off to ride on the ship.

[jfuss]

@gcrum-bc First off, sorry this is a long post but I want to make sure I have enough details here. No worries, it is hard to communicate correctly over written words, I struggle with this myself. Knowing how someone is going to read and interpret a message is hard. I didn't take anything harshly, it is just frustrating on our end. We are in the state we are in and we can't change that but we can improve it. We have plenty of feedback that Homebrew on Linux is not enough, for plenty of valid reasons. For example, I never cared about Homebrew overriding python (on Mac), until I started needed to install specific versions of python. So I understand how some of these things can really impact you guys and not trying to downplay any of that. With that said, I really want to move away from talking about Homebrew (and pip) and more towards what we need to do to make the experience better.

Why was Python chosen if Python best practices for installing and maintaining it doesn't meet the project's needs? You can't have one foot on the boat and the other off to ride on the ship.

The SAM Spec was written in Python (first) and it made sense to integrate the two. Python has a better open source community appeal. What I mean by that is, is a lower entry for people to learn and contribute back (if they so choose). Also, the team was pretty comfortable with the language, so it was easy to hit the ground running and not have to learn another language, best practices, etc. I think those were the main reasons we ended up in Python.

To your "doesn't meets your needs" question: This is a very valid question. Honestly, early on we thought that pip was enough. This worked out until we learned about some of the deeper problems. Some of the main issues where:

• new versions of dependencies were getting installed and breaking things for customers
• customers installing into places with other packages. This caused conflicts between tools (like AWS CLI).
• You can easily install into multiple places due to not know which pip you are using at a given time. This can confuse some systems (like Windows) and cause customer confusion.

Pip would be fine if we could pin our whole dependency closure but we can't do that because people don't follow best practices.

This is when we moved to 'installers'. We followed the AWS CLI's footsteps on the MSI for Windows (which has worked very well for that user group). We felt Homebrew was pretty common in the MacOS space, so that generally made sense. During this, we learned about Linux Homebrew and thought the bang for the buck was good (at the time). From then, we have learned that Homebrew isn't enough for all customers. We have feedback on Mac and Linux for this, though Linux is the most unhappy group.

I think it is weird you say "you can't have one foot in the boat and the other off" because there are plenty of tools that do not distribute through pip (for example docker compose). So it is possible to do this. We are looking at is following what AWS CLI V2 has done. They create a binary with pyinstaller and then distribute that. We think this can provide a much better experience to the Linux community. This would also make it easier to go down the distribute through yum, apt-get, etc, which I feel is much much more appealing to everyone.

We are still in early phases of figuring this out. I would love to start talking about that and getting feedback from you guys. It will really help us understand what is needed and desired. If anyone is in the AWS Developers Slack Workspace and wants to chat deeper, feel free to reach out to me there as well. We do really want to improve this and will need your help to make sure we meet the needs of the community, while keeping our requirements of reproducible installs/artifacts, simple install experience, etc.

[gcrum-bc]

Sorry I posted this using the wrong account. :)

@jfuss Thank you. I now understand a little more about this conundrum you and your team have. Also I take the boat analogy back. Saying something is written in stone, I should know better. Thanks for the reminder.

[RickC37]

Since some weeks the AWS CLI provides a self-containing binary for linux.

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

Even if the CLI requires python, my environment is not forced to have python installed. This makes it much easier for me to use the CLI within my CI/CD pipeline or any other environments where python is just there to install the AWS CLI.

Therefore I would like to have the SAM CLI the same way. A simple binary file containing everything which is required. Unzip, Install script, done.

[nikgibbens]

I'm an engineer that oversees Docker build containers for our CI/CD. We install multiple versions of Python using Pyenv to provide multiple versions of Python to the users for their pipelines. A pip installation of this tool would force them to use a specific version of Python. The Homebrew installation has definitely manifested problems with installing standard Ubuntu packages and Pyenv. The AWS CLI V2 installation being self contained has been a fantastic model where we have environments that need to have isolation between different tools. I vote for this last mentioned method.

[pauldraper]

$ pip3 install aws-saml-cli
Collecting aws-saml-cli
Exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/usr/lib/python3/dist-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/lib/python3/dist-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib/python3/dist-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/usr/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/aws-saml-cli/

EDIT: And it suddenly started working.

[maxim-lobanov]

+1 for this issue.

Both available options are not convenient for using in CI/CD systems and local usage:

• Brew on Linux is uncommon and unpopular and installing brew only for this single tool looks overcomplicated
• Pip installation is not ideal too because it depends on pre-installed Python and brings additional dependencies / packages.

I think you should take over experience of AWS CLI V2 and provide self-containing binary for this tool Based on the likes count, it is one of the most popular issues in repository, could you please prioritize it?

[pbecotte]

At a minimum- just document that the pip package exists? I am a python developer- overwriting my system python using homebrew is not an option. However, I do know how to install python packages without messing up my global package space. I figured out the pypi package name and installed this just fine- but would have saved a bunch of steps if it was just listed in the docs with whatever warnings you want to include about not messing up your environment with pip.

[mhart]

I would also add that pipx is the recommended method of installing global tools by Python itself – and solves a lot of the problems with dependency concerns of pip.

[plantusd]

is there any ETA on the issue?

[loscil06]

I think the docs are bad written

running pip install aws-sam-cli installs everything OK

Doing this combined with pipx looks like it works, try: pipx install aws-sam-cli

[tomjowitt]

This decision falls well below the engineering standards I've come to expect from AWS. Trying to integrate aws-sam-cli into our CI workflows was an exercise in installing a lot of unnecessary software.

Not only does Homebrew override system defaults, but it also requires Ruby to be installed plus a lot of other packages.

For context, our CI system is based on Alpine Linux and has been consistently working well with pip and the aws-cli tooling. There were grumblings of discontent from our Linux developers when SAM moved to Homebrew but it was a local install so we got over it despite a fair amount of pain with Python versions. The issues with Homebrew really came to a head when we needed to run some sam invoke functions in CI on Alpine. The amount of effort to get it working compared to the simplicity of pip was crazy.

It feels like this decision was made purely for OSX developer productivity and ignores decades of solid Linux package management and those of us who maintain infrastructure.

If pip is no longer considered suitable, and there is no case for using apt/yum/apk, please at least provide binary builds alongside Homebrew so that those of us who have to manage Docker images and infrastructure can keep some semblance of sanity in our builds.

Thanks for listening.

[dwhitla]

It feels like this decision was made purely for OSX developer productivity and ignores decades of solid Linux package management and those of us who maintain infrastructure.

Nah - its garbage on OSX too. The team chose it because they believed it to be a cross-platform package management solution - which it purports to be but falls short of delivering in a safe and reliable way.

[Arnaldo-Manjate]

HI @jfuss

so 3 Months later is there any update on a solution for the linux community ?

[Fingalzzz]

+1 for this one. Waiting for an update or an ETA.

[soksy]

Just an update note on this thread. We are aware of parts of the community not happy with installing Homebrew to install SAM CLI. To provide some context here: when we initially went down the path of installers, Homebrew was a good way to reach a lot of the community. Mac was initially the target but then we learned that it can also support Linux. This allowed us to reach both platforms with a 'single' installation option. We still do not recommend PIP, as it does not provide a reproducible installation and is very easy to screw up your python environment (among other things).

We are looking at improving this experience, most likely targeting Linux first but none of this planned yet, as the team is still discussing things. The two likely options are providing an installation script that could install SAM CLI in an isolated and reproducible fashion (most likely still require a supported version of python on the system) or follow the footsteps on the AWS CLI and vend a single binary that has python bundled with it (similar to how we are currently doing MSI's for windows).

I found a) installing homebrew unworkable on CentOS7 b) it screwed up my Python environment.
I had not thought to look at at what homebrew was doing (never used homebrew or looked at how it worked). Installing SAM was such a pain I started using Serverless Framework instead. Can't believe the choice of homebrew, or the fact that pip can deliver a working SAM in 30 seconds, and doesn't mess with your Python - but isn't mentioned on the AWS How to Install SAM page.

[plantusd]

follow the footsteps on the AWS CLI and vend a single binary that has python bundled with it (similar to how we are currently doing MSI's for windows)

Really looking forward for this option

[Duncan3142]

@jfuss Eagerly anticipating this option, is it possible to provide a road-map / ETA?

[crojasd00]

Use pip install --user aws-sam-cli you don't want to "screw up your python environment (among other things)".

[awsjeffg]

@Duncan3142 soon! Work is being done in the current sprint, but ETA is still TBD and may be complicated by reinvent / holidays a bit. Definitely before EoY.

[jake-scott]

Agree with the Pip comment.. this works :

python -mvenv .venv
. .venv/bin/activate
pip install aws-sam-cli

[AmanL04]

Is there any update on this issue? I feel the Linux community finds this solution more than just "lacking".

[jverce]

Is there any update on this issue?

I feel the Linux community finds this solution more than just "lacking".

Meh... We're used to it 🤣 So far using pip to install this has worked fairly well.

[aksel]

Is there any update on this issue? I feel the Linux community finds this solution more than just "lacking".

Meh... We're used to it rofl So far using pip to install this has worked fairly well.

I mean, isn't this is pretty much par for the course when it comes to Python packages?

[backpacked]

Homebrew is a nightmare at best on Mac. I hate the fact that it changes the permissions on /usr/local. While that directory is supposed to be 'usr' accessible, it does not mean that I will change the permissions so that any future install of apps into that directory will not need 'sudo'. Homebrew hates sudo. Sudo has a reason to be around. It lets me decide if I want to give access to someone coming in my domain with a lot of privileges versus stopping them at the gates. We do not allow everyone in our house without some verification. Do we?

AWS should not force Mac users to use homebrew either. Prefer pip over homebrew.

[awsjeffg]

Thanks for the feedback on this issue; we have work currently underway to provide a better install experience via pyinstaller. In the near term we will continue publishing to brew to support people that are already using it, as well as pip but the core artifact will be produced using pyinstaller and hopefully allow for a better experience for Linux and macOS users, and it is my hope this will also en courage downstream packages to be published for Linux distributions eg yum and deb repos similar to how AWS cli is commonly available.

[martimors]

pip3 install aws-sam-cli works fine for now, but it would be nice with some packaged binaries that can be installed like we do the AWS CLI.