search

heroku / heroku-buildpack-python

Heroku's buildpack for Python applications.
https://www.heroku.com/python
MIT License
976 stars 1.84k forks source link

[feature request] Skip python installation #775

Closed duanehutchins closed 4 years ago

duanehutchins commented 5 years ago

Heroku images come preinstalled with python2 and 3. If the system installed version (e.g. 3.6.5 on Heroku-18 stack) meets our needs, then there's no need to install python locally, and skipping it can reduce slug size and save on dyno memory usage. The installation of pip and requirements will then continue using the system python.

We can signal for this feature by using an environment variable or some other method.

kennethreitz commented 5 years ago

This is a bad idea.

duanehutchins commented 5 years ago

@kennethreitz why so?

CaseyFaist commented 5 years ago

thanks @duanehutchins! 😄

I can understand wanting to reduce the slug size, but since we're talking about system python, what you're actually requesting sounds more like you want the stack image itself to be smaller. System python isn't stored in your app's slug - and so removing it from the stack image wouldn't actually make your slug any smaller.

This request is out of scope for the buildpack, as buildpacks in general are not intended to mutate the underlying stack, only add to them. This would also mean components of the buildpack could not be converted to python, which is under consideration.

I'm open to exploring other ways to optimize slug size though - so while I'm closing this issue as out of scope, please feel free to open another issue about slug size minimization if you'd like to continue brainstorming. (If you do, please add the feature request flag to help me more quickly triage issues! 😄)

CaseyFaist commented 5 years ago

I got flipped around while checking for potential side effects - you're specifically asking to not install new python, NOT remove system if current version doesn't match. That's a different conversation. Your app would then be tied to a particular stack, as stacks can have different python images and may get upgraded on security release, but it's worth exploring if those are acceptable trade offs for more slug space.

I don't currently have bandwidth for a proper proof of concept test, but this could be useful to explore during some of the planned refactoring, so I'm going to reopen with the feature-request tag to circle back at a later date 👍 🌮 thanks for bearing with me!

duanehutchins commented 5 years ago

Thanks Casey! I have early protocode for this feature working, but it's not read for a PR. Once I get some extra "bandwidth" (I like that term) myself, I'll circle back.

edmorley commented 5 years ago

(Thoughts from a random passer-by; take with a pinch of salt and all that... :-) )

It seems like there might be a few "gotchas" for end-users if this approach were taken:

  1. if the stack changed, the buildpack would then either have to: (a) switch from system Python to a bundled Python to keep the same Python version, (b) use the new system Python which would be a different Python version (since system Python version varies across Ubuntu releases). Either option would cause potential behaviour differences.
  2. the buildpack would have to force system Python to relocate the per-user site-packages into the slug/or else move the directory retrospectively/fix up package paths etc - and this implementation would be different from that used for vendored Python, so have the potential for behaviour differences between the two (the handling in this area is already pretty complex - eg eggpath-fix, eggpath-fix2 amongst others).
  3. system Python is compiled/packaged by Ubuntu, whereas the vendored Python uses the custom compile scripts in this repository - so uses different compile options etc (and as seen in #305, the choice of options can make a massive difference to app behaviour). This means that a user's app could break in ways that are surprising (unless one knows the internal implementation details of the buildpack and when it uses system Python vs not) when eg upgrading to a new point release of Python (which is then not system Python) or even when staying on the same Python version but changing stack (ie due to (1)).

The added buildpack complexity to handle this would then only benefit the small proportion of apps whose Python version happened to match the current system Python version. In addition, the distro system Python version is typically really out of date, so if anything IMO the buildpack should be encouraging them to use a newer version instead, at which point this feature wouldn't help them.

dzuelke commented 5 years ago

I agree with @edmorley; the added complexity, and the maze of potential pitfalls, is not worth the likely tiny amount of cases where the Python version from the stack happens to be the right one.

duanehutchins commented 5 years ago

The Heroku-18 stack currently uses the same python version which is recommended (and defaulted) by this buildpack. It's reasonable to assume that there would be a fair number of cases where the stack version would be the right one.

Yes, changing the stack would change the python version, but stacks aren't changed lightly, and the dev can still specify a local python version, rather than using the system one.

The goal of this request is simply to allow the option to skip python installation, which, for many, would work perfectly fine while reducing the slug size.

dzuelke commented 5 years ago

That's just coincidence because that stack is new. The stack right now has, and forever will have, 3.6.5; the buildpack has 3.6.6, so that's already a bunch of fixes not available on the stack Python (with the exception of any security backports done by the upstream OS maintainers).

The bigger issue is egg paths and site-packages relocation. It's already a bag of hurt, and having two code paths for the two "models" of Python increases the maintenance burden, while offering little benefit - the size is already not massive compared to the allowed slug size, and could be reduced further through stripping of binaries and libraries.

CaseyFaist commented 4 years ago

Closing this issue, as I don't believe it's a stable choice to use stack-supplied Python in production:

I wouldn't recommend using system Python for production code in on-prem servers or cloud IAAS, so I don't believe we should ignore these issues at this devops abstraction layer either.