search

brefphp / bref

Serverless PHP on AWS Lambda
https://bref.sh
MIT License
3.15k stars 365 forks source link

Bref 2.0: rewritten runtimes #1299

Closed mnapoli closed 1 year ago

mnapoli commented 2 years ago

This issue is about one big change in Bref 2.0: new (rewritten) runtimes. This issue only covers this change.

Background

@deleugpn worked a lot on the runtimes/layers, and started working on #1078 to completely refactor how they are built.

The PR stalled earlier this year, I am picking it back up and finishing the changes for Bref 2.0. Feel free to read the PR, but it doesn't reflect 100% what we have now.

Goals

  1. Easier to maintain and contribute

  2. 1043 (thanks to the above)

  3. Improve Bref compatibility with Terraform, AWS CDK, Pulumi…
  4. Allow newcomers to test Bref in the AWS console without installing it

Not a direct goal, but a pleasant side-effect: makes it easier for advanced users to create their own runtime/customize the existing runtimes.

User impact

Everything will continue to work like in v1 (there will just be more "possibilities"), no code changes should be necessary.

It will be a new major version because we are changing the way PHP is compiled, and it might impact extra layers or extensions. The goal is to be safe.

Details

1. Easier to maintain and contribute

How:

The work in progress is in this repository: https://github.com/brefphp/aws-lambda-layers This repository contains all the git history of Bref (runtime/ directory) and #1043 on top of it. However newer commits are not included (too hard to rebase everything).

Trade-offs:

2. Add Graviton/ARM support (#1043)

How:

Trade-offs:

3. Improve Bref compatibility with Terraform, AWS CDK, Pulumi…

How:

Want to see what this looks like? Here's an example with the "function" runtime:

With the solution described above, the layer is 100% decoupled from Bref's codebase. It is possible to upgrade layers and Bref separately.

Trade-offs:

4. Allow newcomers to test Bref in the AWS console without installing it

How:

Indeed, the user codebase runs in FPM and is not coupled to the Bref API (when using the FPM layer). That means we can create an FPM layer that contains all the code it needs to run. Users can create an empty Lambda function with a single index.php and things will work.

Prototype of externalizing the FPM runtime: https://github.com/brefphp/php-fpm-runtime (the FPM layer contains this code, Bref users do not have to install it anymore).

Trade-offs:

Feedback is welcome.

I am currently building layers and will post layers ARNs soon if you want to test things out and participate.

shouze commented 2 years ago

@mnapoli :100: with everything excepting maybe the monorepo codebase split. Not sure this will be easier to maintain with multiple repos.

shadowhand commented 2 years ago

The biggest weakness I see with these changes is the lack of parity between x64/ARM. As a significant portion of developers move towards Apple Silicon chips, I think there is real benefit to having x64/ARM be as equal as possible. Is there a specific reason to use Remi's repos, over (for example) Alpine Linux or Fedora?

shouze commented 2 years ago

Maybe because Amazon Linux 2 images are very closed/derived from centOS if I remember well?

mnapoli commented 2 years ago

💯 with everything excepting maybe the monorepo codebase split. Not sure this will be easier to maintain with multiple repos.

Yes, very good point, and I think I'll tend to agree with that.

I want to explore options to have a monorepo, I am just afraid about how to make it work (git subtree split scares me, not sure where to start and how to deal with tags/releases).

The biggest weakness I see with these changes is the lack of parity between x64/ARM. As a significant portion of developers move towards Apple Silicon chips, I think there is real benefit to having x64/ARM be as equal as possible. Is there a specific reason to use Remi's repos, over (for example) Alpine Linux or Fedora?

The main limitation is that we have to install stuff with yum, could we do that with what you suggest? Or maybe I am just missing some opportunities 🤔

Also I wanted to double check that Amazon Linux is based on Red Hat, but stumbled upon this: https://serverfault.com/questions/798427/what-linux-distribution-is-the-amazon-linux-ami-based-on#comment1011304_798428

Amazon Linux hasn't been compatible with RHEL anything in quite a while. Indeed, here we have at least hundreds of questions from people trying to use packages for RHEL/CentOS on Amazon Linux and them not working, for precisely this reason.

@deleugpn or anyone else: could it be a problem with Remi's repo?

deleugpn commented 2 years ago

The biggest weakness I see with these changes is the lack of parity between x64/ARM.

I look at this more as: Our parity today is 0. There is no support for ARM. These changes will bring parity between ARM and x64 - ~5 months, which is an improvement over the current state.

As a significant portion of developers move towards Apple Silicon chips, I think there is real benefit to having x64/ARM be as equal as possible. Is there a specific reason to use Remi's repos, over (for example) Alpine Linux or Fedora?

AWS Lambda runs on top of Firecracker and is provisioned with Amazon Linux 2. Alpine Linux is very far away from compatibility (MUSL). I have not looked at Fedora, I just followed this post and things started to work as I expected. I looked at Remi's reputation and release speed and was really impressed with day-1 support for PHP releases.

The 1st thing that I spent most of my time was to create a POC where it is indeed possible to have Bref not compiling PHP and making the experience smoother for contributors. The 2nd thing I looked at was how to write some automation tests to prove that the PHP binary and all of the .so files extraction guaranteed a working environment. My reasoning for focusing on this was so that if something changes in the future and breaks the functionality of the PHP binary, it is easy to catch.

I think overall this sets the scene for something that someone could contribute: propose a new installation and extraction process (without making changes to the automation tests) where the PHP binary gets proven to work and it would be less of a burden for Bref maintainers to approve and merge such a change.

In the end, ARM64 support was what made me look into it, but it stopped being my focus ever since I saw that Amazon and Remi don't fully support it. My take was that with Apple & AWS pushing for M1/Graviton, it would be a matter of time until we have someone maintaining PHP for ARM with the same level of quality Remi maintains x86. So far nobody has stepped up AFAIK.

@deleugpn or anyone else: could it be a problem with Remi's repo?

I think if it does become an issue, the automation tests will kick in and protect us from a broken release, but I take the point that putting our money there and then being locked in without a way to get out could be problematic. To be honest, I think there will always be people wanting to run up-to-date PHP version on Amazon Linux 2 and since Amazon themselves will not provide that possibility, someone will. My argument in defense of installing from a distro instead of compiling here remains that this is not just AWS Lambda, but rather a much broader picture that affects people using Amazon Linux on EC2 or even containers.

shouze commented 2 years ago

@deleugpn 1st of of all, many of your choices were very wise.

:+1: on parity, this is far better to move from nothing → something than staying at Nothing. Furthermore, arm64/Graviton λ are not deployed in all AWS regions so far. For example in my company we can benefit for our preprod env (us-east-1) but not for our prod (eu-west-3) at the moment.

:+1: about relying on Remi's as it introduces binary artifact seggregation. We will see on longer term if it's more pertinent to compile back again bref binary, but by keeping a such decoupled way.

So the main issue to solve & focus right now is probably not the way you've done in the big stalled PR, but the way to split it & deliver all the good things step by step. You can count on me @deleugpn & @mnapoli to bring little help on it.

shadowhand commented 2 years ago

@deleugpn first of all, I am very grateful for your work on this and believe it represents a significant improvement for Bref. Thank you for explaining your choices. I do understand this will move from no parity to some parity, and I appreciate that. I am just asking if there is a known path to full parity between ARM and x64.

Also, ARM support is not just Graviton, but also bref/php-81-fpm-dev Docker container, which is the only image (along with bref/fpm-dev-gateway) in our Docker Compose stack that is still on x64.

mnapoli commented 2 years ago

Monorepo/multi-repo

Here is a visual of the dependencies I have in mind with the current approach:

Current

Maybe a simplification could be to keep everything runtime-related (except FPM) in bref/bref. The downside is that it means installing the Bref CLI (and dependencies) in production, but I may have ideas to simplify that 🤔

Here's the simpler alternative that I'll try exploring:

Alternative

Regardless, in both approaches the runtime scripts (Dockerfile and other) would still be in a separate repo (with separate versioning, that's the whole point). Though to simplify things maybe we can use git submodules (I stumbled on an example here and it was OK in the sense that it allowed newcomers to easily discover sub-repositories of the project: https://github.com/dunglas/frankenphp).

Linux distribution

At this point it seems like our options are:

No other distribution seems compatible, but feel free to research if some could actually work.

The second option is IMO the best compromise, as we avoid the complexity/risk (bus factor) of the compilation, yet we can still provide the best binaries available.

Also, ARM support is not just Graviton, but also bref/php-81-fpm-dev Docker container, which is the only image (along with bref/fpm-dev-gateway) in our Docker Compose stack that is still on x64.

Is that causing any issues? Personally on M1 I get a warning, but that's it.

deleugpn commented 2 years ago

Just wanted to add this here: https://github.com/remicollet/remirepo/issues/214

Not sure yet if it's possible to use it with AL2 and/or how to do it, but there's a chance that parity could be reached at some point.

mnapoli commented 2 years ago

I want to setup the build ASAP so that testing and iterating becomes easier.

First step: I created the v2 branch.

I also published beta layers, you can find the versions here: https://github.com/brefphp/bref/blob/v2/layers.json

Note The AWS account for v2 layers is different. That way we can continue publishing v1 layers without messing up version numbers, and keeping it easy for folks to stay on one version and upgrade. Here is the v2 account ID: 534081306603.

I have updated runtimes.bref.sh and you can find v2 layers there:

https://runtimes.bref.sh/?region=us-east-1&version=v2

I have tested layers as I built them, but haven't really tested those that were published tonight (TBH I'm in a rush tonight but wanted to provide an update). Feel free to test them.

Also feel free to clone https://github.com/brefphp/aws-lambda-layers and start building. It should be much easier to build, you only need make and Docker. Takes 4 minutes to build everything locally with 1 command. Then you can play with Docker images locally, or publish to 1 region easily.

Next I'll try to come up with a concrete TODO list, because I see so many of you that want to contribute and that's awesome!

PS: if one wants to have a go at rewriting https://github.com/brefphp/runtimes.bref.sh on newer PHP versions, replace SAM with Serverless Framework, etc. Feel free, cause updating that website sucked 😅

Edit: oh, runtimes.bref.sh doesn't list ARM layers. Try adding arm- as a prefix for PHP 8.

image
shadowhand commented 2 years ago

Also, ARM support is not just Graviton, but also bref/php-81-fpm-dev Docker container, which is the only image (along with bref/fpm-dev-gateway) in our Docker Compose stack that is still on x64.

Is that causing any issues? Personally on M1 I get a warning, but that's it.

@mnapoli no, it doesn't cause any specific issues, just a minor annoyance to have 2 warnings in our Compose stack.

mnapoli commented 2 years ago

Bref 2.0 overview & contributing

All Bref 2.0 issues are now grouped on this project: https://github.com/orgs/brefphp/projects/1

If you want to contribute, feel free to look at the help-wanted issues. Feel free to also open issues to discuss more changes.

Layers

https://github.com/brefphp/aws-lambda-layers now has CI (build and tests) on main and pull requests. You can now send pull requests and everything will be tested.

ARM layers are also building.

Great news too: the build takes 4 minutes, instead of an hour like Bref 1 🎉 This looks like using GitHub Actions is very viable here, even to build ARM layers.

Note that automatic publication of layers and Docker images isn't implemented yet (only the CI).

Feel free to watch the repository if you want to follow the pull requests.

Package splitting

I have experimented with getting rid of https://github.com/brefphp/php-runtime and keep that code in bref/bref (as described in my comment above).

That seems to be working out well, this is great.

In the end we might only end up with:

deleugpn commented 2 years ago

@mnapoli no, it doesn't cause any specific issues, just a minor annoyance to have 2 warnings in our Compose stack.

If I'm not mistaken, all you need to do is add platform: linux/amd64 to your service definition on Docker Compose. The warning is just telling you that Docker will assume a different platform than your computer. If you opt-in to it yourself, the warning will be gone. Not 100% sure though, as I didn't test it myself. I'm making assumptions based on how I think this works.

shadowhand commented 2 years ago

@deleugpn sure, but I don't want to disable the warning. I want to have it fixed. 😸

mnapoli commented 2 years ago

🎉 just wanted to share some progress here about automation.

Layers are in https://github.com/brefphp/aws-lambda-layers

Whenever a new release is created in https://github.com/brefphp/aws-lambda-layers (e.g. new PHP version):

All this process (excluding manual bref release) takes less than 10 minutes, happens publicly on GitHub Actions, and is fully automated 🎉

So happy with the result. And so far I've been building a lot, we aren't even close to hitting GitHub Actions free quotas.

Screen-000807
mnapoli commented 2 years ago

Preliminary benchmarks for new layers: https://github.com/brefphp/benchmarks/pull/5

mnapoli commented 1 year ago

Getting close to finishing!

Today I worked on the dev images and I have found an interesting approach that could simplify local development. Curious to get your feedback: https://github.com/brefphp/aws-lambda-layers/pull/38

mnapoli commented 1 year ago

Closing this one as well, a v2 beta has been published: https://github.com/brefphp/bref/releases/tag/2.0.0-beta1