search

brefphp / bref

Serverless PHP on AWS Lambda
https://bref.sh
MIT License
3.11k stars 366 forks source link

Upgrade to the new AWS Linux AMI 2018.03 #332

Closed mnapoli closed 5 years ago

mnapoli commented 5 years ago

AWS announced that Lambda is migrating to a new version of their Linux AMI: https://aws.amazon.com/fr/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/

We need to update the layers so that we compile PHP against the 2018.03 AMI.

deleugpn commented 5 years ago

Maybe go straight to Amazon Linux 2?

mnapoli commented 5 years ago

This is not mentioned in the blog post right? I would make more sense to target the AMI that will actually be used in production.

deleugpn commented 5 years ago

You're definitely right. I noticed NodeJS 10.x will be running on Amazon Linux 2 and thought we could do the same, but we don't have control over which OS is being used by the custom Runtime. For some reason I thought that if we compiled PHP using the AL2, the layer would use AL2.

atrope commented 5 years ago

@mnapoli do we Must do that? I've just opted in to the new AMI and everything is working as before.

What can we do to test that we must recompile PHP for this AMI? What would be the benefits?

mnapoli commented 5 years ago

If everything is working that is perfect!

I would say that if nothing breaks that's a good sign. However we should still update the references to 2017.03 into 2018.03 in the compilation scripts (just to be safe).

mnapoli commented 5 years ago

We gave it a try with @Guillaume-Rossignol but the compilation of PHP fails, which is very weird:

/bin/sh /tmp/build/php/libtool --silent --preserve-dup-deps --mode=compile g++ -I/usr/include  -Wno-write-strings -D__STDC_LIMIT_MACROS -DZEND_ENABLE_STATIC_TSRMLS_CACHE=1 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fno-strict-aliasing -W -Wall -pedantic -Wpointer-arith -Wwrite-strings -Wno-long-long -std=c++11    -DU_USING_ICU_NAMESPACE=1 -Iext/intl/ -I/tmp/build/php/ext/intl/ -DPHP_ATOM_INC -I/tmp/build/php/include -I/tmp/build/php/main -I/tmp/build/php -I/tmp/build/php/ext/date/lib -I/opt/bref/include/libxml2 -I/opt/bref/include -I/tmp/build/php/ext/mbstring/oniguruma -I/tmp/build/php/ext/mbstring/libmbfl -I/tmp/build/php/ext/mbstring/libmbfl/mbfl -I/tmp/build/php/ext/sqlite3/libsqlite -I/tmp/build/php/ext/zip/lib -I/tmp/build/php/TSRM -I/tmp/build/php/Zend  -fstack-protector-strong -fpic -fpie -Os -I/opt/bref/include -I/usr/include -ffunction-sections -fdata-sections -D_REENTRANT  -g -O2 -pthread   -c /tmp/build/php/ext/intl/dateformat/dateformat_format_object.cpp -o ext/intl/dateformat/dateformat_format_object.lo
In file included from /tmp/build/php/Zend/zend_types.h:27:0,
                 from /tmp/build/php/ext/intl/intl_convertcpp.h:27,
                 from /tmp/build/php/ext/intl/intl_convertcpp.cpp:21:
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_increment_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:446:68: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_decrement_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:500:68: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_add_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:554:80: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_sub_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:657:80: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
make: *** [ext/intl/intl_convertcpp.lo] Error 1
make: *** Waiting for unfinished jobs....
In file included from /tmp/build/php/Zend/zend_types.h:27:0,
                 from /tmp/build/php/Zend/zend.h:29,
                 from /tmp/build/php/main/php.h:35,
                 from /tmp/build/php/ext/intl/common/common_enum.h:27,
                 from /tmp/build/php/ext/intl/common/common_enum.cpp:26:
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_increment_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:446:68: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_decrement_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:500:68: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_add_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:554:80: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_sub_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:657:80: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/ext/intl/common/common_enum.cpp: At global scope:
/tmp/build/php/ext/intl/common/common_enum.cpp:169:30: warning: unused parameter 'ce' [-Wunused-parameter]
 static zend_object_iterator *IntlIterator_get_iterator(
                              ^
make: *** [ext/intl/common/common_enum.lo] Error 1
In file included from /tmp/build/php/Zend/zend_types.h:27:0,
                 from /tmp/build/php/Zend/zend.h:29,
                 from /tmp/build/php/main/php.h:35,
                 from /tmp/build/php/ext/intl/common/../php_intl.h:23,
                 from /tmp/build/php/ext/intl/common/common_date.cpp:22:
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_increment_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:446:68: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_decrement_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:500:68: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_add_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:554:80: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_sub_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:657:80: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
make: *** [ext/intl/common/common_date.lo] Error 1
In file included from /tmp/build/php/Zend/zend_types.h:27:0,
                 from /tmp/build/php/ext/intl/dateformat/../intl_convertcpp.h:27,
                 from /tmp/build/php/ext/intl/dateformat/dateformat_format_object.cpp:25:
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_increment_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:446:68: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_decrement_function(zval*)':
/tmp/build/php/Zend/zend_operators.h:500:68: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), 1, &lresult))) {
                                                                    ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_add_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:554:80: error: '__builtin_saddl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_saddl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/Zend/zend_operators.h: In function 'void fast_long_sub_function(zval*, zval*, zval*)':
/tmp/build/php/Zend/zend_operators.h:657:80: error: '__builtin_ssubl_overflow' was not declared in this scope
  if (UNEXPECTED(__builtin_ssubl_overflow(Z_LVAL_P(op1), Z_LVAL_P(op2), &lresult))) {
                                                                                ^
/tmp/build/php/Zend/zend_portability.h:312:52: note: in definition of macro 'UNEXPECTED'
 # define UNEXPECTED(condition) __builtin_expect(!!(condition), 0)
                                                    ^
/tmp/build/php/ext/intl/dateformat/dateformat_format_object.cpp: In function 'bool valid_format(zval*)':
/tmp/build/php/ext/intl/dateformat/dateformat_format_object.cpp:51:66: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
   for (int i = 0; i < sizeof(valid_styles) / sizeof(*valid_styles); i++) {
                                                                  ^
make: *** [ext/intl/dateformat/dateformat_format_object.lo] Error 1

After searching for this error: error: '__builtin_ssubl_overflow' was not declared in this scope, I found a mention of a minimum required version for GCC (5).

At the moment we install gcc frum yum. Has anyone any idea? @bubba-h57?

bubba-h57 commented 5 years ago

We can fix the build to use the new AMI. We can probably make it run on AL2 as well. We will tackle that later.

The current build will most likely just work due to the nature of our compartmentalized build. That being said, there are just as likely to be potential weirdness where we interact with system libraries.

We recommend not running it in production on the new Linux until we get an official build. We can have a new build by early next week.

Side Note: My laptop battery swole up and Apple took the machine back to work on. I might not have my machine back till Monday/Tue ... so, despite having something to fill the gap, everything I am working on is slowed down considerably until I get my workhorse back.

ejunker commented 5 years ago

If we do switch to a newer Docker image, I wonder if there would be advantages to using lambci/lambda It might even serve as a replacement for bref/runtime/compiler:latest since they have a build-provided tag which I believe includes build tools. In #99 one of the devs of SAM cli even mentioned it.

A sandboxed local environment that replicates the live AWS Lambda environment almost identically – including installed software and libraries, file structure and permissions, environment variables, context objects and behaviors – even the user and running process are the same.

mnapoli commented 5 years ago

@ejunker yes that's a good point! @Guillaume-Rossignol has been adding that in #367 (https://github.com/brefphp/bref/pull/367/files#diff-c52630f7c3061e64ef083ca4efa22d7aR10). The last blocker there is the compilation error with 2018.03.

It might even serve as a replacement for bref/runtime/compiler:latest since they have a build-provided tag which I believe includes build tools.

Oh that's a good point! We did see that image but I didn't think about that. Might be a good way to simplify things hopefully.

bubba-h57 commented 5 years ago

While they do provide some build tools, they do not provide the ones we require. We have used them for building in the past and they always came up short. We should get more mileage from building on the official images https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html and avoiding attempting the builds in such a constrained environment as lambci/lambda .

That being said, lambci/lambda is a fantastic option fo testing our output. We recommend doing precisely that.

To sum up, it would not add value to make the change (in either simplification or quality/reliability) and could very possibly make things more complicated than they are today.

nealio82 commented 5 years ago

I thought lambci/lambda was the AWS provided image!? :O

bubba-h57 commented 5 years ago

@nealio82 nope, LambCI is a continuous integration system built on AWS Lambda - they created a docker image for themselves to test with, and other people started using it.

They create their Docker Image by tarring the full filesystem in Lambda, uploading that to S3, and then piping into Docker to create a new image from scratch – then creating mock modules that will be required/included in place of the actual native modules that communicate with the real Lambda coordinating services. Only the native modules are mocked out – the actual parent JS/PY/Java runner files are left alone, so their behaviors don't need to be replicated (like the overriding of console.log, and custom-defined properties like callbackWaitsForEmptyEventLoop)

As you may guess, tarring up a system and then mocking out modules is great for testing.

bubba-h57 commented 5 years ago

Our build issue is with the Internationalization extension. Particularly the icu4c version that the intl extension builds against.

There have been a handful of noteworthy bugs in the intl extension with finickiness around the icu4c version. We attempted building against a handful of versions, but have yet to nail it down.

However, to demonstrate that this is indeed the root cause, I removed --enable-intl=shared from the ./configure arguments and had no problems building the system. We just need to find that sweet spot of AmazonLinux + PHP + INTL + ICU versions to make it work.

I'm headed out on vacation with my family for the rest of the week, so if anyone else wants to pick this up feel free to take a look at the pull request on our fork to see what has been committed/attempted so far.

mnapoli commented 5 years ago

@bubba-h57 awesome thank you for the details and the diff! Enjoy your holidays!

TheNodi commented 5 years ago

I've been looking into this issues for a few hours now and I think I figured out what's the problem.

I started from @bubba-h57 update-build-runtime branch, which was resulting in a different compiling error related to custom ICU library. Since he was installing both system libicu-devel package and custom ICU version, they were colliding. To solve it, we can add --with-icu-dir=${INSTALL_DIR} flag to php's configure to make it use the correct library. Once we do that, we're back at the original error.

Arithmetic with Overflow Checking were introduced into GCC in version 5 (changelog), while Amazon Linux is using GCC 4.8 (in both 2017.03 and 2018.03). We can solve everything by upgrading our GCC installation, which can be easily done with:

RUN set -xe \
# Download yum repository data to cache
 && yum makecache \
# Default Development Tools
 && yum groupinstall -y "Development Tools"  --setopt=group_package_types=mandatory,default
 && yum groupinstall -y "Development Tools"  --setopt=group_package_types=mandatory,default \
# Upgrade gcc
 && yum install -y gcc72 gcc72-c++

And ta-da, bref builds again!

At this point I was asking myself, why didn't it failed building it on Amazon Linux 2017.03? Compiler is the same, and PHP versions too... I dove into the php-src repository, and found out the first part of the answer: PHP is actually checking the support for __builtin_*_overflow instructions (see acinclude.m4).

Why is it failing now then? If you take a look at the gcc and g++ packages using yum info we can see that they are both at version 4.8 in both Amazon Linux 2017.03 and 2018.03. After installing "Development Tools" group, we can check out the installed packages in both systems and we find our final answer:

# Amazon Linux 2017.03
$ yum list installed | grep -E 'gcc|cpp'
cpp48.x86_64                      4.8.3-9.111.amzn1                @amzn-main   
gcc.noarch                        4.8.3-3.20.amzn1                 @amzn-main   
gcc-c++.noarch                    4.8.3-3.20.amzn1                 @amzn-main   
gcc-gfortran.noarch               4.8.3-3.20.amzn1                 @amzn-main   
gcc48.x86_64                      4.8.3-9.111.amzn1                @amzn-main   
gcc48-c++.x86_64                  4.8.3-9.111.amzn1                @amzn-main   
gcc48-gfortran.x86_64             4.8.3-9.111.amzn1                @amzn-main   
libgcc48.x86_64                   4.8.3-9.111.amzn1                installed
# Amazon Linux 2018.03
$ yum list installed | grep -E 'gcc|cpp'
cpp48.x86_64                      4.8.5-28.142.amzn1               @amzn-updates
cpp72.x86_64                      7.2.1-2.59.amzn1                 @amzn-main   
gcc.noarch                        4.8.5-1.22.amzn1                 @amzn-main   
gcc-c++.noarch                    4.8.5-1.22.amzn1                 @amzn-main   
gcc-gfortran.noarch               4.8.5-1.22.amzn1                 @amzn-main   
gcc48.x86_64                      4.8.5-28.142.amzn1               @amzn-updates
gcc48-c++.x86_64                  4.8.5-28.142.amzn1               @amzn-updates
gcc48-gfortran.x86_64             4.8.5-28.142.amzn1               @amzn-updates
gcc72.x86_64                      7.2.1-2.59.amzn1                 @amzn-main   
libgcc48.x86_64                   4.8.5-28.142.amzn1               @amzn-updates
libgcc72.x86_64                   7.2.1-2.59.amzn1                 installed

It installed gcc 7.2, but it's lacking gcc72-c++. When PHP checks for the supported features, it's doing it using gcc (because it's built in C and it uses gcc 7.2), but when we compile intl extention we need to compile C++ code, it uses g++ 4.8, and everything blows up.

I've tried to remove the cpp72 package, but it is required by kernel-devel package:

Error: Package: kernel-devel-4.14.128-87.105.amzn1.x86_64 (@amzn-updates)
           Requires: gcc >= 7.2.1
           ....

We could setup a restricted COMPILER_PATH for building PHP, but maybe we should just stick with GCC 7.2. It is not that breaking edge anyway.

Final version is available in my fork, I can PR it if you want. (I started over from master because at this point it's just a few changes) I have just tested the resulting image by running it locally with docker, I don't have time right now to test it on lambda (and test all the extentions), but it looks promising.

mnapoli commented 5 years ago

@TheNodi wow this is amazing! Thank you so much for all these explanations, that's really interesting to learn about all that.

A PR would be more than welcome indeed :)

bubba-h57 commented 5 years ago

Nice work! I concur with sticking with GCC 7.2. PR that!

On Fri, Jul 19, 2019, 6:18 AM Matthieu Napoli notifications@github.com wrote:

@TheNodi https://github.com/TheNodi wow this is amazing! Thank you so much for all these explanations, that's really interesting to learn about all that.

A PR would be more than welcome indeed :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/brefphp/bref/issues/332?email_source=notifications&email_token=AAETL3TQJYKZDCPIOOPKI63QAGIGJA5CNFSM4HM5PGSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2LHKVA#issuecomment-513176916, or mute the thread https://github.com/notifications/unsubscribe-auth/AAETL3Q3VE4VPUDPR63ON63QAGIGJANCNFSM4HM5PGSA .

mnapoli commented 5 years ago

PR merged, thanks @TheNodi!