Testing v0.3 - Githubissues

mnapoli commented 5 years ago

Update

Please follow the documentation at bref.sh to try the new version!

Disregard the outdated instructions below :)

Here is an example of a serverless web app built with Bref 0.3: https://github.com/mnapoli/bref-runtime-versions

These instructions are outdated

Following my last comment here https://github.com/mnapoli/bref/issues/114#issuecomment-448752042 I am opening this issue to coordinate testing v0.3.

If you want to help test v0.3 I suggest you subscribe to this issue. Let's post here questions and updates on how to try it out, since it's in a very incomplete state right now.

So to get started:

install AWS SAM on your machine (you should already have aws configured)
create a template.yml file: this is the configuration for SAM, it replaces serverless.yml (examples below)
FYI always use region us-east-2 for now (the runtimes are only available there)
require the branch 0.3 of Bref: composer require mnapoli/bref:0.3.x-dev@dev

CLI application

Example of a template.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
    CliFunction:
        Type: AWS::Serverless::Function
        Properties:
            FunctionName: 'my-function'
            CodeUri: .
            Handler: index.php
            Runtime: provided
            Layers:
                - 'arn:aws:lambda:us-east-2:416566615250:layer:php-72:9'

And index.php:

<?php

require __DIR__.'/vendor/autoload.php';

λ(function (array $event) {
    return [
        'hello' => $event['name'] ?? 'world',
    ];
});

HTTP application

Example of a template.yml:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
    HttpFunction:
        Type: AWS::Serverless::Function
        Properties:
            FunctionName: 'my-function'
            CodeUri: .
            Handler: index.php
            Runtime: provided
            Layers:
                - 'arn:aws:lambda:us-east-2:416566615250:layer:php-72-fpm:4'
            Events:
                HttpRoot:
                    Type: Api
                    Properties:
                        Path: /
                        Method: ANY
                HttpSubPaths:
                    Type: Api
                    Properties:
                        Path: /{proxy+}
                        Method: ANY
Outputs:
    DemoHttpApi:
        Description: 'API Gateway URL for our function'
        Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/'

In index.php run your favorite framework, or simply do echo 'Hello world!', it works just like usual with PHP-FPM.

Runtimes

To learn more about the runtimes, how they work and what can be done please read https://github.com/mnapoli/bref/tree/0.3/runtime

I've hardcoded some version numbers above, I'll keep you updated here if I publish new versions.

Running locally

These require Docker.

CLI applications: sam local invoke --no-event --region us-east-2 (remember the region even locally!)

HTTP applications: sam local start-api --region us-east-2

Please read about SAM and learn about what can be done with it: https://github.com/awslabs/aws-sam-cli/blob/develop/docs/usage.rst I only know the very basics, maybe there are awesome stuff to discover and share.

Deploying

Not as easy as serverless, this is done in 3 steps. Official documentation here

first create a S3 bucket (region us-east-2!), you only need to do this once, it will store the code
package: sam package --template-file template.yaml --output-template-file template-output.yaml --s3-bucket <s3-bucket-name>
deploy: sam deploy --template-file template-output.yaml --stack-name <my-stack-name> --capabilities CAPABILITY_IAM --region us-east-2

Repeat package and deploy every time you want to deploy.

pmayet commented 5 years ago

On locally running, I'm getting like @nealio82 a 500 response and Error: Runtime failed to start: fork/exec /var/task/bootstrap: permission denied Runtime.ExitError in my Cloudwatch logs.

With both arn:aws:lambda:us-east-2:416566615250:layer:php-72-fpm:3 and my own layer build with bref fpm runtime.

I have check the permission for bootstrap : -rwxr-xr-x 1 pmayet staff 790 20 déc 13:49 bootstrap

Any idea ?

pmayet commented 5 years ago

After deploy Lambda : Fatal error: Uncaught Error: Call to undefined method Bref\Runtime\LambdaRuntime::processNextEvent() in /opt/bootstrap:25

mnapoli commented 5 years ago

Woops I need to re-release because of #121 I think, my bad!

Error: Runtime failed to start: fork/exec /var/task/bootstrap: permission denied Runtime.ExitError

That's weird! I'll re-release and try again locally to see if it works for me (for now it did).

mnapoli commented 5 years ago

OK please use version 4 for the -fpm layer now. I tested locally and it works.

I test with the template.yaml that is in this repository on the 0.3 branch, so I guess you can clone Bref, switch to 0.3 and try it yourself? If it still doesn't work can you tell us which OS you are using? (I'm using OS X)

pmayet commented 5 years ago

Thank you, it works. I am now trying to update the code for Laravel, based on what you mentioned in the doc but I have some difficulties.

mnapoli commented 5 years ago

Update: with #122 (merged) you can now run Symfony Console or Laravel Artisan on Lambda!

SAM integration is worked upon in #126 (WIP). You can check out the default SAM templates in there to get some ideas on how to create a lambda from scratch.

pmayet commented 5 years ago

With Laravel 5.7, Bref 0.3, SAM template.yml (Local) with your php-fpm runtime, same error :

{
  "errorType": "Runtime.ExitError",
  "errorMessage": "RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Error: fork/exec /var/task/bootstrap: permission denied"
}

But, when I clone the repository of Bref and switch to branch 0.3, SAM Local with your php-fpm runtime works fine.

I have no log...

The content of the bref.php on the root directory :

<?php

define('LARAVEL_START', microtime(true));

require __DIR__.'/vendor/autoload.php';

$app = require_once __DIR__.'/bootstrap/app.php';

if (!is_dir(storage_path('framework/views'))) {
    if (!mkdir(storage_path('framework/views'), 0755, true)) {
        die('Cannot create directory ' . storage_path('framework/views'));
    }
}

lambda($app->getBrefHttpAdapter());

Any idea ?

mnapoli commented 5 years ago

@pmayet /var/task/bootstrap is weird, it means there is a bootstrap file in your project.

The file used by lambda should be /opt/bootstrap instead (the one provided by the layer). Can you make sure there is no bootstrap file in your project?

Also are you deploying from Windows? Which version of the layer are you using?

Juhlinus commented 5 years ago

With Laravel 5.7, Bref 0.3, SAM template.yml (Local) with your php-fpm runtime, same error

<?php

define('LARAVEL_START', microtime(true));

require __DIR__.'/vendor/autoload.php';

$app = require_once __DIR__.'/bootstrap/app.php';

if (!is_dir(storage_path('framework/views'))) {
    if (!mkdir(storage_path('framework/views'), 0755, true)) {
        die('Cannot create directory ' . storage_path('framework/views'));
    }
}

lambda($app->getBrefHttpAdapter());

I'm running this exact setup. Clean git clone of the 0.3 branch.

bootstrap/app.php:

// $app = new Illuminate\Foundation\Application(
$app = new Bref\Bridge\Laravel\Application(
    $_ENV['APP_BASE_PATH'] ?? dirname(__DIR__)
);

$app->useStoragePath(env('APP_STORAGE', $app->storagePath()));

template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
    HttpFunction:
        Type: AWS::Serverless::Function
        Properties:
            FunctionName: 'my-function'
            CodeUri: .
            Handler: bref.php
            Runtime: provided
            Layers:
                - 'arn:aws:lambda:us-east-2:416566615250:layer:php-72-fpm:4'
            Events:
                HttpRoot:
                    Type: Api
                    Properties:
                        Path: /
                        Method: ANY
                HttpSubPaths:
                    Type: Api
                    Properties:
                        Path: /{proxy+}
                        Method: ANY
Outputs:
    DemoHttpApi:
        Description: 'API Gateway URL for our function'
        Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/'

When I hit the API Endpoint i get the following error:

Fatal error: Uncaught TypeError: Argument 1 passed to lambda() must be callable, object given, called in /var/task/bref.php on line 17 and defined in /var/task/vendor/mnapoli/bref/src/functions.php:14 Stack trace: #0 /var/task/bref.php(17): lambda(Object(Bref\Bridge\Laravel\LaravelAdapter)) #1 {main} thrown in /var/task/vendor/mnapoli/bref/src/functions.php on line 14

I interpreted this thread as though I was supposed to pass the LaravelAdapter.

pmayet commented 5 years ago

@pmayet /var/task/bootstrap is weird, it means there is a bootstrap file in your project.

The file used by lambda should be /opt/bootstrap instead (the one provided by the layer). Can you make sure there is no bootstrap file in your project?

Also are you deploying from Windows? Which version of the layer are you using?

I think the problem is the presence of the directory bootstrap at the root of the Laravel project. I will try by modifying the CodeUri of the template.yaml as Stackery does.

I deploy from MacOS.

mnapoli commented 5 years ago

Ohh that's a good point! Laravel's bootstrap directory will probably collide with Lambda's behavior.

I'm not sure what we can do here, I'll open a separate issue: https://github.com/mnapoli/bref/issues/134

Unrelated but for those interested in testing Bref 0.3 I have almost finished the new documentation if that helps: https://github.com/mnapoli/bref/pull/127

davidenegrini commented 5 years ago

Hi! I'm trying out the layer '[...]php-72-fpm:4' with a simple echo hello world in index.php, every tool mentioned should be at the last version (running an ubuntu fresh install on vm) and aws configured to run in ohio (us-east-2), but when opening the api gateway link in lambda function it returns only "Access denied". Any idea?

pmayet commented 5 years ago

@davidenegrini Is it on SAM Local ? Because only POST query return 200.

davidenegrini commented 5 years ago

@pmayet No I’m packaging and deploying directly to AWS

mnapoli commented 5 years ago

Have you looked into the logs (on Cloudwatch)? Anything there? Are you using the correct URL with the /Prod/ prefix in the URL? Are you getting the URL from the CloudFormation "output" tab? Have you tried to run it locally with SAM? If so does it work?

Because only POST query return 200.

@pmayet not sure what you mean? GET requests work fine for me 🤔

davidenegrini commented 5 years ago

Have you looked into the logs (on Cloudwatch)? Anything there?

Nothing useful

START RequestId: a97fa003-10fb-11e9-9f39-17c2f92fd31c Version: $LATEST
END RequestId: a97fa003-10fb-11e9-9f39-17c2f92fd31c
REPORT RequestId: a97fa003-10fb-11e9-9f39-17c2f92fd31c Duration: 31.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 64 MB
START RequestId: a9f8a10b-10fb-11e9-93fe-55d7b00f7a5a Version: $LATEST
END RequestId: a9f8a10b-10fb-11e9-93fe-55d7b00f7a5a
REPORT RequestId: a9f8a10b-10fb-11e9-93fe-55d7b00f7a5a Duration: 1.73 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 64 MB

Are you using the correct URL with the /Prod/ prefix in the URL? Are you getting the URL from the CloudFormation "output" tab?

Of course yes!

Have you tried to run it locally with SAM? If so does it work?

Getting a 500 internal server error calling both 127.0.0.1:3000 and 127.0.0.1:3000/{proxy+}

2019-01-05 16:45:03 Invoking index.php (provided)
2019-01-05 16:45:03 arn:aws:lambda:us-east-2:416566615250:layer:php-72-fpm:4 is already cached. Skipping download
2019-01-05 16:45:03 Requested to skip pulling images ...

2019-01-05 16:45:03 Mounting /home/davide/lambdabref as /var/task:ro inside runtime container
START RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72 Version: $LATEST
END RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72
REPORT RequestId: 52fdfc07-2182-154f-163f-5f0f9a621d72  Init Duration: 63.53 msDuration: 15.61 ms   Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 42 MB  
2019-01-05 16:45:04 127.0.0.1 - - [05/Jan/2019 16:45:04] "GET /favicon.ico HTTP/1.1" 500 -
2019-01-05 16:45:04 Error on request:
Traceback (most recent call last):
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 270, in run_wsgi
    execute(self.server.app)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/serving.py", line 258, in execute
    application_iter = app(environ, start_response)
  File "/home/davide/.local/lib/python3.6/site-packages/flask/app.py", line 2309, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/davide/.local/lib/python3.6/site-packages/flask/app.py", line 2299, in wsgi_app
    return response(environ, start_response)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/wrappers.py", line 1325, in __call__
    app_iter, status, headers = self.get_wsgi_response(environ)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/wrappers.py", line 1313, in get_wsgi_response
    headers = self.get_wsgi_headers(environ)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/wrappers.py", line 1211, in get_wsgi_headers
    headers = Headers(self.headers)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/datastructures.py", line 949, in __init__
    self.extend(defaults)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/datastructures.py", line 1080, in extend
    self.add(key, value)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/datastructures.py", line 1165, in add
    self._validate_value(_value)
  File "/home/davide/.local/lib/python3.6/site-packages/werkzeug/datastructures.py", line 1172, in _validate_value
    raise ValueError('Detected newline in header value.  This is '
ValueError: Detected newline in header value.  This is a potential security problem

I'm sorry not to be able to help more than reporting this :/

mnapoli commented 5 years ago

Detected newline in header value. This is a potential security problem

That's interesting 🤔

julienmortuaire commented 5 years ago

Hello,

I just tried the HTTP layer with PHP-FPM on AWS Lambda and it work like a charm. However, i'm unable to get any custom ENV variable set with SAM or using AWS Lambda GUI. I check on $_SERVER, $_ENV and with phpinfo() function.

Layer used : arn:aws:lambda:us-east-2:416566615250:layer:php-72-fpm:4

Does environment variable is forwarded from PHP FPM proxy to final code?

mnapoli commented 5 years ago

@julienmortuaire thank you for the report! They should work but I don't think I tested that.

Could you open an issue for that? That way we can track it. Thank you!

mnapoli commented 5 years ago

Also in the issue can you detail how you set the environment variables in lambda?

julienmortuaire commented 5 years ago

Hi,

Just deep dive a bit on the ENV issue and the problem seems to be linked to the _clearenv php-fpm.conf directive.

From php.net :

Clear environment in FPM workers. Prevents arbitrary environment variables from reaching FPM worker processes by clearing the environment in workers before env vars specified in this pool configuration are added. Default value seems to be yes, aka the environment var is "cleaned" by default.

I rebuild my own FPM layer with "clear_env = no" inside the php-fpm.conf thanks to your publish.sh (which works very well by the way) and lambda environment var is now accessible from my PHP code on $_ENV and $_SERVER, tested with phpinfo() function.

I don't mesure the exact impact of the _clearenv FPM directive especially from a security point of view. My first understanding is that this is not a problem on our lambda context, but it will be great to have an other advice.

bertrandjamin commented 5 years ago

Hi,

I'm trying to handle a POST request with an HTTP application model but I cannot retrieve the requested body content.

It is always empty...

configuration

# template.yaml

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: ''

Resources:
    MyFunction:
        Type: AWS::Serverless::Function
        Properties:
            FunctionName: 'gitlab-webhook'
            Description: ''
            CodeUri: .
            Handler: index.php
            Timeout: 30 # in seconds (API Gateway has a timeout of 30 seconds)
            Runtime: provided
            Layers:
                - 'arn:aws:lambda:us-east-2:416566615250:layer:php-72-fpm:4'
            Events:
                # The function will match all HTTP URLs
                HttpRoot:
                    Type: Api
                    Properties:
                        Path: /
                        Method: ANY
                HttpSubPaths:
                    Type: Api
                    Properties:
                        Path: /{proxy+}
                        Method: ANY

# Outputs show up in the CloudFormation dashboard
Outputs:
    DemoHttpApi:
        Description: 'URL of our function in the *Prod* environment'
        Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/'

// index.php
// Example inspired from `vendor/mnapoli/bref/demo/http.php`

declare(strict_types=1);

use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;

require __DIR__ . '/vendor/autoload.php';

$slim = new \Slim\App;

$slim->post('/', function (ServerRequestInterface $request, ResponseInterface $response) {

    $response->getBody()->write($request->getBody()->getContents());

    return $response;
});

$slim->run();

Usecase

I send a POST request with a body content (json_object or string)
The output is empty...

FYI i have tried to read directly the php://input flux without success either.

Did I miss something ?

mnapoli commented 5 years ago

Thanks @julienmortuaire I've opened #161

@bertrandjamin could you open a separate issue? For information here is a test that passes: https://github.com/mnapoli/bref/blob/0.3/tests/Runtime/PhpFpmTest.php#L142-L171 Can you try with a similar request? (e.g. set the Content-Type and Content-Length headers. Maybe the Content-Length header is missing and PHP-FPM ignores the body? If with apache/nginx the content length is auto-added by the server we could replicate that behavior)

swiffer commented 5 years ago

Successfully deployed a CakePHP console application.

When calling via bref locally I get back the source code instead of the actual Response (Code not executed).
UPDATE: Think I need to Point to bin/cake.php instead of bin/cake, right?
Is the bref Composer package still needed in production or should the Docs be updated to composer require mnapoli/bref --dev

mnapoli commented 5 years ago

@swiffer it is used in production so it should be installed as a hard requirement.

Regarding 1. are you trying to run a HTTP application or a console application? Here is the documentation for both:

Also can you confirm you run everything with the us-east-2 region and you use the latest layer versions?

mnapoli commented 5 years ago

Good news! The new runtimes (built with the new pipeline) have been published and distributed on all regions!

Check out https://runtimes.bref.sh/ to find out the version to use in your templates.

Btw this website is made using Bref, this is a good showcase of how to use v0.3, see the source code here: https://github.com/mnapoli/bref-runtime-versions (see #168)

Please test those new runtimes (the Bref documentation is up to date) and report any issue you find 🙏

pmayet commented 5 years ago

@mnapoli the http status code for https://runtimes.bref.sh/ is 404

swiffer commented 5 years ago

console layer is missing atm?

Juhlinus commented 5 years ago

Good news! The new runtimes (built with the new pipeline) have been published and distributed on all regions!

You're still missing the asian regions, unfortunately for me, haha.

ap-south-1 ap-northeast-3 ap-northeast-2 ap-southeast-1 ap-southeast-2 ap-northeast-1

mnapoli commented 5 years ago

@pmayet, @swiffer, @Juhlinus thanks I've opened 3 separate issues.

From now on let's open issues and keep this thread for announcements regarding the testing of v0.3.

mnapoli commented 5 years ago

Last chance for testing before the release in a couple of days!

Remember to update all your runtimes to the latest version at https://runtimes.bref.sh (and update Bref).

nealio82 commented 5 years ago

@mnapoli when are you planning to release? I've been completely stacked up with other stuff for a number of weeks, but I can do a fair amount of Symfony testing this coming weekend

mnapoli commented 5 years ago

@nealio82 I'll release Thursday morning. But it's fine if we have bug fixes coming next week as patch releases, I don't think there's any reason to break BC.

mnapoli commented 5 years ago

https://twitter.com/matthieunapoli/status/1090977423347380224 🎉 congrats everyone!

Now off to the next releases, this is just the beginning!

😄

(also I'm sure you will be able to find new bugs ;) please open separate issues from now on, I'll close this one)

brefphp / bref

Testing v0.3 #116

CLI application

HTTP application

Runtimes

Running locally

Deploying

configuration

Usecase