search

brefphp / bref

Serverless PHP on AWS Lambda
https://bref.sh
MIT License
3.16k stars 365 forks source link

Exception thrown unless Symfony cache directory is also changed to `/tmp/...` #39

Closed nealio82 closed 1 year ago

nealio82 commented 6 years ago

Following the example code for Symfony and deploying, the request fails with the response {"message": "Internal server error"}.

Checking CloudWatch logs shows 2 relevant entries:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/task/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php on line 171

and

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 73728 bytes) in /var/task/vendor/symfony/debug/DebugClassLoader.php on line 145

Altering Kernel.php to include the following makes the problem go away

    public function getCacheDir()
    {
        // When on the lambda only /tmp is writable
        if (getenv('LAMBDA_TASK_ROOT') !== false) {
            return '/tmp/cache/'.$this->environment;
        }

        return $this->getProjectDir().'/var/cache/'.$this->environment;
    }

However...

Presumably setting the cache dir to Lambda's local /tmp kind-of defeats the point pre-warming the cache with the hooks in the .bref.yml file? The object is to avoid launching the application without the cache already in place, right?

.bref.yml:

hooks:
    build:
        - 'APP_ENV=prod php bin/console cache:clear --no-debug --no-warmup'
        - 'APP_ENV=prod php bin/console cache:warmup'

serverless.yml:

service: test

provider:
  name: aws
  runtime: nodejs6.10

package:
  exclude:
    - '*'
    - '**'
  include:
    - bref.php
    - 'src/**'
    - 'vendor/**'
    - composer.json # Symfony uses it to figure out the root directory
    - 'bin/**'
    - 'config/**'
    - 'var/cache/prod/**' # We want to deploy the production caches

functions:
  # By default we create one "main" function
  main:
    handler: handler.handle
    timeout: 20 # Timeout in seconds, the default is 6 seconds
    # The function will match all HTTP URLs
    events:
      - http: 'ANY /'
      - http: 'ANY {proxy+}'
    environment:
      APP_ENV: 'prod'
      APP_DEBUG: '0'

bref.php:

<?php

use App\Kernel;
use Bref\Bridge\Symfony\SymfonyAdapter;
use Symfony\Component\Debug\Debug;
use Symfony\Component\Dotenv\Dotenv;

require __DIR__.'/vendor/autoload.php';

Debug::enable();

// The check is to ensure we don't use .env in production
if (!isset($_SERVER['APP_ENV'])) {
    (new Dotenv)->load(__DIR__.'/.env');
}
if ($_SERVER['APP_DEBUG'] ?? ('prod' !== ($_SERVER['APP_ENV'] ?? 'dev'))) {
    umask(0000);
}
$kernel = new Kernel($_SERVER['APP_ENV'] ?? 'dev', (bool) ($_SERVER['APP_DEBUG'] ?? ('prod' !== ($_SERVER['APP_ENV'] ?? 'dev'))));

$app = new \Bref\Application;
$app->httpHandler(new SymfonyAdapter($kernel));
$app->cliHandler(new \Symfony\Bundle\FrameworkBundle\Console\Application($kernel));
$app->run();

I get the same behaviour with cloning the mnapoli/bref-symfony-demo repo and running bref deploy, and also with adding bref to a symfony/website-skeleton project.

mnapoli commented 6 years ago

Regarding the memory size problem, you could try increasing the memory limit of PHP by customizing the php.ini config, see https://github.com/mnapoli/bref#php-configuration to see how to set php.ini flags. However 128Mb seems already quite much, I'm surprised Symfony would consume that much.

Presumably setting the cache dir to Lambda's local /tmp kind-of defeats the point pre-warming the cache with the hooks in the .bref.yml file? The object is to avoid launching the application without the cache already in place, right?

Agreed, this is not really efficient in production that's why it's better to pre-generate the cache.

I get the same behaviour with cloning the mnapoli/bref-symfony-demo repo and running bref deploy

This is surprising, but since I wrote (and tested) bref-symfony-demo we have added some extensions to PHP (mainly opcache, the other extensions are disabled by default), maybe that could be it? I don't really know much more right now I'll try to dig in.

nealio82 commented 6 years ago

The first thing I tried was to increase the memory_limit (I should have mentioned that above, sorry!)

I think the cache is not being pre-warmed during the deploy process, because presumably it should exist and be readable if it has been!?

As mentioned, if you set the cache dir to the writeable /tmp/ dir then the problem disappears and PHP doesn't complain about memory any more

mnapoli commented 6 years ago

OK, I'll try to reproduce that tonight and see how it goes. Thanks for opening the issue!

t-geindre commented 6 years ago

We use our own kernel here, but since we also use symfony/dependency-injection we had to find a solution to pre-warm the cache, which essentially means to build and dump the container.

The only way we found to do so is to use relative paths.

For instance, we have a parameter %cache_dir%. Its value is always ./var/cache. This allows us to dump the container before uploading it on lambda.

Our kernel always check that the current working directory is $_ENV['LAMBDA_TASK_ROOT'].

This works and we don't build the container on each execution. I think you could do something similar with the Symfony kernel.

That beeing said, if you have a better solution, I would be very interested.

mnapoli commented 6 years ago

@nealio82 I've updated (composer update) everything to the latest versions on the symfony-demo project and redeployed, it still works: https://7oaryq3rzl.execute-api.eu-west-3.amazonaws.com/dev I don't understand what breaks for you 🤔

@t-geindre Yes! I had issues with absolute/relative paths when deploying the API platform demo for my benchmarks. In the end I gave up and stored the cache in /tmp because I was not measuring the cold starts, so I just had to warmup the application and that's it. But this is not ideal at all.

nealio82 commented 6 years ago

I probably won’t have time tonight, but i’ll create a version which breaks / screengrab reproduction steps

On Mon, 9 Jul 2018 at 18:49, Matthieu Napoli notifications@github.com wrote:

@nealio82 https://github.com/nealio82 I've updated (composer update) everything to the latest versions on the symfony-demo project and redeployed, it still works: https://7oaryq3rzl.execute-api.eu-west-3.amazonaws.com/dev I don't understand what breaks for you 🤔

@t-geindre https://github.com/t-geindre Yes! I had issues with absolute/relative paths when deploying the API platform demo for my benchmarks. In the end I gave up and stored the cache in /tmp because I was not measuring the cold starts, so I just had to warmup the application and that's it. But this is not ideal at all.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnapoli/bref/issues/39#issuecomment-403563405, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCVBs2lAnH5tdsVqgYsKKU_LW_aU-swks5uE5eOgaJpZM4VHpC8 .

nealio82 commented 6 years ago

Interestingly, if I clone the mnapoli/bref-symfony-demo repo, I get a different (and nicer) exception message about the cache.

The symfony/website-skeleton still gives me out of memory, but the SF demo now specifically mentions the cache.

I made a screencast here: https://youtu.be/Ar3USl8h8Ug

mnapoli commented 6 years ago

Thanks for the screencast! Could you try one last time mnapoli/bref-symfony-demo by running composer update before bref deploy to install the last version of Bref?

nealio82 commented 6 years ago

I reverted my changes in Kernel.php and did the following:

➜  bref-symfony-demo git:(master) ✗ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)

Prefetching 2 packages 🎵
  - Downloading (100%)

Package operations: 2 installs, 20 updates, 0 removals
  - Installing matomo/ini (2.0.0) Loading from cache
  - Updating symfony/process (v4.0.9 => v4.1.1) Loading from cache
  - Updating zendframework/zend-diactoros (1.7.1 => 1.8.1) Loading from cache
  - Updating aws/aws-sdk-php (3.55.9 => 3.62.12) Loading from cache
  - Updating symfony/http-foundation (v4.0.9 => v4.1.1) Loading from cache
  - Installing symfony/polyfill-ctype (v1.8.0) Loading from cache
  - Updating symfony/yaml (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/finder (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/filesystem (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/console (v4.0.9 => v4.1.1) Loading from cache
  - Updating mnapoli/bref (0.2.4 => 0.2.18) Loading from cache
  - Updating symfony/routing (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/event-dispatcher (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/debug (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/http-kernel (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/dependency-injection (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/config (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/cache (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/framework-bundle (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/twig-bridge (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/twig-bundle (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/dotenv (v4.0.9 => v4.1.1) Loading from cache
Writing lock file
Generating autoload files

What about running composer global require symfony/thanks && composer thanks now?
This will spread some 💖  by sending a ★  to the GitHub repositories of your fellow package maintainers.

Executing script cache:clear [OK]

➜  bref-symfony-demo git:(master) ✗ php vendor/bin/bref deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (22.52 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..
Serverless: Stack update finished...
Service Information
service: bref-demo-symfony
stage: dev
region: eu-west-3
stack: bref-demo-symfony-dev
api keys:
  None
endpoints:
  ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev
  ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev/{proxy+}
functions:
  main: bref-demo-symfony-dev-main
Deployment success
 8/8 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  1 min%

I see the same exception(s) related to the cache again

[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
2018-07-10T12:19:17+00:00 [critical] Uncaught PHP Exception RuntimeException: "Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc)." at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Exception thrown when handling an exception (RuntimeException: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc). at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59)
[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
Fatal error: Uncaught RuntimeException: Unable to create the cache directory (/var/task/var/cache/prod/twig/ba). in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php:55
Stack trace:
#0 /var/task/vendor/twig/twig/lib/Twig/Environment.php(369): Twig_Cache_Filesystem->write('/var/task/var/c...', '<?php\n\n/* hello...')
#1 /var/task/vendor/twig/twig/lib/Twig/Environment.php(289): Twig_Environment->loadTemplate('hello.html.twig')
#2 /var/task/vendor/symfony/framework-bundle/Controller/ControllerTrait.php(224): Twig_Environment->render('hello.html.twig', Array)
#3 /var/task/src/Controller/HomeController.php(12): Symfony\Bundle\FrameworkBundle\Controller\Controller->render('hello.html.twig')
#4 /var/task/vendor/symfony/http-kernel/HttpKernel.php(149): App\Controller\HomeController->index()
#5 /var/task/vendor/symfony/http-kernel/HttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#6 /var/task/vendor/symfony/http-kernel/Kernel.php(188): Symfony\Compone in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php on line 59
nealio82 commented 6 years ago

If I add the cache param to config/packages/twig.yml with the value of either /tmp/... or false, then everything works again.

twig:
    paths: ['%kernel.project_dir%/templates']
    debug: '%kernel.debug%'
    strict_variables: '%kernel.debug%'
    cache: '/tmp/twig'

see https://twig.symfony.com/doc/2.x/api.html#environment-options and http://symfony.com/doc/current/reference/configuration/twig.html#cache

However, I don't like setting /tmp/... in the twig config, and I don't know enough about Twig to tell if disabling the cache is a bad idea.

I also don't get why Twig needs to write to the cache if the templates are pre-compiled!?

mnapoli commented 6 years ago

OK I think I remember now where the problem came from (when I tried deploying the API platform demo).

Some things in the Symfony cache are using absolute paths, as @t-geindre mentioned. You need to tweak Symfony and its config to force it to use relative paths (because the paths on your machine and on AWS lambda are different).

That works, except for Twig that IIRC uses realpath() as some point in the code (I think it's here). Because of that call, a relative path is turned into an absolute path and it breaks things. I don't remember the details anymore though.

Also it is important to note that not all the Symfony cache is pregenerated, for example you have to put Doctrine's cache (which is not pregenerated) in /tmp manually.

Here is an example of what I did for Doctrine in config/packages/doctrine.yaml:

parameters:
    writable_cache_dir: '/tmp/cache'

[…]

doctrine:
    orm:
        […]
        metadata_cache_driver:
            cache_provider: metadata_cache
        result_cache_driver:
            cache_provider: result_cache
        query_cache_driver:
            cache_provider: query_cache

[…]

doctrine_cache:
    providers:
        metadata_cache:
            aliases: [doctrine.orm.default_metadata_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/metadata"
        result_cache:
            aliases: [doctrine.orm.default_result_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/result"
        query_cache:
            aliases: [doctrine.orm.default_query_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/query"

I did the same for the user cache:

framework:
    cache:
        directory: "%writable_cache_dir%/app"
nealio82 commented 6 years ago

Ok, so I assume that means at some point we have to have a writeable filesystem in order for Twig to work properly?

However, that raises 2 questions:

  1. Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?
  2. Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!
mnapoli commented 6 years ago

Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?

That could be a temporary solution. I still hope to find the correct configuration to apply to solve that (or maybe fix the problem in Twig/Symfony). But yeah it's better to have helpful documentation for now.

Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!

Yes I don't know how to explain that either :/

nealio82 commented 6 years ago

Ok, I can make a PR for this tomorrow :)

On Fri, 13 Jul 2018 at 20:25, Matthieu Napoli notifications@github.com wrote:

Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?

That could be a temporary solution. I still hope to find the correct configuration to apply to solve that (or maybe fix the problem in Twig/Symfony). But yeah it's better to have helpful documentation for now.

Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!

Yes I don't know how to explain that either :/

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnapoli/bref/issues/39#issuecomment-404930108, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCVBtB0FkhbTNKAuTqR0t1-Rb5GHQVEks5uGPQVgaJpZM4VHpC8 .

nealio82 commented 6 years ago

Workaround documentation added in #42

mnapoli commented 6 years ago

Does anyone here has a public repository to reproduce this?

nealio82 commented 6 years ago

I can make one early next week (probably Monday)

On Fri, 24 Aug 2018 at 14:07, Matthieu Napoli notifications@github.com wrote:

Does anyone here has a public repository to reproduce this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mnapoli/bref/issues/39#issuecomment-415753257, or mute the thread https://github.com/notifications/unsubscribe-auth/ABCVBovYn0MTlVJHvnXypDufuXWx6MmXks5uT_qPgaJpZM4VHpC8 .

mnapoli commented 6 years ago

For those reading this issue, I reproduced it when writing a simple Symfony 4 application, starting with Flex. I just have a base template (the layout) and a few pages that extend from that. I don't have time right now to create a whole repository to reproduce that but that can maybe be a starting point for others.

nealio82 commented 6 years ago

Weirdly I tried yesterday; starting with Symfony 4 & extending a base template, then passing in some variables to render. I didn’t see the issue at all. (And i’m pretty sure the first time I saw the issue I didn’t even need to extend a base template)

thibaudlemaire commented 5 years ago

Hi! I personally chose to redirect Twig's cache to config/packages/prod/twig.yaml to be able to warm up other bundles.

# config/packages/prod/twig.yaml
twig:
    cache: '/tmp/cache/twig'

I know it's a temporary solution but I preferer doing this because making /tmp the default Symfony cache directory increases the cold start from 500ms to 2 seconds.
I hope Twig's team will work on this problem to comply with the Symfony4 best practices (var/cache should be read-only and warmable).

mnapoli commented 5 years ago

Thanks that's a much better solution! I'll try to update the documentation in the coming weeks (I should have more time than this month), if anyone wants to get on this though feel free.

nealio82 commented 5 years ago

@thibaudlemaire by any chance do you know why it adds a 4x overhead? Have you looked into that with Blackfire, for example? The Symfony best practice is to override it in the Kernel class: https://symfony.com/doc/current/configuration/override_dir_structure.html#override-the-cache-directory

thibaudlemaire commented 5 years ago

When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp dir that is not packed by sam package command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.

The 4x overhead is due to the cache warmup when a Lambda container is created. I don't know exactly what is compiled at the first execution but I suppose : Doctrine proxies, annotations, Twig views, Kernel config, services container, etc.

That's why we need to make Symfony use the pre-warmed cache. To do so, I thought about two solutions :

I use the second one because it's easier. But it's not ideal because twig views still need to be compiled on first invoke.

nealio82 commented 5 years ago

When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp dir that is not packed by sam package command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.

Ah, of course! 🤦‍♂

mnapoli commented 1 year ago

I'll be closing this issue since it was opened in 2019, and since then we have the Symfony bridge that should take care of the cache 🎉