search

front / wp-cli-build

Fetch specific versions of plugins/themes from wordpress.org using a build file
43 stars 7 forks source link

Memory exhausted error despite set memory limit #19

Closed florianbepunkt closed 4 years ago

florianbepunkt commented 4 years ago

Encountering another issue while integrating this package into a CI workflow on CircleCi.

I had to increase the PHP memory limit in order to install the plugin. So I set the limit to 1024M. However when running the build command, I get a memory error – but only for the wordpress core. All plugins are downloaded, but the core is missing.

docker run -it --rm \
  --volumes-from wp-container \
  --network wp-network \
  --user 33:33 \
  -e HOME=. \
  cm-wordpress-cli:latest build --clean --yes --path=cm
^@^@Parsing build.json, please wait...
- Downloading WordPress 5.2.3 (de_DE): [27-Sep-2019 14:26:46 UTC] PHP Fatal error:  Allowed memory size of 2097152 bytes exhausted (tried to allocate 32768 bytes) in phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php on line 1474
- Downloading acf-content-analysis-for-yoast-seo (2.3.0): done
- Downloading broken-link-checker (1.11.8): done
- Downloading enable-media-replace (3.3.6): done
- Downloading ewww-image-optimizer (4.9.3): done
- Downloading woocommerce-germanized (2.3.4): done
- Downloading instagram-feed (1.12.2): done
- Downloading rocket-lazy-load (2.3.2): done
- Downloading post-types-order (1.9.4.1): done
- Downloading regenerate-thumbnails (3.1.1): done
- Downloading rename-media-files (1.0.1): done
- Downloading simple-301-redirects (1.07): done
- Downloading w3-total-cache (0.10.1): done
- Downloading woocommerce (3.7.0): done
- Downloading woocommerce-admin (0.19.0): done
- Downloading woocommerce-email-validation (2.1.1): done
- Downloading woocommerce-gateway-stripe (4.2.4): done
- Downloading wordfence (7.4.0): done
- Downloading wp-retina-2x (5.5.7): done
- Downloading wordpress-seo (12.1): done
Finished.

The command does not return an exit code, but since the core files are missing my subsequent test steps fail.

Any idea what might causing this?

Interestingly when I change the wordpress doccker images fron php7.1 to wordpress:latest and wordpress:cli the error message changes.

Parsing build.json, please wait...
- Downloading WordPress 5.2.3 (de_DE): [27-Sep-2019 12:57:57 UTC] PHP Fatal error:  Allowed memory size of 2097152 bytes exhausted (tried to allocate 65536 bytes) in phar:///usr/local/bin/wp/vendor/wp-cli/mustangostang-spyc/src/Spyc.php on line 1110

Since my memory limit is much higher than the 2mb and the default php memory limit is also higher, I think it must be something else.

fabioneves commented 4 years ago

Depending on how PHP is setup, sometimes a different php.ini for the CLI is used, you can run in the command line for eg. php -i |grep php.ini, just to make sure what's the php.ini you should change.

florianbepunkt commented 4 years ago

It is the correct one.

Loaded Configuration File: /usr/local/etc/php/php.ini

Also both php -r "echo ini_get('memory_limit').PHP_EOL;" and php -i | grep memory_limit return 1024M.

fabioneves commented 4 years ago

I see, but anyway this has nothing to do with wp build itself. What wp build uses internally to download the core is basically this wp-cli command wp core download, if you run this command alone, it should give you the same memory limit issue, unless it's omitting php errors.

florianbepunkt commented 4 years ago

Thought so too, but wp core download works without a hitch.

For debugging I just ran both wp-core download and wp build in the same container with --debug flag

docker run -it --rm \
  --volumes-from wp-container \
  --network wp-network \
  --user 33:33 \
  -e HOME=. \
  cm-wordpress-cli:latest core download --path=cm --debug --locale=de_DE
^@^@Debug (bootstrap): Loading packages from: ./.wp-cli/packages/vendor/autoload.php (0.01s)
Debug (bootstrap): Loading detected autoloader: ./.wp-cli/packages/vendor/autoload.php (0.011s)
Debug (commands): Adding command: build (0.018s)
Debug (commands): Adding command: build-generate (0.018s)
Debug (bootstrap): Fallback autoloader paths: phar://wp-cli.phar/vendor/autoload.php (0.018s)
Debug (bootstrap): Loading detected autoloader: phar://wp-cli.phar/vendor/autoload.php (0.018s)
Debug (commands): Adding command: cache (0.02s)
Debug (commands): Adding command: transient (0.021s)
Debug (commands): Adding command: comment (0.022s)
Debug (commands): Adding command: meta in comment Namespace (0.023s)
Debug (commands): Adding command: menu (0.024s)
Debug (commands): Adding command: item in menu Namespace (0.025s)
Debug (commands): Adding command: location in menu Namespace (0.025s)
Debug (commands): Deferring command: network meta (0.025s)
Debug (commands): Adding command: option (0.026s)
Debug (commands): Adding command: post (0.028s)
Debug (commands): Adding command: meta in post Namespace (0.029s)
Debug (commands): Adding command: term in post Namespace (0.03s)
Debug (commands): Adding command: post-type (0.03s)
Debug (commands): Adding command: site (0.032s)
Debug (commands): Adding command: meta in site Namespace (0.032s)
Debug (commands): Adding command: option in site Namespace (0.033s)
Debug (commands): Adding command: taxonomy (0.033s)
Debug (commands): Adding command: term (0.035s)
Debug (commands): Adding command: meta in term Namespace (0.035s)
Debug (commands): Adding command: user (0.038s)
Debug (commands): Adding command: meta in user Namespace (0.039s)
Debug (commands): Adding command: session in user Namespace (0.039s)
Debug (commands): Adding command: term in user Namespace (0.039s)
Debug (commands): Adding command: network (0.04s)
Debug (hooks): Processing hook "after_add_command:network" with 1 callbacks (0.04s)
Debug (hooks): On hook "after_add_command:network": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.04s)
Debug (commands): Adding command: meta in network Namespace (0.04s)
Debug (commands): Adding command: db (0.044s)
Debug (commands): Adding command: plugin (0.048s)
Debug (commands): Adding command: theme (0.05s)
Debug (commands): Adding command: mod in theme Namespace (0.05s)
Debug (commands): Adding command: scaffold (0.053s)
Debug (commands): Adding command: core (0.053s)
Debug (commands): Adding command: verify-checksums in core Namespace (0.054s)
Debug (commands): Adding command: verify-checksums in plugin Namespace (0.054s)
Debug (commands): Adding command: export (0.055s)
Debug (commands): Adding command: config (0.057s)
Debug (commands): Adding command: core (0.059s)
Debug (commands): Adding command: eval (0.06s)
Debug (commands): Adding command: eval-file (0.06s)
Debug (commands): Adding command: import (0.061s)
Debug (commands): Adding command: media (0.063s)
Debug (commands): Adding command: package (0.065s)
Debug (commands): Adding command: cron (0.066s)
Debug (commands): Adding command: event in cron Namespace (0.067s)
Debug (commands): Adding command: schedule in cron Namespace (0.067s)
Debug (commands): Adding command: embed (0.067s)
Debug (commands): Adding command: fetch in embed Namespace (0.068s)
Debug (commands): Adding command: provider in embed Namespace (0.069s)
Debug (commands): Adding command: handler in embed Namespace (0.069s)
Debug (commands): Adding command: cache in embed Namespace (0.069s)
Debug (commands): Adding command: i18n (0.07s)
Debug (commands): Adding command: make-pot in i18n Namespace (0.071s)
Debug (commands): Adding command: make-json in i18n Namespace (0.072s)
Debug (commands): Deferring command: language core (0.073s)
Debug (commands): Deferring command: language plugin (0.073s)
Debug (commands): Deferring command: language theme (0.074s)
Debug (hooks): Immediately invoking on passed hook "after_add_command:site": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/language-command/language-command.php at line 39 (0.074s)
Debug (commands): Adding command: switch-language in site Namespace (0.074s)
Debug (commands): Adding command: language (0.074s)
Debug (hooks): Processing hook "after_add_command:language" with 3 callbacks (0.074s)
Debug (hooks): On hook "after_add_command:language": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.074s)
Debug (commands): Adding command: core in language Namespace (0.075s)
Debug (hooks): On hook "after_add_command:language": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.075s)
Debug (commands): Adding command: plugin in language Namespace (0.075s)
Debug (hooks): On hook "after_add_command:language": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.075s)
Debug (commands): Adding command: theme in language Namespace (0.075s)
Debug (commands): Adding command: maintenance-mode (0.076s)
Debug (commands): Adding command: rewrite (0.077s)
Debug (commands): Adding command: rewrite (0.077s)
Debug (commands): Adding command: cap (0.078s)
Debug (commands): Adding command: role (0.078s)
Debug (commands): Adding command: search-replace (0.08s)
Debug (commands): Adding command: server (0.081s)
Debug (commands): Adding command: shell (0.081s)
Debug (commands): Adding command: super-admin (0.082s)
Debug (commands): Adding command: widget (0.083s)
Debug (commands): Adding command: sidebar (0.083s)
Debug (bootstrap): Adding framework command: phar://wp-cli.phar/vendor/wp-cli/wp-cli/php/commands/cli.php (0.083s)
Debug (commands): Adding command: cli (0.085s)
Debug (commands): Adding command: cache in cli Namespace (0.085s)
Debug (commands): Adding command: alias in cli Namespace (0.086s)
Debug (bootstrap): Adding framework command: phar://wp-cli.phar/vendor/wp-cli/wp-cli/php/commands/help.php (0.086s)
Debug (commands): Adding command: help (0.087s)
Debug (bootstrap): No readable global config found (0.087s)
Debug (bootstrap): No project config found (0.087s)
Debug (bootstrap): argv: /usr/local/bin/wp core download --path=cm --debug --locale=de_DE (0.087s)
Debug (bootstrap): ABSPATH defined: /var/www/html/cm/ (0.087s)
Debug (hooks): Processing hook "before_run_command" with 1 callbacks (0.087s)
Debug (hooks): On hook "before_run_command": WP_CLI\Bootstrap\RegisterDeferredCommands->add_deferred_commands() (0.087s)
Debug (bootstrap): Running command: core download (0.087s)
Downloading WordPress 5.2.3 (de_DE)...
md5 hash verified: e87d022e90abefab8a892f4e6c92811a
Success: WordPress downloaded.

and here is wp build

docker run -it --rm \
  --volumes-from wp-container \
  --network wp-network \
  --user 33:33 \
  -e HOME=. \
  cm-wordpress-cli:latest build --clean --yes --path=cm --debug
^@^@Debug (bootstrap): Loading packages from: ./.wp-cli/packages/vendor/autoload.php (0.01s)
Debug (bootstrap): Loading detected autoloader: ./.wp-cli/packages/vendor/autoload.php (0.01s)
Debug (commands): Adding command: build (0.017s)
Debug (commands): Adding command: build-generate (0.017s)
Debug (bootstrap): Fallback autoloader paths: phar://wp-cli.phar/vendor/autoload.php (0.017s)
Debug (bootstrap): Loading detected autoloader: phar://wp-cli.phar/vendor/autoload.php (0.017s)
Debug (commands): Adding command: cache (0.019s)
Debug (commands): Adding command: transient (0.02s)
Debug (commands): Adding command: comment (0.021s)
Debug (commands): Adding command: meta in comment Namespace (0.022s)
Debug (commands): Adding command: menu (0.023s)
Debug (commands): Adding command: item in menu Namespace (0.024s)
Debug (commands): Adding command: location in menu Namespace (0.024s)
Debug (commands): Deferring command: network meta (0.024s)
Debug (commands): Adding command: option (0.026s)
Debug (commands): Adding command: post (0.027s)
Debug (commands): Adding command: meta in post Namespace (0.028s)
Debug (commands): Adding command: term in post Namespace (0.029s)
Debug (commands): Adding command: post-type (0.029s)
Debug (commands): Adding command: site (0.031s)
Debug (commands): Adding command: meta in site Namespace (0.031s)
Debug (commands): Adding command: option in site Namespace (0.032s)
Debug (commands): Adding command: taxonomy (0.032s)
Debug (commands): Adding command: term (0.034s)
Debug (commands): Adding command: meta in term Namespace (0.034s)
Debug (commands): Adding command: user (0.037s)
Debug (commands): Adding command: meta in user Namespace (0.037s)
Debug (commands): Adding command: session in user Namespace (0.038s)
Debug (commands): Adding command: term in user Namespace (0.038s)
Debug (commands): Adding command: network (0.038s)
Debug (hooks): Processing hook "after_add_command:network" with 1 callbacks (0.039s)
Debug (hooks): On hook "after_add_command:network": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.039s)
Debug (commands): Adding command: meta in network Namespace (0.039s)
Debug (commands): Adding command: db (0.043s)
Debug (commands): Adding command: plugin (0.047s)
Debug (commands): Adding command: theme (0.049s)
Debug (commands): Adding command: mod in theme Namespace (0.049s)
Debug (commands): Adding command: scaffold (0.052s)
Debug (commands): Adding command: core (0.052s)
Debug (commands): Adding command: verify-checksums in core Namespace (0.053s)
Debug (commands): Adding command: verify-checksums in plugin Namespace (0.053s)
Debug (commands): Adding command: export (0.054s)
Debug (commands): Adding command: config (0.056s)
Debug (commands): Adding command: core (0.059s)
Debug (commands): Adding command: eval (0.059s)
Debug (commands): Adding command: eval-file (0.059s)
Debug (commands): Adding command: import (0.06s)
Debug (commands): Adding command: media (0.063s)
Debug (commands): Adding command: package (0.065s)
Debug (commands): Adding command: cron (0.065s)
Debug (commands): Adding command: event in cron Namespace (0.066s)
Debug (commands): Adding command: schedule in cron Namespace (0.066s)
Debug (commands): Adding command: embed (0.067s)
Debug (commands): Adding command: fetch in embed Namespace (0.067s)
Debug (commands): Adding command: provider in embed Namespace (0.068s)
Debug (commands): Adding command: handler in embed Namespace (0.068s)
Debug (commands): Adding command: cache in embed Namespace (0.069s)
Debug (commands): Adding command: i18n (0.069s)
Debug (commands): Adding command: make-pot in i18n Namespace (0.07s)
Debug (commands): Adding command: make-json in i18n Namespace (0.071s)
Debug (commands): Deferring command: language core (0.072s)
Debug (commands): Deferring command: language plugin (0.072s)
Debug (commands): Deferring command: language theme (0.073s)
Debug (hooks): Immediately invoking on passed hook "after_add_command:site": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/language-command/language-command.php at line 39 (0.073s)
Debug (commands): Adding command: switch-language in site Namespace (0.073s)
Debug (commands): Adding command: language (0.073s)
Debug (hooks): Processing hook "after_add_command:language" with 3 callbacks (0.073s)
Debug (hooks): On hook "after_add_command:language": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.073s)
Debug (commands): Adding command: core in language Namespace (0.074s)
Debug (hooks): On hook "after_add_command:language": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.074s)
Debug (commands): Adding command: plugin in language Namespace (0.074s)
Debug (hooks): On hook "after_add_command:language": Closure in file phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/class-wp-cli.php at line 634 (0.074s)
Debug (commands): Adding command: theme in language Namespace (0.074s)
Debug (commands): Adding command: maintenance-mode (0.075s)
Debug (commands): Adding command: rewrite (0.076s)
Debug (commands): Adding command: rewrite (0.076s)
Debug (commands): Adding command: cap (0.077s)
Debug (commands): Adding command: role (0.077s)
Debug (commands): Adding command: search-replace (0.079s)
Debug (commands): Adding command: server (0.08s)
Debug (commands): Adding command: shell (0.08s)
Debug (commands): Adding command: super-admin (0.081s)
Debug (commands): Adding command: widget (0.082s)
Debug (commands): Adding command: sidebar (0.082s)
Debug (bootstrap): Adding framework command: phar://wp-cli.phar/vendor/wp-cli/wp-cli/php/commands/cli.php (0.082s)
Debug (commands): Adding command: cli (0.085s)
Debug (commands): Adding command: cache in cli Namespace (0.085s)
Debug (commands): Adding command: alias in cli Namespace (0.086s)
Debug (bootstrap): Adding framework command: phar://wp-cli.phar/vendor/wp-cli/wp-cli/php/commands/help.php (0.086s)
Debug (commands): Adding command: help (0.086s)
Debug (bootstrap): No readable global config found (0.087s)
Debug (bootstrap): No project config found (0.087s)
Debug (bootstrap): argv: /usr/local/bin/wp build --clean --yes --path=cm --debug (0.087s)
Debug (bootstrap): ABSPATH defined: /var/www/html/cm/ (0.087s)
Debug (hooks): Processing hook "before_run_command" with 1 callbacks (0.087s)
Debug (hooks): On hook "before_run_command": WP_CLI\Bootstrap\RegisterDeferredCommands->add_deferred_commands() (0.087s)
Debug (bootstrap): Running command: build (0.087s)
Parsing build.json, please wait...
- Downloading WordPress 5.2.3 (de_DE): [27-Sep-2019 15:07:08 UTC] PHP Fatal error:  Allowed memory size of 2097152 bytes exhausted (tried to allocate 32768 bytes) in phar:///usr/local/bin/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php on line 1474
- Downloading acf-content-analysis-for-yoast-seo (2.3.0): done
- Downloading broken-link-checker (1.11.8): done
- Downloading enable-media-replace (3.3.6): done
- Downloading ewww-image-optimizer (4.9.3): done
- Downloading woocommerce-germanized (2.3.4): done
- Downloading instagram-feed (1.12.2): done
- Downloading rocket-lazy-load (2.3.2): done
- Downloading post-types-order (1.9.4.1): done
- Downloading regenerate-thumbnails (3.1.1): done
- Downloading rename-media-files (1.0.1): done
- Downloading simple-301-redirects (1.07): done
- Downloading w3-total-cache (0.10.1): done
- Downloading woocommerce (3.7.0): done
- Downloading woocommerce-admin (0.19.0): done
- Downloading woocommerce-email-validation (2.1.1): done
- Downloading woocommerce-gateway-stripe (4.2.4): done
- Downloading wordfence (7.4.0): done
- Downloading wp-retina-2x (5.5.7): done
- Downloading wordpress-seo (12.1): done
Finished.
fabioneves commented 4 years ago

I don't have any other suggestions without trying to replicate or testing your use case, sorry.

florianbepunkt commented 4 years ago

If it is alright with you I will leave this open for a few days and try to debug. My guess is that it has to do with the way commands are launched in the Utilities class that gets messy when used in a docker environment. Probably a test case would be to use a docker compose setup as recommended in the wordpress docker repo. But I have to check this first.

florianbepunkt commented 4 years ago

@fabioneves i created a minimal reproducable test case... this is not related to CircleCi, but to using the package in a docker repo.

https://github.com/florianbepunkt/wp-build-docker-test

The steps are straight from the wordpress docker repo HOWTO. I use custom images to copy the build.json and to set the memory limit in the WP CLI image. You can verify that PHP memory limit is set correctly. Also you can see that wp core download works fine

fabioneves commented 4 years ago

I'll have a look at this, since I'll try to optimise the memory used.

florianbepunkt commented 4 years ago

Did some more tests. When I use the wordpress cli image that is based on php 5.6 the error is gone, but the download of the core is completely skipped.

In the Dockerfile of the wp-cli-docker folder I just changed the first line from

FROM wordpress:cli

to

FROM wordpress:cli-php5.6

Again, also with php5.6 image running wp core download works fine

EDIT: Just to clarify… plugins are installed when running the wp build command, but core gets skipped.

florianbepunkt commented 4 years ago

I'm gonna close this. I haven't found the exact root cause, but it is only partially related to this package. Mainly it seems to be a file permission problem related to the Docker setup.

Since the problem doesn't occur when using the native wp core download command, I guess that the way the wp cli core download command is started from this package causes the command to attempt to download the file to a tmp location for which the docker image has no writing permissions. What struck me as odd is that it fails to allocate a file that is much smaller than the actual php memory limit. So it does not seem to be an actual memory limit problem in the first place.

However, when using a Docker Compose setup, the problem no longer occurs.