search

PDMLab / docker-compose

Manage Docker-Compose via Node.js
https://pdmlab.github.io/docker-compose/
MIT License
199 stars 78 forks source link

Cannot create a string longer than 0x1fffffe8 characters #208

Open dugajean opened 2 years ago

dugajean commented 2 years ago

This is happening on this method call:

await dockerCompose.pullAll( dockerComposeConfig );

The full error looks like this:

⠧ Downloading sources.Container 3599aeb67a0e501419c5faae3c737c23-mysql-1  Started
✖ Cannot create a string longer than 0x1fffffe8 characters
Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:621:37)
    at Buffer.toString (node:buffer:812:14)
    at FSReqCallback.readFileAfterClose [as oncomplete] (node:internal/fs/read_file_context:63:23) {
  code: 'ERR_STRING_TOO_LONG'
}

Can't help with the actual source because this is coming from a 3rd party library that I'm using. Right here:

https://github.com/WordPress/gutenberg/blob/trunk/packages/env/lib/commands/start.js#L115

Help would be appreciated.

AlexZeitler commented 2 years ago

0x1fffffe8 is 512 MB. Is this happening on a machine (maybe a CI process) which only provides this amount of RAM?

Steveb-p commented 2 years ago

Depending on the actual code we might be able to circumvent the issue by using Buffer.slice instead. But we would need to pinpoint the exact location where is happens in non-minified code.

Nonetheless, could you provide more context @dugajean ? We had reports related to Wordpress before, but overall it seemed to be working properly. Is there anything different in your application, that could be related?

dugajean commented 2 years ago

@AlexZeitler This is happening on my machine. I have 8 GB of RAM. This never happened on my other machine though, so it could be resources related. But then again, I think it should work on every machie.

@Steveb-p Like I mentioned above, this is coming from wp-env, but I'm on a slightly weaker laptop at this time, which is probably what's causing it. My specs look like this:

Screen Shot 2022-04-06 at 11 07 17 AM

The exact line of code throwing is in wp-env: https://github.com/WordPress/gutenberg/blob/trunk/packages/env/lib/commands/start.js#L115

I believe that simply buffering would solve this issue rather than trying to read a gigantic string at once. Like you said. That's the most common suggestion for this error around the internet.

dugajean commented 2 years ago

Any news here? This is still happening sometimes, unfortunately.

AlexZeitler commented 2 years ago

@dugajean Which images do you try to pull here? @Steveb-p Do you understand where this happens? Is this stdout/stderr buffer?

dugajean commented 2 years ago

@AlexZeitler Here's the docker-compose file

{
    "version": "3.7",
    "services": {
        "mysql": {
            "image": "mariadb",
            "ports": [
                "3306"
            ],
            "environment": {
                "MYSQL_ROOT_PASSWORD": "password",
                "MYSQL_DATABASE": "wordpress"
            },
            "volumes": [
                "mysql:/var/lib/mysql"
            ]
        },
        "tests-mysql": {
            "image": "mariadb",
            "ports": [
                "3306"
            ],
            "environment": {
                "MYSQL_ROOT_PASSWORD": "password",
                "MYSQL_DATABASE": "tests-wordpress"
            },
            "volumes": [
                "mysql-test:/var/lib/mysql"
            ]
        },
        "wordpress": {
            "build": ".",
            "depends_on": [
                "mysql"
            ],
            "image": "wordpress:php7.4",
            "ports": [
                "${WP_ENV_PORT:-8890}:80"
            ],
            "environment": {
                "WORDPRESS_DB_USER": "root",
                "WORDPRESS_DB_PASSWORD": "password",
                "WORDPRESS_DB_NAME": "wordpress"
            },
            "volumes": [
                "/Users/project/local/wordpress:/var/www/html",
                "/Users/project/wp-content/plugins:/var/www/html/wp-content/plugins",
                "/Users/project/wp-content/uploads:/var/www/html/wp-content/uploads",
                "/Users/project/wp-content/themes:/var/www/html/wp-content/themes",
                "/Users/project/wp-content/mu-plugins:/var/www/html/wp-content/mu-plugins"
            ]
        },
        "tests-wordpress": {
            "depends_on": [
                "tests-mysql"
            ],
            "image": "wordpress:php7.4",
            "ports": [
                "${WP_ENV_TESTS_PORT:-8891}:80"
            ],
            "environment": {
                "WORDPRESS_DB_USER": "root",
                "WORDPRESS_DB_PASSWORD": "password",
                "WORDPRESS_DB_NAME": "tests-wordpress",
                "WORDPRESS_DB_HOST": "tests-mysql"
            },
            "volumes": [
                "/Users/username/.wp-env/feeccf209cc7c87fe55317936d452b23/tests-wordpress:/var/www/html",
                "/Users/project/local/wordpress/.htaccess:/var/www/html/.htaccess",
                "/Users/project/local/wordpress/index.php:/var/www/html/index.php",
                "/Users/project/local/wordpress/license.txt:/var/www/html/license.txt",
                "/Users/project/local/wordpress/mysql.sql:/var/www/html/mysql.sql",
                "/Users/project/local/wordpress/phpunit-wp-config.php:/var/www/html/phpunit-wp-config.php",
                "/Users/project/local/wordpress/readme.html:/var/www/html/readme.html",
                "/Users/project/local/wordpress/wp-activate.php:/var/www/html/wp-activate.php",
                "/Users/project/local/wordpress/wp-admin:/var/www/html/wp-admin",
                "/Users/project/local/wordpress/wp-blog-header.php:/var/www/html/wp-blog-header.php",
                "/Users/project/local/wordpress/wp-comments-post.php:/var/www/html/wp-comments-post.php",
                "/Users/project/local/wordpress/wp-cron.php:/var/www/html/wp-cron.php",
                "/Users/project/local/wordpress/wp-includes:/var/www/html/wp-includes",
                "/Users/project/local/wordpress/wp-links-opml.php:/var/www/html/wp-links-opml.php",
                "/Users/project/local/wordpress/wp-load.php:/var/www/html/wp-load.php",
                "/Users/project/local/wordpress/wp-login.php:/var/www/html/wp-login.php",
                "/Users/project/local/wordpress/wp-mail.php:/var/www/html/wp-mail.php",
                "/Users/project/local/wordpress/wp-settings.php:/var/www/html/wp-settings.php",
                "/Users/project/local/wordpress/wp-signup.php:/var/www/html/wp-signup.php",
                "/Users/project/local/wordpress/wp-trackback.php:/var/www/html/wp-trackback.php",
                "/Users/project/local/wordpress/xmlrpc.php:/var/www/html/xmlrpc.php",
                "/Users/project/wp-content/plugins:/var/www/html/wp-content/plugins",
                "/Users/project/wp-content/uploads:/var/www/html/wp-content/uploads",
                "/Users/project/wp-content/themes:/var/www/html/wp-content/themes",
                "/Users/project/wp-content/mu-plugins:/var/www/html/wp-content/mu-plugins"
            ]
        },
        "cli": {
            "depends_on": [
                "wordpress"
            ],
            "image": "wordpress:cli-php7.4",
            "volumes": [
                "/Users/project/local/wordpress:/var/www/html",
                "/Users/project/wp-content/plugins:/var/www/html/wp-content/plugins",
                "/Users/project/wp-content/uploads:/var/www/html/wp-content/uploads",
                "/Users/project/wp-content/themes:/var/www/html/wp-content/themes",
                "/Users/project/wp-content/mu-plugins:/var/www/html/wp-content/mu-plugins"
            ],
            "user": "33:33",
            "environment": {
                "WORDPRESS_DB_USER": "root",
                "WORDPRESS_DB_PASSWORD": "password",
                "WORDPRESS_DB_NAME": "wordpress"
            }
        },
        "tests-cli": {
            "depends_on": [
                "tests-wordpress"
            ],
            "image": "wordpress:cli-php7.4",
            "volumes": [
                "/Users/dugi/.wp-env/feeccf209cc7c87fe55317936d452b23/tests-wordpress:/var/www/html",
                "/Users/project/local/wordpress/.htaccess:/var/www/html/.htaccess",
                "/Users/project/local/wordpress/index.php:/var/www/html/index.php",
                "/Users/project/local/wordpress/license.txt:/var/www/html/license.txt",
                "/Users/project/local/wordpress/mysql.sql:/var/www/html/mysql.sql",
                "/Users/project/local/wordpress/phpunit-wp-config.php:/var/www/html/phpunit-wp-config.php",
                "/Users/project/local/wordpress/readme.html:/var/www/html/readme.html",
                "/Users/project/local/wordpress/wp-activate.php:/var/www/html/wp-activate.php",
                "/Users/project/local/wordpress/wp-admin:/var/www/html/wp-admin",
                "/Users/project/local/wordpress/wp-blog-header.php:/var/www/html/wp-blog-header.php",
                "/Users/project/local/wordpress/wp-comments-post.php:/var/www/html/wp-comments-post.php",
                "/Users/project/local/wordpress/wp-cron.php:/var/www/html/wp-cron.php",
                "/Users/project/local/wordpress/wp-includes:/var/www/html/wp-includes",
                "/Users/project/local/wordpress/wp-links-opml.php:/var/www/html/wp-links-opml.php",
                "/Users/project/local/wordpress/wp-load.php:/var/www/html/wp-load.php",
                "/Users/project/local/wordpress/wp-login.php:/var/www/html/wp-login.php",
                "/Users/project/local/wordpress/wp-mail.php:/var/www/html/wp-mail.php",
                "/Users/project/local/wordpress/wp-settings.php:/var/www/html/wp-settings.php",
                "/Users/project/local/wordpress/wp-signup.php:/var/www/html/wp-signup.php",
                "/Users/project/local/wordpress/wp-trackback.php:/var/www/html/wp-trackback.php",
                "/Users/project/local/wordpress/xmlrpc.php:/var/www/html/xmlrpc.php",
                "/Users/project/wp-content/plugins:/var/www/html/wp-content/plugins",
                "/Users/project/wp-content/uploads:/var/www/html/wp-content/uploads",
                "/Users/project/wp-content/themes:/var/www/html/wp-content/themes",
                "/Users/project/wp-content/mu-plugins:/var/www/html/wp-content/mu-plugins"
            ],
            "user": "33:33",
            "environment": {
                "WORDPRESS_DB_USER": "root",
                "WORDPRESS_DB_PASSWORD": "password",
                "WORDPRESS_DB_NAME": "tests-wordpress",
                "WORDPRESS_DB_HOST": "tests-mysql"
            }
        },
        "composer": {
            "image": "composer",
            "volumes": [
                "/Users/project:/app"
            ]
        },
        "phpunit": {
            "image": "wordpressdevelop/phpunit:8-php-7.4-fpm",
            "depends_on": [
                "tests-wordpress"
            ],
            "volumes": [
                "/Users/dugi/.wp-env/feeccf209cc7c87fe55317936d452b23/tests-wordpress:/var/www/html",
                "/Users/project/local/wordpress/.htaccess:/var/www/html/.htaccess",
                "/Users/project/local/wordpress/index.php:/var/www/html/index.php",
                "/Users/project/local/wordpress/license.txt:/var/www/html/license.txt",
                "/Users/project/local/wordpress/mysql.sql:/var/www/html/mysql.sql",
                "/Users/project/local/wordpress/phpunit-wp-config.php:/var/www/html/phpunit-wp-config.php",
                "/Users/project/local/wordpress/readme.html:/var/www/html/readme.html",
                "/Users/project/local/wordpress/wp-activate.php:/var/www/html/wp-activate.php",
                "/Users/project/local/wordpress/wp-admin:/var/www/html/wp-admin",
                "/Users/project/local/wordpress/wp-blog-header.php:/var/www/html/wp-blog-header.php",
                "/Users/project/local/wordpress/wp-comments-post.php:/var/www/html/wp-comments-post.php",
                "/Users/project/local/wordpress/wp-cron.php:/var/www/html/wp-cron.php",
                "/Users/project/local/wordpress/wp-includes:/var/www/html/wp-includes",
                "/Users/project/local/wordpress/wp-links-opml.php:/var/www/html/wp-links-opml.php",
                "/Users/project/local/wordpress/wp-load.php:/var/www/html/wp-load.php",
                "/Users/project/local/wordpress/wp-login.php:/var/www/html/wp-login.php",
                "/Users/project/local/wordpress/wp-mail.php:/var/www/html/wp-mail.php",
                "/Users/project/local/wordpress/wp-settings.php:/var/www/html/wp-settings.php",
                "/Users/project/local/wordpress/wp-signup.php:/var/www/html/wp-signup.php",
                "/Users/project/local/wordpress/wp-trackback.php:/var/www/html/wp-trackback.php",
                "/Users/project/local/wordpress/xmlrpc.php:/var/www/html/xmlrpc.php",
                "/Users/project/wp-content/plugins:/var/www/html/wp-content/plugins",
                "/Users/project/wp-content/uploads:/var/www/html/wp-content/uploads",
                "/Users/project/wp-content/themes:/var/www/html/wp-content/themes",
                "/Users/project/wp-content/mu-plugins:/var/www/html/wp-content/mu-plugins"
            ],
            "environment": {
                "LOCAL_DIR": "html",
                "WP_PHPUNIT__TESTS_CONFIG": "/var/www/html/phpunit-wp-config.php",
                "WORDPRESS_DB_USER": "root",
                "WORDPRESS_DB_PASSWORD": "password",
                "WORDPRESS_DB_NAME": "tests-wordpress",
                "WORDPRESS_DB_HOST": "tests-mysql"
            }
        }
    },
    "volumes": {
        "mysql": {},
        "mysql-test": {},
        "phpunit-uploads": {}
    }
}
dugajean commented 2 years ago

This is just a matter of the string becoming too large to process.

Steveb-p commented 2 years ago

As far as I know, we are not performing any pre-processing for the service declaration file (configuration). We pass the file name as-is as a parameter to the docker-compose binary - unless the config is provided as a string (using configAsString option).

So more likely culprit would be the output of a container. I'll need to check what we can do to store it in a different manner.

dugajean commented 2 years ago

@Steveb-p How's this looking?