not all symfony packages updated

[BigMichi1]

What Renovate type are you using? Renovate CLI

Describe the bug i have a php/composer project which is based on the symfony skeleton. in my composer.json i have currently this

     "require": {
        "php": "^7.2",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "alterphp/easyadmin-extension-bundle": "2.1.3",
        "friendsofsymfony/user-bundle": "2.1.2",
        "lexik/translation-bundle": "4.0.13",
        "sensio/framework-extra-bundle": "5.3.1",
        "symfony/asset": "4.2.5",
        "symfony/console": "4.2.5",
        "symfony/dotenv": "4.2.5",
        "symfony/expression-language": "4.2.5",
        "symfony/flex": "1.2.2",
        "symfony/form": "4.2.5",
        "symfony/framework-bundle": "4.2.5",
        "symfony/monolog-bundle": "3.3.1",
        "symfony/orm-pack": "1.0.6",
        "symfony/process": "4.2.5",
        "symfony/requirements-checker": "1.1.4",
        "symfony/security-bundle": "4.2.5",
        "symfony/serializer-pack": "1.0.2",
        "symfony/swiftmailer-bundle": "3.2.5",
        "symfony/translation": "4.2.5",
        "symfony/twig-bundle": "4.2.5",
        "symfony/validator": "4.2.5",
        "symfony/web-link": "4.2.5",
        "symfony/webpack-encore-bundle": "1.5.0",
        "symfony/yaml": "4.2.5"
    },
    "replace": {
        "paragonie/random_compat": "2.*",
        "symfony/polyfill-ctype": "*",
        "symfony/polyfill-iconv": "*",
        "symfony/polyfill-php56": "*",
        "symfony/polyfill-php70": "*",
        "symfony/polyfill-php71": "*"
    },
    "conflict": {
        "symfony/symfony": "*"
    },
    "require-dev": {
        "localheinz/composer-normalize": "1.1.3",
        "symfony/debug-pack": "1.0.7",
        "symfony/maker-bundle": "1.11.5",
        "symfony/profiler-pack": "1.0.4",
        "symfony/test-pack": "1.0.5",
        "symfony/web-server-bundle": "4.2.5"
    },
    "config": {
        "optimize-autoloader": true,
        "platform": {
            "php": "7.2.16"
        },
        "preferred-install": {
            "*": "dist"
        },
        "sort-packages": true
    },
    "extra": {
        "symfony": {
            "allow-contrib": false,
            "require": "4.2.5"
        }
    },

when i now run renovate it results in a merge request (want's to update 4.2.5 to 4.2.7) for only the specified symfony packages from composer.json. so the update in the composer.lock file still contains other symfony packages with the old version. also the version in the extra section is not updated to the new one.

Did you see anything helpful in debug logs?

INFO: Manual rebase requested via PR checkbox for #4 (repository=php/test-symfony, dependencies=symfony/asset,symfony/console,symfony/dotenv,symfony/expression-language,symfony/flex,symfony/form,symfony/framework-bundle,symfony/process,symfony/security-bundle,symfony/translation,symfony/twig-bundle,symfony/validator,symfony/web-link,symfony/web-server-bundle,symfony/yaml, branch=renovate/symfony)
 INFO: Running composer via global composer (repository=php/test-symfony, dependencies=symfony/asset,symfony/console,symfony/dotenv,symfony/expression-language,symfony/flex,symfony/form,symfony/framework-bundle,symfony/process,symfony/security-bundle,symfony/translation,symfony/twig-bundle,symfony/validator,symfony/web-link,symfony/web-server-bundle,symfony/yaml, branch=renovate/symfony)
 INFO: Generated lockfile (repository=php/test-symfony, dependencies=symfony/asset,symfony/console,symfony/dotenv,symfony/expression-language,symfony/flex,symfony/form,symfony/framework-bundle,symfony/process,symfony/security-bundle,symfony/translation,symfony/twig-bundle,symfony/validator,symfony/web-link,symfony/web-server-bundle,symfony/yaml, branch=renovate/symfony)
       "seconds": 182,
       "type": "composer.lock",
       "stdout": "",
       "stderr": "Do not run Composer as root/super user! See https://getcomposer.org/root for details\nLoading composer repositories with package information\nUpdating dependencies (including require-dev)\nPackage operations: 123 installs, 0 updates, 0 removals\n  - Installing symfony/flex (v1.2.3): Downloading (100%)\n  - Installing ocramius/package-versions (1.4.0): Downloading (100%)\n  - Installing pyrech/composer-changelogs (v1.6.0): Downloading (100%)\n  - Installing sebastian/diff (3.0.2): Downloading (100%)\n  - Installing localheinz/json-printer (2.0.1): Downloading (100%)\n  - Installing justinrainbow/json-schema (5.2.8): Downloading (100%)\n  - Installing localheinz/json-normalizer (0.9.0): Downloading (100%)\n  - Installing localheinz/composer-json-normalizer (1.0.2): Downloading (100%)\n  - Installing localheinz/composer-normalize (1.1.3): Downloading (100%)\n  - Installing symfony/dotenv (v4.2.7): Downloading (100%)\n  - Installing psr/link (1.0.0): Downloading (100%)\n  - Installing fig/link-util (1.0.0): Downloading (100%)\n  - Installing symfony/web-link (v4.2.7): Downloading (100%)\n  - Installing symfony/process (v4.2.7): Downloading (100%)\n  - Installing symfony/polyfill-mbstring (v1.11.0): Downloading (100%)\n  - Installing symfony/http-foundation (v4.2.7): Downloading (100%)\n  - Installing symfony/contracts (v1.0.2): Downloading (100%)\n  - Installing symfony/event-dispatcher (v4.2.7): Downloading (100%)\n  - Installing psr/log (1.1.0): Downloading (100%)\n  - Installing symfony/debug (v4.2.7): Downloading (100%)\n  - Installing symfony/http-kernel (v4.2.7): Downloading (100%)\n  - Installing psr/container (1.0.0): Downloading (100%)\n  - Installing symfony/dependency-injection (v4.2.7): Downloading (100%)\n  - Installing symfony/console (v4.2.7): Downloading (100%)\n  - Installing symfony/filesystem (v4.2.7): Downloading (100%)\n  - Installing symfony/config (v4.2.7): Downloading (100%)\n  - Installing symfony/web-server-bundle (v4.2.7): Downloading (100%)\n  - Installing twig/twig (v2.8.1): Downloading (100%)\n  - Installing symfony/validator (v4.2.7): Downloading (100%)\n  - Installing symfony/inflector (v4.2.7): Downloading (100%)\n  - Installing symfony/property-access (v4.2.7): Downloading (100%)\n  - Installing symfony/routing (v4.2.7): Downloading (100%)\n  - Installing symfony/finder (v4.2.7): Downloading (100%)\n  - Installing symfony/var-exporter (v4.2.7): Downloading (100%)\n  - Installing psr/simple-cache (1.0.1): Downloading (100%)\n  - Installing psr/cache (1.0.1): Downloading (100%)\n  - Installing symfony/cache (v4.2.7): Downloading (100%)\n  - Installing symfony/framework-bundle (v4.2.7): Downloading (100%)\n  - Installing symfony/options-resolver (v4.2.7): Downloading (100%)\n  - Installing symfony/intl (v4.2.7): Downloading (100%)\n  - Installing symfony/polyfill-intl-icu (v1.11.0): Downloading (100%)\n  - Installing symfony/form (v4.2.7): Downloading (100%)\n  - Installing psr/http-message (1.0.1): Downloading (100%)\n  - Installing league/uri-parser (1.4.1): Downloading (100%)\n  - Installing league/uri-interfaces (1.1.1): Downloading (100%)\n  - Installing league/uri-schemes (1.2.1): Downloading (100%)\n  - Installing league/uri-hostname-parser (1.1.1): Downloading (100%)\n  - Installing league/uri-components (1.8.2): Downloading (100%)\n  - Installing league/uri-manipulations (1.5.0): Downloading (100%)\n  - Installing twig/extensions (v1.5.4): Downloading (100%)\n  - Installing symfony/twig-bridge (v4.2.7): Downloading (100%)\n  - Installing symfony/twig-bundle (v4.2.7): Downloading (100%)\n  - Installing symfony/translation (v4.2.7): Downloading (100%)\n  - Installing symfony/security-core (v4.2.7): Downloading (100%)\n  - Installing symfony/security-http (v4.2.7): Downloading (100%)\n  - Installing symfony/security-guard (v4.2.7): Downloading (100%)\n  - Installing symfony/security-csrf (v4.2.7): Downloading (100%)\n  - Installing symfony/security-bundle (v4.2.7): Downloading (100%)\n  - Installing symfony/expression-language (v4.2.7): Downloading (100%)\n  - Installing doctrine/lexer (v1.0.1): Downloading (100%)\n  - Installing doctrine/annotations (v1.6.1): Downloading (100%)\n  - Installing doctrine/reflection (v1.0.0): Downloading (100%)\n  - Installing doctrine/event-manager (v1.0.0): Downloading (100%)\n  - Installing doctrine/collections (v1.6.1): Downloading (100%)\n  - Installing doctrine/cache (v1.8.0): Downloading (100%)\n  - Installing doctrine/persistence (v1.1.0): Downloading (100%)\n  - Installing symfony/doctrine-bridge (v4.2.5): Downloading (100%)\n  - Installing doctrine/inflector (v1.3.0): Downloading (100%)\n  - Installing doctrine/common (v2.10.0): Downloading (100%)\n  - Installing symfony/asset (v4.2.7): Downloading (100%)\n  - Installing pagerfanta/pagerfanta (v2.1.2): Downloading (100%)\n  - Installing doctrine/instantiator (1.2.0): Downloading (100%)\n  - Installing doctrine/dbal (v2.9.2): Downloading (100%)\n  - Installing doctrine/orm (v2.6.3): Downloading (100%)\n  - Installing jdorn/sql-formatter (v1.2.17): Downloading (100%)\n  - Installing doctrine/doctrine-cache-bundle (1.3.5): Downloading (100%)\n  - Installing doctrine/doctrine-bundle (1.10.2): Downloading (100%)\n  - Installing easycorp/easyadmin-bundle (v2.1.1): Downloading (100%)\n  - Installing alterphp/easyadmin-extension-bundle (v2.1.3): Downloading (100%)\n  - Installing symfony/templating (v4.2.5): Downloading (100%)\n  - Installing friendsofsymfony/user-bundle (v2.1.2): Downloading (100%)\n  - Installing lexik/translation-bundle (v4.0.13): Downloading (100%)\n  - Installing sensio/framework-extra-bundle (v5.3.1): Downloading (100%)\n  - Installing symfony/stopwatch (v4.2.5): Downloading (100%)\n  - Installing zendframework/zend-eventmanager (3.2.1): Downloading (100%)\n  - Installing zendframework/zend-code (3.3.1): Downloading (100%)\n  - Installing ocramius/proxy-manager (2.2.2): Downloading (100%)\n  - Installing doctrine/migrations (v2.0.0): Downloading (100%)\n  - Installing doctrine/doctrine-migrations-bundle (v2.0.0): Downloading (100%)\n  - Installing symfony/yaml (v4.2.7): Downloading (100%)\n  - Installing symfony/orm-pack (v1.0.6): Downloading (100%)\n  - Installing symfony/requirements-checker (v1.1.4): Downloading (100%)\n  - Installing symfony/serializer (v4.2.5): Downloading (100%)\n  - Installing symfony/property-info (v4.2.5): Downloading (100%)\n  - Installing webmozart/assert (1.4.0): Downloading (100%)\n  - Installing phpdocumentor/reflection-common (1.0.1): Downloading (100%)\n  - Installing phpdocumentor/type-resolver (0.4.0): Downloading (100%)\n  - Installing phpdocumentor/reflection-docblock (4.3.0): Downloading (100%)\n  - Installing symfony/serializer-pack (v1.0.2): Downloading (100%)\n  - Installing symfony/polyfill-php72 (v1.11.0): Downloading (100%)\n  - Installing symfony/polyfill-intl-idn (v1.11.0): Downloading (100%)\n  - Installing egulias/email-validator (2.1.7): Downloading (100%)\n  - Installing swiftmailer/swiftmailer (v6.2.0): Downloading (100%)\n  - Installing symfony/swiftmailer-bundle (v3.2.5): Downloading (100%)\n  - Installing symfony/webpack-encore-bundle (v1.5.0): Downloading (100%)\n  - Installing symfony/var-dumper (v4.2.5): Downloading (100%)\n  - Installing symfony/web-profiler-bundle (v4.2.5): Downloading (100%)\n  - Installing symfony/profiler-pack (v1.0.4): Downloading (100%)\n  - Installing monolog/monolog (1.24.0): Downloading (100%)\n  - Installing easycorp/easy-log-handler (v1.0.7): Downloading (100%)\n  - Installing symfony/monolog-bridge (v4.2.5): Downloading (100%)\n  - Installing symfony/monolog-bundle (v3.3.1): Downloading (100%)\n  - Installing symfony/debug-bundle (v4.2.5): Downloading (100%)\n  - Installing symfony/debug-pack (v1.0.7): Downloading (100%)\n  - Installing nikic/php-parser (v4.2.1): Downloading (100%)\n  - Installing symfony/maker-bundle (v1.11.5): Downloading (100%)\n  - Installing symfony/phpunit-bridge (v4.2.5): Downloading (100%)\n  - Installing symfony/dom-crawler (v4.2.5): Downloading (100%)\n  - Installing symfony/browser-kit (v4.2.5): Downloading (100%)\n  - Installing facebook/webdriver (1.6.0): Downloading (100%)\n  - Installing symfony/panther (v0.3.0): Downloading (100%)\n  - Installing symfony/css-selector (v4.2.5): Downloading (100%)\n  - Installing symfony/test-pack (v1.0.5): Downloading (100%)\nWriting lock file\n"
 INFO: Branch updated (repository=php/test-symfony, dependencies=symfony/asset,symfony/console,symfony/dotenv,symfony/expression-language,symfony/flex,symfony/form,symfony/framework-bundle,symfony/process,symfony/security-bundle,symfony/translation,symfony/twig-bundle,symfony/validator,symfony/web-link,symfony/web-server-bundle,symfony/yaml, branch=renovate/symfony)
 INFO: PR updated (repository=php/test-symfony, dependencies=symfony/asset,symfony/console,symfony/dotenv,symfony/expression-language,symfony/flex,symfony/form,symfony/framework-bundle,symfony/process,symfony/security-bundle,symfony/translation,symfony/twig-bundle,symfony/validator,symfony/web-link,symfony/web-server-bundle,symfony/yaml, branch=renovate/symfony)
       "committedFiles": true,
       "pr": 4

To Reproduce

1. create a symfony project from the skeleton
2. set symfony version to an older version than the current one
3. renovate

Expected behavior

1. should also update the version in the extra section in the composer.json
2. should also update other symfony packages which are in composer.lock

Screenshots none available

Additional context currently i'm using renovate in version 16.0.4

[BigMichi1]

when i run composer out after i merged the pull request the output is the following

root@i7-4770 ~/test-symfony # composer out
Do not run Composer as root/super user! See https://getcomposer.org/root for details
symfony/browser-kit         v4.2.5 v4.2.7 Symfony BrowserKit Component
symfony/css-selector        v4.2.5 v4.2.7 Symfony CssSelector Component
symfony/debug-bundle        v4.2.5 v4.2.7 Symfony DebugBundle
symfony/doctrine-bridge     v4.2.5 v4.2.7 Symfony Doctrine Bridge
symfony/dom-crawler         v4.2.5 v4.2.7 Symfony DomCrawler Component
symfony/monolog-bridge      v4.2.5 v4.2.7 Symfony Monolog Bridge
symfony/phpunit-bridge      v4.2.5 v4.2.7 Symfony PHPUnit Bridge
symfony/property-info       v4.2.5 v4.2.7 Symfony Property Info Component
symfony/serializer          v4.2.5 v4.2.7 Symfony Serializer Component
symfony/stopwatch           v4.2.5 v4.2.7 Symfony Stopwatch Component
symfony/templating          v4.2.5 v4.2.7 Symfony Templating Component
symfony/var-dumper          v4.2.5 v4.2.7 Symfony mechanism for exploring and dumping PHP variables
symfony/web-profiler-bundle v4.2.5 v4.2.7 Symfony WebProfilerBundle

[rarkins]

The ones you say are not updated are indirect dependencies?

[rarkins]

If you wanted to manually update all symfony packages from 4.2.5 to 4.2.7 - and not update any non-symfony packages - which commands would you run to do it?

[BigMichi1]

as an example symfony/phpunit-bridge v4.2.5 v4.2.7 this package is a transitive dependency of "symfony/test-pack": "1.0.5" which hasn't been updated, but mixing versions might break symfony, looks like it is not so easy achievable without running a global composer update which might than also bring in outher dependecy updates from transitive dependencies

[rarkins]

@BigMichi1 I think you and I are thinking along the same lines now. I agree that you should ideally have all symfony packages on the same version in a project, and in fact maybe in some cases you "must" have them all on the same. But when they're deeply nested packages inside the lock file, I'm not sure how you'd achieve this - whether as a bot or as a human. If you update every dependency in the lock file then you might end up changing 10x more versions than just symfony.

Longer term the solution is that Renovate would have the capability to update nested dependencies in lock files, but by default have that functionality would be disabled because it would be too noisy otherwise. Then in cases like this you could have the capability to turn nested updates on just for symfony packages.

[msheakoski]

The issue occurs when Symfony does a major or minor version upgrade. Renovate correctly updates the require and require-dev sections, but there is another part in composer.json that also needs to be changed because it restricts the maximum allowed version of the symfony/* packages:

"extra": {
  "symfony": {
    "allow-contrib": false,
    // This should match the version of Symfony packages in the require/require-dev sections
    "require": "5.1.*" // <---
  }
}

These links explain in more detail:

• Upgrading a Major Version (e.g. 4.4.0 to 5.0.0)
• Upgrading a Minor Version (e.g. 4.0.0 to 4.1.0)
• Upgrading a Patch Version (e.g. 5.0.0 to 5.0.1)

[rarkins]

@msheakoski good find, thanks. I'm confused though because the docs for this extra field do not include any mention of the meaning of "require": https://getcomposer.org/doc/04-schema.md#extra

I was trying to work out if this extra field is used by other packages or just symfony ones.

If we are trying to solve symfony alone, would it work if we treat extra.symfony.requires as if it represents the package symfony/symfony?

[msheakoski]

@rarkins The symfony/flex package registers a composer plugin which uses the configuration in composer.json's extra.symfony.* section.

I'm not an expert in this domain, but I believe that the version in extra.symfony.require is the source of truth for the constraints of symfony/* packages.

The question is how to determine what the latest major/minor/patch version of Symfony is. Because symfony/* packages do not all have their versions in sync with each other, there has to be another way. As you suggested, symfony/symfony might be the best package to monitor. Perhaps @fabpot or @nicolas-grekas can offer some advice on this.

[nicolas-grekas]

extra.symfony.require applies only to packages that are listed in the replace section of symfony/symfony. symfony/phpunit-bridge is not one of them, neither is e.g symfony/monolog-bundle.

[msheakoski]

@nicolas-grekas Interesting, I always wondered how that worked!

@rarkins I think that it would be safe to treat extra.symfony.require the same way that you would treat symfony/symfony. This change should fix the issue of keeping Symfony dependencies up to date.

Packages like symfony/monolog-bundle, which are not affected by extra.symfony.require, look like they can be updated the usual way without any issues, so nothing special needs to be done, just the usual require/require-dev version bumping.

[rarkins]

Thanks @msheakoski @nicolas-grekas.

Yes, it seems like we can use symfony/symfony as the package to determine the value in extra.symfony.require.

The next question I have is whether we need to modify Renovate's default installation of composer. e.g. do we need to install the Symfony Flex plugin so that the composer.lock is updated correctly?

https://github.com/renovatebot/docker-buildpack/blob/master/src/php/build/composer.sh

[msheakoski]

@rarkins It should work without any changes to the buildpack. If extra.symfony.require is present, it can be assumed that symfony/flex is already listed in the require section of composer.json and everything will update accordingly.

[rarkins]

Great. Last remaining step: can someone create a simple public repo that reproduces the current Renovate not working correctly?

[msheakoski]

@rarkins See https://github.com/msheakoski/issue-3558-symfony

It should want to update from 5.0.9 -> 5.1.0.

[rarkins]

@msheakoski thanks. I need your help understanding what the desired behavior is. For example the first PR raised is this one:

Renovate's default behavior is to "pin" dependencies for applications so that's what you're seeing. Should it be pinning extras.symfony.require as well? Or you prefer to not pin dependencies at all in this case, etc?

[msheakoski]

@rarkins Yes, I believe that extras.symfony.require should be pinned to whatever version that symfony/symfony would be pinned as.

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team