Documentation on Upgrading Symfony, How Flex Works

brooksvb commented 4 years ago

There is a lack of documentation on how to properly use Symfony Flex.

For example, the behavior described in this comment is not documented anywhere: https://github.com/symfony/flex/issues/433#issuecomment-437325751

The guide for upgrading Symfony doesn't describe how to upgrade with Flex: https://symfony.com/doc/current/setup/upgrade_major.html#update-to-the-new-major-version-via-composer

When it comes to composer I was not very familiar with how everything worked, so trying to upgrade Symfony left me with a lot of questions that don't seem to be answered anywhere. I'm thankful that I was able to have someone explain Flex a bit better and how to modify my composer.json to upgrade by asking in Slack. But this should be available in documentation.

nicolas-grekas commented 4 years ago

Could you please list the specific things you missed and you learned in the process?

brooksvb commented 4 years ago

This is just going to be a brain dump so it might not be great format and it might not all be strictly relevant to Flex. Here is my composer.json before and after for reference

I was able to be ignorant to how flex worked for a while because all I had to do was composer require X for everything like usual, but now I have some nice quick aliases and it does some auto-magical config stuff for me. The problems arose when I wanted to upgrade Symfony.

Why are all these component versions 4.3.5 or similar, but there's a couple that do not follow that scheme ("odd-ones-out")?
What is this extra key at the bottom of my composer.json? What's looking at that or using that and how?
The upgrade documentation says to update the symfony/symfony requirement, but in my case, what do I do? Do I have to change this extra key? Do I need to update every component version? What about the odd-ones-out with the unusual version numbers?
How does flex work? How does it hook into the composer workflow?
Why is there this conflict: symfony/symfony thing? (I realize this is answered briefly on Upgrading Existing Applications to Symfony Flex but isn't this a weird place to find it? If I never upgraded because I started from scratch, how would I figure out what it is?
What is this weird format in scripts: auto-scripts that refers to symfony-cmd? How does that work? I especially feel this should be explained because this is something someone might want to leverage this symfony-cmd more for additional tasks.

Examples of the "odd-ones-out":

symfony/maker-bundle
symfony/orm-pack
symfony/webpack-encore-bundle
symfony/flex

Some key points I learned:

All components need to have their versions updated, but it's more convenient to leave their requirements as *. Normally, this is bad, and my IDE warned me, but in this case it is okay because Symfony flex wraps Composer and will restrict those versions based on the version in the extra data
Some of my questions I figured out through reading the composer documentation in-depth,

Questions I still have:

What exactly is the symfony.lock? Why does it seem to duplicate a lot of information from composer.lock?
I know I can use * version constraints for most of the components, but it's still unclear (and was also unclear to the person in chat helping me) whether Flex will handle this similarly for the "odd-ones-out"
```
"replace": {
    "paragonie/random_compat": "2.*",
    "symfony/polyfill-ctype": "*",
    "symfony/polyfill-iconv": "*",
    "symfony/polyfill-php56": "*",
    "symfony/polyfill-php70": "*",
    "symfony/polyfill-php71": "*"
},
```
? Even after reading the composer documentation on this config key, it doesn't make any sense to me in this context.
Am I able to use Flex for similar behavior for my own open source project or framework (managing the versions of many packages)?

I feel maybe there could be a lot of benefit to inexperienced developers to have a page describing the composer.json file included in the skeleton project, and why things are configured that way.

brooksvb commented 4 years ago

Also I just discovered this in the skeleton:

    "flex-require": {
        "symfony/console": "*",
        "symfony/dotenv": "*",
        "symfony/framework-bundle": "*",
        "symfony/yaml": "*"
    },

This seems like a nicer way to separate the Symfony packages, but I didn't even know this existed. Why doesn't Flex move things into this format? What exactly can and can't I put in there?

mikatrash commented 4 years ago

I share @brooksvb interogations. In my mind Flex/Composer interaction is some kind of magic stuff. Recently migrated 4.3 to 4.4, similar upgrade path to my composer.json with the exception on * version on symfony components dependencies, because some compoments were updated to 5.0 if not locked individualy to ^4. A doc about the points he mentionned should be great, or at least explanations.

lyrixx commented 4 years ago

This is also related to https://github.com/symfony/symfony-docs/issues/12698

stof commented 4 years ago

* Am I able to use Flex for similar behavior for my own open source project or framework (managing the versions of many packages)?

no. The Symfony Flex plugin is specific for Symfony packages, for its version filtering behavior

stof commented 4 years ago

flex-require in the skeleton is something which works only during create-project.

The polyfill replacement are not something specific to Flex. It is a standard composer feature that we use here, to avoid installing polyfills packages which are not needed due to the platform requirements (if you require php >= 7.2, symfony/polyfill-php56 will be useless for instance). And here, * is precisely what we want to expose

Examples of the "odd-ones-out":

symfony/maker-bundle
symfony/orm-pack
symfony/webpack-encore-bundle
symfony/flex

these are packages which are not part of the symfony/symfony packages, and so are versionned on their own.

* What exactly is the `symfony.lock`? Why does it seem to duplicate a lot of information from `composer.lock`?

symfony.lock stores info about which packages got their recipes applied. It is the lock file of Flex.

* The upgrade documentation says to update the `symfony/symfony` requirement, but in my case, what do I do?

Where does it say that ? I don't find anything about that in the doc page you linked.

brooksvb commented 4 years ago

@stof

Where does it say that ? I don't find anything about that in the doc page you linked.

It looks like the page got updated since I originally pointed it out.

And here, * is precisely what we want to expose

I don't think I understand this sentence. But RE the polyfills being listed as replace packages, this is to prevent unnecessarily installing them since we have already require a higher php version?

With that I think most of the information I didn't have, I now have. It just needs to be documented. There could still be some details shared about the flex-require key. There is a reference here https://github.com/symfony/flex/pull/520

gggeek commented 3 years ago

Fully agree that more doc on how Flex works and how it can be tweaked would go a long way in making it more palatable to developers who still want to maintain a little control over their applications, or who are wondering wether they could add flex recipes to their existing public bundles.

Non-exhaustive list of questions off the top of my mind:

what happens on composer update if a flex recipe says to copy eg. a config file from pkg src, and the user has manually modified the target file? Is flex going to do a merge or an overwrite, leaving it to git diff to spot the problems?
how can an end-user dev retrieve the code of the flex recipes which have been executed on composer install/update, to double-check what has happened and what was in fact supposed to happen
is there a dry-run mode to see what flex recipes would do?
why are flex recipes separated from the contents of the packages themselves? is there some architectural benefit to that?
are bundles which use flex recipes supposed to always be useable even without recipe execution, or is it ok to create bundles which are completely unusable unless their flex recipes are run?
which servers does flex talk to to retrieve recipes? are those fixed or configurable?
if the flex-recipes server requires a specific software to manage recipes, is there an oss implementation?
lifecycle of flex recipes: what happens if a recipe has a bug? are recipes versioned independently from the version of the package they refer to? (see also https://github.com/symfony/flex/issues/629)
role of the existing flex console commands compared to only running composer install and composer update
as already stated, role and usage of symfony.lock

Radiergummi commented 2 years ago

To add to this, just as @gggeek, I was wondering why exactly recipes must be separated from the package repository. It would be great to include sample configuration and environment variables with my private package, by simply having a recipe manifest inside the repository.
Instead, we need to copy-paste those things manually after the installation.

So I'm curious why the decision was made to externalise the recipe, and whether there's any way to include it with the package instead.

nicolas-grekas commented 2 years ago

Anyone up for sending PRs to the doc and/or improving the README?

symfony / flex

Documentation on Upgrading Symfony, How Flex Works #575