Support for bundle inline recipe configuration

[linaori]

Now that Flex has matured a bit, I would like to bring this point up again. Currently, having dozens of internal bundles (and several open-source), having dozens of applications using those bundles, we prefer to follow the recommendations and standards as much as possible. This means that we will start using Flex from 4.0 on.

The selling point of flex, is the ability to provide an "installation" into the application by using manifests. However, this being in an external repository, poses several issues:

• It doesn't work for closed source bundles, I can't add them.
• Double bookkeeping, if I add a new configuration option or provide something new, I have to remember to update the recipe in another repository as well.
• If I would use a closed source variant, I'd need to set this up and maintain this repository

To me, these are a few blockers that prevent Flex from being useful for me. I have dozens of bundles that have a mandatory configuration, but I won't be able to utilize flex to increase the Developer experience. It will actually make behavior different between the recipe'd packages and internal packages when it comes to installing, making it annoying to find out whether or not you still have to add some config file for example.

---

Recipes must be stored on their own repositories, outside of your Composer package repository https://github.com/symfony/recipes/blob/master/README.rst#creating-recipes

I'm not sure if this is a technical limitation, but I would love to be able to do the exact opposite, inline the configuration in my package. This gives me several benefits:

• My package provides optional integration into flex (if used).
• I can control multiple version based on the current version of the files, rather than making a directory per version.
• When I release my bundle version, I don't have to rely on the recipe repository to make it possible to install the newest config, I control the release.
• I won't forget to update the recipe repository, nor does someone else have to do this when a new feature is introduced.
• No need for privately hosted repositories, mirrors or proxies.

---

The directory structure of an inlined flex config could be as simple as adding a .flex directory or something similar.

[Grafikart]

I think this is a must have. A simple solution would be to add a flex.json file within a project. Then, when we do a composer require the post-package-install hook will look for this file and run the recipe if needed.

It's more convenient for a developer to manage the recipes from his repository. And it will allow private recipes easily.

I'll have a look at the source code this week to see if it's something that could be implemented easily or not.

[nicolas-grekas]

To me, inline recipes are like DI by annotations: something we don't want to support because it's putting the responsibility in the wrong hands. Configuration/wiring belongs to the consumer side, not to the code author's. Having configuration in dedicated repositories proves every day to be improving a lot the quality of the ecosystem, by forcing authors to have a discussion about how they shaped their bundles. See what's happening here, that's real. This way more important than providing convenience, but a very loose one in end.

[linaori]

To me, inline recipes are like DI through annotations: something we don't want to support because it's putting the responsibility in the wrong hands.

I disagree on the DI through annotations part, it feels more like configuring routes to me, it provides the link between the bundle (a route) and the framework (router). The responsibility is not in the wrong hands, it's still the author's responsibility to make it work, except that inlining it, will make the author's life a lot easier.

Configuration/wiring belongs to the consumer side, not to the code author's.

The consumer won't even notice this unless the consumer opens the directory containing this config.

Having configuration in dedicated repositories proves every day to be improving a lot the quality of the ecosystem.

Having them in different git repositories is quite cumbersome, a lot of extra work and a lot of problems keeping things in sync and test them properly. With a dedicated repository, there's need for a staging environment to test if your recipe works. With inline repositories, you merely have to install the fork or branch.

by forcing authors to have a discussion about how they shaped their bundles. See what's happening here, that's real.

Yes, this is indeed a benefit of the collective repository.

---

Long story short: I don't want the repository to be removed, I think it should still be leading. However, Symfony is recommending us to use flex, supply all kind of fancy tools, and we still have to do it the old fashioned way, because we can't open-source the majority of our bundles. We can't use a private github repository either, because then we still have to place our code external, rather than in-house. On top of that, maintaining an in-house repository, still has all the overhead I've mentioned, which we are not looking for.

When we have version 10.0.0 of a bundle, we have no need to keep the config of 3.0.0. Yes, we can delete those directories, but if we have 2 majors running side-by-side (for a few days max), there's so much extra work and overhead, that maintaining it, is not worth it. There's just too much that can (and thus will) go wrong when you keep it in an external repository.

[fabpot]

"We can't use a private github repository either, because then we still have to place our code external": only the recipes repository must be on Github.

[linaori]

@fabpot You're right, but it would still "leak" outside, which is the problem

[Grafikart]

If you think it's a security problem to let developer manage their configuration (without being validated) we could make this option optin or ask the end user if he really wants to use the recipe (like you did for contrib https://github.com/symfony/flex/blob/master/src/Flex.php#L158)

[covex-nn]

There are 3 types of packages:

• Packages, useful for many people
• Set-of-files, useful only for their owner/creator
• Private packages

I think, that recipes and recipes-contrib are only for a first type. For the rest - keeping "recipes" in a same repository is the only way to re-use it with Fiex. So now, i have two options:

• Keep my own symfony-standard-edition with all my files
• Split my SE into parts and install them with my own composer plugin over Flex, when necessary

But, I do not like any: SE contains a lot of extra files, and plugin is a crutch =(

[covex-nn]

@Pierstoval copy-from-package is a great feature, somehow i missed it when I read it for the first time about recipes. It is suitable for all kinds of packages, except private of course.

Anyway, i've created a composer command, that allows to apply recipe manually right from the package. That command does not use symfony.sh, recipes and recipes-contrib. It just reads manifest.json from package and installs a "recipe". And it does not update composer.json after applying. How do you think, is there a place for such command in Flex?

[francoispluchino]

@covex-nn There is now a solution for private repositories, but read my entire comment to know the limitations.

@iltar, @Grafikart I had already explained the problem of private repositories and presented solutions, but the Symfony team explained very well their motivations and their reasons for refusing to use any other means than the official and contrib recipes on Symfony.sh.

Of course, like you, I proposed the addition of the manifest in the component, what you call an 'inline configurationwith theflex.json` file, to be able to use Symfony Flex with private repositories (issue #71). Very good, a solution has been recently introduced for private repositories, but it is limited to private Github repositories, and connected to Symfony.sh's Github application. The problem is that repositories can be present on other platforms like Gitlab, Bitbucket, AWS CodeCommit, Mercurial, Fossil, SVN, CVS, local, etc ..., and in this case, it does not exist of solution currently.

@fabpot only the recipes repository must be on Github., yes I agree with you, however, you add a useless cost of $300/year for private repositories, and unnecessary complexity in the process of a team not using Github.

I understand that the Symfony team wants to talk with the developers of third-party components to make the Symfony ecosystem better, and therefore to lock it as it wishes, it is legitimate, but debatable.

Given that the Symfony team does not want to take responsibility for a likely bad quality on the configuration of the third-party component (because not verified by them or the community), I tried to express my vision of what Symfony Flex can represent for the third-party developers. I invite you to read my very (too?) long comment) in the issue #70, as well as the following different answers. I sincerely believe that Symfony Flex should move in this direction, despite the reluctance of the Symfony team.

@nicolas-grekas On this subject, and with all due respect, say that To me, inline recipes are like DI by annotations: something we don't want to support because it's putting the responsibility in the wrong hands. this is really a pseudo argument. Who better than the developers of the component to propose a configuration??? Of course, the community can make excellent proposals to projects, but also to components.

@nicolas-grekas Configuration/wiring belongs to the consumer side, not to the code author's. I agree and disagree with you on this point. It's good that the recipe is not in the component that wants to be generic and not oriented for a dedicated framework, but it is also interesting to have the recipe in the component for components dedicated to a project, such as plugins, themes, etc. of a modular project.

Of course, Symfony Flex should not offer everything and anything, but I think it should purpose a common base, and that's why I see much more Symfony Flex as a great automation tool to standardize the project configuration system, than a simple tool dedicated to the simplification of the Symfony's project configurations, despite what the team say, and even if you have added directly into the plugin a special configurator for the Symfony Bundles (Symfony\Flex\Configurator\BundlesConfigurator).

From this discussion, I therefore proposed in the issue #173 to add the ability to extend Symfony Flex, just as we can extend Composer, and I have propose in the PR #174, a simpler implementation than that presented in this issue. Several advantages to this solution:

• There is no need to reinvent the wheel and forked the Symfony Flex, but have a common and standardized base
• Add new configuration features not originally planned, like as adding new Configurators and Downloaders (such as using recipes in the component folder or on a private server, or other)
• All recipes added by a plugin are considered like a recipes-contrib, so, developers may or may not agree to use it
• Other examples and arguments are in this comment and his answers

Of course, the Symfony team does not agree to add a feature that for them, will not be used by the community. This results in an unnecessary maintenance cost for a feature that is not really useful for the majority of developers, and I can understand if they are right. In ths case, it's up to us developers of the Symfony community, to show that this feature is claimed, if you find it interesting.

And yes, I come back to the charge on this subject, but it's for a good cause ;-). I sincerely believe that this feature will be widely used for many projects, especially large open source or private modular projects. Symfony Flex could become an excellent tool for the configuration of projects and allow the community to design new Flex-based tools to do things that the Symfony's team did not initially envision or want to support, such as a plugin that supports recipes installed in the component.

It is obvious that I will still be satisfied if you only accept the inline configuration, although I think that giving the opportunity to extend Flex would be better.

Thank you for your great works, and have a good end-of-year parties :-)

[rrajkomar]

Hi, I totally agree on the fact that limiting the flex server to Github only will be an issue especially for private projects/repositories that may not necessarily be stored on Github. Allowing either inline manifest config (which may be used only by those interested by said feature) or making a version of the app available for other systems (which may not be practical because it wouldn't be simple to maintain) is a must in order to give all potential flex users access to the full potential of this amazing tool. Personally I find this tool very useful but unfortunately I just can't use it on a vast majority of my projects because part of said projects use private code and I do not use nor do I want to use Github for my projects. Therefore I'd be very interested by having this inline manifest configuration available ; such a feature could be used by the projects that can't used the contrib or official recipes repository for legal or other reasons. Another option could be to allow a configuration variable to specify the package recipe repository url.

[nicolas-grekas]

Flex recipes are validated and that's 100% part of the Flex experience. Inline recipes bypass any such validation. Also, asking contributors of a free plugin to develop and maintain a set of features for private needs is an economic nonsense. You can create your own plugin if you want to propose an alternative way to ship default configuration. I wish you all the best to justify the time you're going to spend on it for private projects. GitHub is also a false issue: only recipes must be on it, not projects themselves.

[rrajkomar]

@nicolas-grekas : True, only recipes are placed on github but what is the sense in putting a recipe on a github repository when the only projects that would be using it are in a private self hosted gitlab intance for example... quick example : you're creating a proprietary application that uses symfony and want to use symfony-flex. I truly don't get what would be the point in putting the related recipes on a github project (even privately owned, thus paying a fee for something that you could host on your self hosted server for free...) Don't get me wrong the recipes and recipes-contrib are a great thing for shared packages recipes but not for private ones and enforcing the use of a non free private repository on github is not the solution IMHO. Also based on the fact that since the packages maintainers are the ones who should write the recipes, I just don't get why giving them the ability to add them directly in the package if they want is wrong. Sure it bypasses the whole recipes-contrib checks but a simple warning on the package install should suffice to warn about it. If an end user installs it, he will be made aware that it was not validated as packages from recipes or recipes-contrib are.

[rrajkomar]

P.S. : Then again I might be wrong but I don't think I would be the only one thinking that way. Only time will tell. But I strongly believe that not giving an alternative for private packages (and by that I mean not just a non free one) will be somewhat a blocker for the maintainers of privately owned packages.

Note : the issue could also be approached the other way around : the symfony.sh process could be adapted to either be able to run on a remote git based repository or to be able to run against recipes from a remote git based repository (publicly available by connecting to it using credentials, which could be those of a github account)...

I do not have any insights on how the process in question works so I can only make suggestions, which could be possible or not.

[moay]

I strongly agree with @rrajkomar - not being able to use our private packages in the build process is a strong blocker. We have a lot of internal packages that we cannot (not permitted by the customers) host anywhere else but that need to be integrated in Symfony. Think of a monitoring package, some connection to internal systems and so on. If we can't load them via Flex, this takes the fun out of it and will possibly break Flex, because we still need to register stuff manually (which is by far more prone to errors than having our own flex recipes and packages which we take our own responsibility for.

As an alternative to this, we could of course maintain our own standard edition of Symfony, but that's kind of the opposite of what Symfony 4 should become...

For us, the ability to load internal packages that we maintain on our local gitlab server is a deal breaker for Flex usage.

[slootjes]

I think inline recipes would be great and when detected it could ask the user if it wants to use them. I'm too very interested in creating recipes for some of our internal bundles.

The alternative solution which I'm testing with now is to run a custom version of flex server https://github.com/aurimasniekis/flex-server and a private (non-github) repository where I store the recipes. This seems to work very good but I run into issues with recipes which would require sessions and security changes which is currently unsupported.

[tseho]

The current private recipes beta does mention a future price of 249€/month. You can check by yourself: https://symfony.sh/z/register

If the unconfirmed intention of having private recipes as a paid feature, It would make sense they are opposing any inline recipe configuration.

It would be sad, as I was hoping to use this feature for our private repositories.

[francoispluchino]

@tseho Thank you for your link, I thought the integration would be free as nothing was mentioned in this way in the links provided by @fabpot in my issue #71, and that your page is not referenced on the Symfony.sh website. So now I understand better their confirmed intentions, and this justifies much more the unwillingness of the Symfony team to add other solution allowing to use recipes for private packages.

This remains a choice, and I do not despair to see the Symfony team reconsider their choice, allowing the community to add functionality to Flex with plugins (#174), if they do not want to support them natively (this issue).

However, other service with the same principle, does not force their users to subscribe to a paid offer to use an open source tool with private repositories. Take for example Composer and Packagist, they propose the service Private Packagist to manage private packages, but it is still possible to use VCS Repositories, private servers Satis or Packagist, "inlines" packages, Artifacts, Path directories, etc. ... to manage private dependencies. Same for NPM with NPM Registry, they offer to manage private packages with a paid subscription, but you can still use Git/Github/etc links in the dependencies. In both cases, we do not have to use their servers if we wish. Consequently, we subscribe to their services by choice (simplicity, speed, security, serenity, etc ...) and not because we are constrained technically.

I think that Symfony Flex should offer the same principle, and provide a solution that does not impose their proprietary Symfony.sh service. Of course, I understand and I encourage the promotion of this service, because it is important for SensioLabs SAS to diversify its income, allowing in return, to provide open source and proprietary quality tools (and everything that goes around). For example, SensioLabs' Blackfire is a great proprietary SaaS tool that is priced right. But concerning Flex and Symfony.sh, impose the service and propose it at a price of 249 EUR / mounth (297 USD), it's simply incomprehensible, especially since the services of the same type are rather between 7 and 16 USD per month.

@nicolas-grekas To answer your sentence

You can create your own plugin if you want to propose an alternative way to ship default configuration. I wish you all the best to justify the time you're going to spend on it for private projects.

For a service at 297 USD / month (in case it would be the final price, and not counting the 25 USD / month for Github if I use another service), I quickly justify the development of a plugin to work with private repositories. And I did not wait to realize it: Less than 4 hours on my free time to make a Composer plugin to use inline recipes, to define a package in require-dev as repository of recipes (like symfony/recipes), and allow to other plugins to add custom configurators or new type of recipe repositories. As I already told you, it's stupid to reinvent the wheel, because Flex is a very good base to build custom tools, and that's why I opted for the hack solution, even if it's not the one I prefer. The only disadvantage is that I have to use the reflection on 4 properties (and therefore subject to your changes), because you refuse to give the possibility to a Composer plugin to extend the functionalities of Flex (configurator and recipe repository).

So I do with, because in the end, my profitability is much higher, but I will prefer to see this capability natively integrated in Flex. It goes without saying that I am always open to making changes to my PR if you want to offer the capability to extend Flex for other plugins. Of course, I know that the Flex's extensibility is another issue, but it is closely related to this one.

@fabpot, @nicolas-grekas, the Symfony Core' team, please, reconsider the orientation of Symfony Flex.

Thank you

[slootjes]

@francoispluchino did you consider the flex server I mentioned before? I think it does exactly what you want I guess.

[fabpot]

I will just mention one thing: there is no relationships between this issue and the paid offering.

I've given the reasons why I'm opposed to this proposal. I still stand by them. I don't see why I would need to change the orientation of Flex. I have good reasons to think that the direction you want to take is not the right one.

Now, discussions are good. Discussions are open. I'm demonstrated more than once that people can convince me to change my initial plans.

But quite frankly, I don't even understand why one would think that I would consciously limit a tool just to be able to propose a paid offering. 14 years of free work for the community. Think about this. Anyway, have a good day. I'm done for now. I will go out, take a break.

[tseho]

@fabpot Sorry I joined the conversation with an unrelated pricing. I was surprised by the amount since pricing of SensioLabs tools, like Blackfire, have always been fair. It was not an attack, I have a lot of respect for you and your work. I'm using tools you created and contributed to everyday.

I have been following the progress of Flex since you announced it and I have great hopes for it. Without inline recipes, which are against your vision, I understand it's not a tool for me (and for a few others commenting here), and that's fine.

Flex will still be a welcome tool for open-source projects and curated recipes.

[francoispluchino]

@fabpot We misunderstood each other, so I will try to add a clearer explanation to my comments.

When I talk about orientation, I'm talking about features, I'm talking about giving the Symfony community the opportunity to do more, by bringing their little stone to this edifice that you've managed to wear and do grow over the years. I appreciate and respect your work a lot, so please do not make me say what I did not say :-).

So when I talk about orientation for Symfony Flex, I'm not talking about forcing people to use a paid service, but the orientation to only use recipes from Symfony.sh, and so, consequently, to limit the recipes for the open source packages or being forced to use Symfony.sh blocked on Github for private recipes (which obliges all those who do not use Github for private repositories to subscribe in addition for this service).

When I say confirmed intention, however, I'm talking about the intention to do pay for the Flex Server service after the Beta-test phase. Confirmed because it is an information on a official page of this service. And as I said, it's a perfectly normal logic that does not bother me (we can always discuss the price of the service, but that's another subject). I indicate that I understand better the reluctance of the team because precisely, this point is not negligible and for a company, it is justifiable to find an economic model or new services especially when there is like you, a consequent work for an open source project requiring more and more resources. I also do not think it's wise to focus solely on the community (but yes, we must not forget him). So this is not a criticism as you think, on the contrary, but a statement of choice should not be neglected in the development of a tool, whether open source or proprietary. On the other hand, what bothers me is to leave only the ability to set the recipes on Symfony.sh, which is a paradox for a tool to make Symfony Framework more flexible.

Of course, I include in my comment the capabilities to extend Flex, because if it were possible, the question would not arise, and those wishing to use this method or another, could do it. It would not be harmful for Symfony Flex if some used this method, because it would concern them, and only them, but not other users of Symfony Flex.

To return to inline recipes, your 3 arguments are interesting but questionable, and as you wish, I will take them resume:

Many features I want to add need a central repository. But even today, that's a must.

This is not an argument, but as I already said, I excited to discover your new ideas!

Even a basic feature like removing a package needs more than a local recipe as the recipe might have evolved between the time you installed it and the time you are removing it.

I gladly accept your argument, even if I do not consider it imperative to have an updated recipe to be a necessity, because the recipe has to work with the package version. If it is not the case, it is a bug and the recipe must be fixed. Besides, a recipe can copy files from the installed package. And of course, the file is not overwritten when updating a new version (normal, it's for the config). So if this file is buggy, in the end, it is the recipe that is buggy. It can be updated in the package or on a server. I'm sorry, but it's the same problem, the end user must do the correction by hand. And for those who install the package with an updated recipe, the file is still in the package, if it is not as updated, the problem will still be present. So a new version of the package must be published.

So in this logic, a configuration should not move for a given version, but indeed, the configuration could move in the case where the configuration of the project must evolve (like Symfony 2.x to 3.0, and 3.x to 4.0). Moreover, on this subject, In this case, how will we do with the recipes ? You create a new remote repository, and you update Flex to be compatible by automatically detecting the version of the Symfony project to automatically select the good repositories? Otherwise, a simple variable in the composer.json file may be much simpler, or something else, because I did not look at it in more detail, but of course, we are not there.

Moreover, the fact that we curate the recipe and validate them is an important part of the process and the value we want to propose. I don't want packages to be able to do anything they want.

As I've already explained several times, I understand this argument, and you've added precisions for the recipes-contrib, but for inline recipes for private packages, it's useless. I specify it, because we never know, I never said that the server mode was useless and it was necessary to use only inline recipes (see below).

Last, but not least, the recipes repository is not just for Symfony bundles, but for any Composer packages (like PHPUnit, Twig or Swiftmailer). For these packages, I doubt that their maintainers will want to add some specific Symfony recipes.

I always agree with you for open-source packages, and a server is therefore mandatory and relevant to achieve this functionality for all existing packages, and should not be directly associated with tools such as Flex. However, I do not agree with you about private packages for one or more private projects, because if we create independent private packages, it is for reuse on the same type of project 99% of the time, because otherwise the code is placed in the project's App namespace (and it's not disturbing to have a useless recipe folder for other private projects that do not use Flex - personally, in PHP I do not have any).

I do not think it is relevant to be limited to one distribution channel for recipes. For me, Symfony.sh must be the official remote repository of recipes, and it would be good to propose several types of repositories for recipes, just as we can use different types of repositories for Composer, knowing that the official Packagist.org repository is enabled by default.

As I said in this comment, I think that it would be very useful to propose a flexible approach on repositories, by defining Symfony.sh as official remote repository, and to allow to add packages like recipe repositories, but also of inline recipes directly in the package of the library, and of course allow to define a custom private server.

You do not agree with me on this point, and I can understand it and in any case accept it, but in this case, can you please, reconsider the possibility to the third party plugins, to extend the functionality of Flex, as we can extend the functionalities of Composer?

Outside the argument of @nicolas-grekas regarding the cost of maintenance to keep a stable public API, I do not see why the concept of this idea is rejected. This concerns issue #173, but it flows from the issue of the private repositories and this issue, even if it goes much further in the value proposition. On this subject, I invite you to re-read the issue #70 and mainly my proposals in this comment, especially this one, and this last.

Thank you again, and I hope I have convinced you that my intentions were good.

[apfelbox]

I just want to also add my two cents:

we are an agency and therefore are quite often bootstrapping new projects. In general I like the direction in which Symfony is going with flex, but there are also quite some issues:

1. Obviously the issue discussed here, private recipes without exposing anything (that includes public recipes, as the config files can be leaks, too).
2. If you want to skip using Flex, Symfony 4 has gotten quite a bit more uncomfortable in comparison to symfony/standard-edition, as my config is by default just empty. So for every new component I install, I always have to hunt down the documentation or go to a third place (beside the code / packagist repository and my IDE) which is symfony/recipe and manually copy all the files from there.

So in general using Sf4 without Flex is a clear DX downgrade in comparison to using Sf3 with the standard edition. So you can say "well, just don't use flex", but that also implies "please use our more complicated product" which goes against the main idea behind Flex, as I recall.

I have the feeling that while providing a more opinionated framework we also have lost the possibility to configure some things or opt out from them. This also includes things like being forced to use env files instead of indulgently falling back to "just working"™ (see https://github.com/symfony/recipes/pull/247#issuecomment-356599423 for example).

So while in general I love the way Sf(4) is going, I just have the feeling that we are currently stomping on the valid use cases of several users (myself included).

Also, asking contributors of a free plugin to develop and maintain a set of features for private needs is an economic nonsense.

I am 10000% sure, that here are at least 10+ people willing to implement this feature as soon it is green lighted here.

[nicolas-grekas]

@apfelbox maybe you missed composer create-project symfony/website-skeleton? See http://symfony.com/blog/the-end-of-the-symfony-standard-edition That's how you can bootstrap something similar to the standard edition. THEN, if after that you don't want to use Flex, just remove it. And you'll basically get back to pre-Flex state. So actually your post doesn't make really sense to me: you lost nothing in the process, quite the contrary.

that here are at least 10+ people willing to implement this feature

That's only 20% of the sentence: you missed the "and maintain" part. How many of them would be willing to make their own Flex-like alternative? That's basically the question.

[apfelbox]

@nicolas-grekas

maybe you missed composer create-project symfony/website-skeleton? [...]

You are right, that is indeed still possible.

I was rather referring to the "best of both worlds": having a minimal project and the comfort that is provided by Flex as well as not being "locked in" to certain restrictions that are currently still in place by Flex. (which are mainly by design, though)

The step away from a huge config file where 90% of the lines are commented out was a good one. But the only way (?) to have this managable is by using Flex – which imposes its own restrictions.

you missed the "and maintain" part

That is indeed the question. But that is a discussion without actual result: some say "there is nobody", some say "yeah, there is/will be somebody" – so not very goal oriented.

Especially as it seems that there might even be an alternative to this whole topic by just allowing Flex itself to be extended in certain key places, as @francoispluchino proposed. So that the support for in-package manifests can be added via a plugin.

[francoispluchino]

@apfelbox As @nicolas-grekas says, there is a skeleton to get something similar to Symfony Standard Edition with symfony/website-skeleton.

However you have a very cool novelty allowing unpack a pack, and therefore simply remove parts that you do not want to use. @fabpot explained it very well in this post.

With this vision, we can easily replace SSE with Flex. So I would not say it's DX downgrade, it's even the opposite, but it's frustrating in its limitations. To explain myself, I will take your example with symfony/dotenv.

With SSE, you clone and modify SSE to create your own project template to quickly bootstrap a new project. You have lots of code, config files, etc ... For your example, you modify the composer.json file, and the index.php and bin/console files to remove the use of this component. Personally I always hated this method, because you had to manually modify all the changes for the template, like for example changes in the files app.php, app_dev.php, etc ... if you wanted to stay proximate of best practices. It's tedious for a template, especially if it's small changes.

So now with Flex, the skeleton is only a composer.json file. Super simple to maintain (I can only say that this method is cool, because it is this principle that I put in place for more than 2 years with a self configuration of the components but with some differences), you remove your unnecessary dependencies. And for your example, you're actually going to have a problem with Flex:

Flex will copy the recipe for symfony/framework-bundle including the public/index.php file, as well as the recipe for symfony/console including the bin/console file.

What happens during the installation process with your new template skeleton, and more precisely when running the cache:clear? A beautiful exception:

APP_ENV environment variable is not defined. You need to define environment variables for configuration or add "symfony/dotenv" as a Composer dependency to load variables from a .env file.

So to fix it, you have to modify the 2 files, which is not problematic for the installation of a new project with the official template, but totally ineffective for a system allowing to design project templates using Flex flexibility. Because currently, to solve the problem, you end up redoing a pseudo SSE based on Flex including all modified files ... And if you have several different templates, well, you multiply the number of modifications by the number of project templates, like SSE. In this case, we can say that it is a DX downgrade against Flex, since we lose all its power and its updated recipes.

So sorry, I'm going to come back to my work and my vision again, because I recently migrated my tool to merge it with Flex because there is a very strong similarity in the methodology as mentioned above:

• unique composer.json file in template
• Composer plugin to retrieve manifests (recipes) for each installator (configurator)
• manifest (recipe) placed either in a package acting as repositories for several manifests (recipes), or directly in the package of the module (a module is a Symfony bundle specific to my platform)
• a manifest containing the files to be copied, as well as a JSON file indicating the actions to be performed for each installator (configurator): adding the bundle in the kernel (listed in a json file and not a php file because it is technically easier to do, but alas, less quick), adding routes, data loader, adding assets for webpack, creating the doctrine entities (metadata and classes), validation config, serializer config, etc...

So using Flex, I deleted some configurators, I have a lot less code to maintain, I can use the power of Flex, while adding my features (custom configurators, inline recipes and package repositories for recipes, etc). You understand that I hacked Flex because it is impossible to extend it for the moment. It works fine until Flex changes without warning its internal APIs, which is normal since there is no system to extend it. However, I consider that the game is worth it because I hate to use tools that do 90% of the same work with a similar approach, and that I deeply believe that Flex can reach a lot more people than if I propose my own plugin to replace Flex. Maybe someday you'll understand why it's so important for me to extend Flex.

So, regarding the problem mentioned above in my message: adding the ability to extend Flex, we could create plugins for Flex to:

• manage recipes inlines, but also recipes in other repositories that Symfony.sh (servers, package repositories)
• add custom configurators
• select custom recipes instead of standard recipes
• remove the operations of Flex for the private packages, thus avoiding "leaks" of information
• ability to create completely customized project templates while retaining the power and flexibility of Flex

I do not think that these features should be natively integrated with Flex, but I think that these features can be useful for many people, but not all. Therefore it would be preferable to outsource them with plugins. To achieve all that I have indicated, the PR #174 needs to be updated.

@fabpot, @nicolas-grekas My additional examples are still not enough for you to accept my value proposition?

Given that the discussion is rather about the Flex extensibility and not just about the inline recipes, I suggest continuing on the issue #173 and closing this issue. What do you think?

PS. I know, my comments are really too long right now :-)

[zorang]

The discussion is big, I have a simple question:

we have our company private bundles git repositories on our private servers. We want to integrate that bundles in our Symfony private application on Flex way. Is there any free way todo that?!

[moay]

@zorang We are currently implementing our own solution for this. It is self hosted and allows private recipes from a private repo behind a firewall. For the private bundles themselves, you should use satis, toran or private packagist. See here: https://github.com/moay/symfony-flex-server

I am still working on this, give a week and it should be ready to be used. I'll release a beta. Currently, the flex functionality works, but it still needs some tweaking, some tests and a proper documentation.

EDIT: This answers @zorang's question, but I think, it does not actually touch this issue.

[slootjes]

Looks good, we are using https://github.com/aurimasniekis/flex-server and also works very well :)

[moay]

@slootjes Yes, we do too for now, but the project neither well coded, nor supported nor compliant to the current version of flex. F.i. it does not support symfony.lock properly. That's why we created our own solution.

[slootjes]

Good to know, I've starred the repo 👍

[route1rodent]

I think it's time Symfony core developers listen to us.

I still don't understand why the need of centralise this tool instead of giving freedom to the developers to decide. I agree that for official packages it makes sense, but not for other packages. In some business they don't even use Packagist, they use a self hosted Satis with a self hosted git server.

It makes no sense for me to prevent developers to use the flex tool for their own private packages, and it doesn't make sense having to maintain the recipes in a separated repository. Unnecessary burocracy.

It's the responsibility of the end user/developer to decide what to include in his/her project and to verify that it works and it is secure.

For private packages seems that the only solution is to create our own composer-plugin that is able to work with our bundles without the need of a centralised recipes...

[zanbaldwin]

There is definitely merit in having central repositories of recipes is to ensure quality (reliability, validity, and "officiality" if that's even a word) at least to some degree.

However I agree that "third-party" (not listed in a core or contrib repositories) recipes need to be implemented, because currently recipes cannot be implemented for closed-source/private packages. It would greatly improve the experience for digital agencies working with Symfony on multiple projects.

If "third-party" recipes are implemented somehow, it should be to allow flexibility in the cases where it's absolutely needed without sacrificing the benefits of centralized repositories.

Proposal

I'm not a part of the Flex project, so I don't know off the top of my head how complex something like the following would be - I offer it purely as an idea, not hey-please-implement-this-okay-thanks.

Currently, the extra.symfony.allow-contrib is added to composer.json to allow recipes from the contrib repository. One idea is to add another piece of extra information called allow-untrusted to allow third-party recipes where the manifest data is included in the package's composer.json (or manifest.json file included in the package source code). Using an array (with no automatic help from the CLI) to explicitly whitelist packages would be a good way to:

• Continue the trust in the centralised core and contrib repositories, by not allowing untrusted packages that are dependencies of whitelisted packages.
• Be enough of an inconvenience to prevent forming bad habits for the majority of developers.

{
    "extra": {
        "symfony": {
            "allow-contrib": true,
            "allow-untrusted": [
                "vendor/package-name",
                "vendor/package-name2"
            ]
        }
    }
}

Obviously this comes with its own down-sides, just as having to host your own private recipes repository has its downsides. I'm not the one to make this kind of decision, but considering how often this issue keeps cropping up it would be nice to see a solution to this problem (whatever that may be) in the future.

[drupol]

I wish I could have such Composer plugin that would read a recipe from inside the package and then play it. Just like Linux distributions are doing with RPM, APT and DEB packages.

[drupol]

Just out of curiosity I did a PR where I added support for inline recipe, only ~20 lines of code is required for this to work.

How it works ? easy. If no official recipe exist for the package and if the file manifest.json exist in the package, then it will be used as recipe.

This proof of concept is still a work in progress but adding support for local recipes should be definitely quite easy to do and I might dedicate more time if I see some interest.

[nicolas-grekas]

This is not something that is going to happen. Call it opinionated - it is. That aligns to what recipes are from our pov: opinionated ways of using some libraries. i.e., not something that belongs to the same domain as the code - but that belongs to packaging. See also https://github.com/symfony/flex/pull/572#issuecomment-555903491

Thank you for submitting the discussion anyway.