Support using Backdrop CMS with composer

[itsdarrylnorris]

Have BackdropCMS consider making core and contrib modules available in https://packagist.org/? I guess the answer is yes - https://github.com/backdrop-ops/backdropcms.org/issues/8#issuecomment-24637054

But there are any issues/reasons why Backdrop is not composer base ?

[mikemccaffrey]

Hey @darol100, the move to composer was one of the reasons that we forked Backdrop, since it seemed like an unessesary complexity that was not friendly to novice developers. Instead, we have been working to improve the built-in package manager in core, and have even gotten it to the point where you can install new modules through the site ui!

I suppose that we could also make backdrop and it's modules available for download via composer, but I'm not sure how many people that would actually help.

[Kazanir]

I think it should be possible to enable the use of Composer without requiring it for novice developers. The big advantage of doing this is that there would be a way to manage a Backdrop site in version control without committing the entirety of either Backdrop Core or all contributed modules to the repository for a given site. This also allows for really easy management of custom patches to either core or contrib modules, similar to a Drush makefile.

I prototyped this using Satis but based on https://github.com/backdrop-ops/backdropcms.org/issues/8 it looks like this should be doable using the backdropcms.org project release infrastructure. Here's the basic steps:

• [ ] Port some version of https://www.drupal.org/project/project_composer to the backdropcms.org release infrastructure. This module provides faked composer.json metadata based on module .info files, lists the packages with a type of backdrop-module or backdrop-theme, and also converts Backdrop-style version numbers (i.e. 1.x-1.2) into Composer-style version numbers (1.2) by stripping the core version.
• [ ] Expose a Composer repository endpoint (such as https://packages.backdropcms.org/1) which serves up this Composer metadata. (Like I said above, I prototyped a version of this which used Satis and the Github repos directly.)
• [ ] Provide a sample project which will enable Composer users to use composer create-project backdrop. This sample project contains the metadata that defines the location of e.g. a backdrop-module as well as allowing people to define their own patches and so on. Once the infrastructure is in place, most of https://github.com/drupal-composer/drupal-project/blob/8.x/composer.json should be quite usable as a template.

[serundeputy]

Hey @darol100, the move to composer was one of the reasons that we forked Backdrop, since it seemed like an unessesary complexity that was not friendly to novice developers.

This is the first time I've ever heard this cited as a reason for the fork. Symfony yes, but not Composer.

That is not to say Symfony is bad (it is good), but it changed alot of APIs really quickly.

That being said I'm for a composer initiative that allows for modern PHP module, package and dependency management without requiring it for the novice user. I think in that regard this is similar to D8 you are not required to use composer to manage D8, but it is there if you want it.

~Geoff

[klonos]

Seems like a reasonable request to me if to be implemented in an opt-in way.

[olafgrabienski]

Composer as an option might be helpful, but we should really keep an eye to not introduce even 'soft' requirements, if only because on some shared hostings there is no composer nor you can't install it by yourself.

To use D8, Composer may not be a formal requirement but in my opinion the situation tends to become ambiguous.

(1) Some important modules require Composer, see https://www.drupal.org/project/address as an example:

Address must be installed via Composer, in order to get the required libraries.

Unfortunately not 'helpful' to this effect (https://stackoverflow.com/questions/39264789/how-to-install-libraries-without-composer-in-drupal-8):

Composer is an important part of the modern PHP ecosystem. It brings considerable benefits. I recommend asking about how to get it to work (it's not hard) instead of fighting against it.

(2) The situation regarding Drupal core and Composer isn't very clear nor well documented. There are a lot of enthusiastic blog posts and tutorials about how to use D8 with Composer but if you look for help if/how to install and/or manage D8 without Composer, it becomes difficult. As an example, see the following issue:

• Proper way to install Drupal, missing vendor folders, example.gitignore, https://www.drupal.org/node/2737773

[generalredneck]

I think this should be accomplishable much the same way drupal 7 composer builds started out. It's by no means necessary to build a site in D7 using composer but totally achievable using https://github.com/drupal-composer/drupal-project/tree/7.x?files=1 Before D.O took on hosting the packagist (and before D8 was even using composer for module installation) webflo, will milton and the team at Promet Source were doing it. I feel like since contrib is under one github org this should make this easier and the packagist would run pretty freaking fast. Would also require a port of the composer_autoload module.

[klonos]

@drupol has very recently done some work in https://github.com/drupol/backdrop-docker and https://github.com/drupol/backdrop-conventions if you wanna take a look @generalredneck

[drupol]

We had a discussion about it yesterday in the weekly technical meeting.

@quicksketch @jenlampton Can we use this issue or should I create a new one ?

[drupol]

An example of composer.json would be:

{
    "name": "backdrop/backdrop",
    "description": "The free and Open Source CMS that helps you build websites for businesses and non-profits.",
    "license": "GPL-2.0",
    "homepage": "https://backdropcms.org/",
    "support": {
        "docs": "https://api.backdropcms.org",
        "issues": "https://github.com/backdrop/backdrop-issues",
        "source": "https://github.com/backdrop/backdrop"
    },
    "require": {
        "php": ">= 5.3.2"
    }
}

[serundeputy]

@drupol Can the docs key be an array?

"docs": [
  "https://backdropcms.org/user-guide",
  "https://api.backdropcms.org"
]

[drupol]

@drupol Can the docs key be an array?

"docs": [
  "https://backdropcms.org/user-guide",
  "https://api.backdropcms.org"
]

The docs is supposed to be a string and not an array... so no.

Only one url is allowed.

Reference: https://getcomposer.org/schema.json

[serundeputy]

thanks; i'd vote for the api URL then.

[drupol]

I updated the snippet.

[klonos]

I am all for this to happen 👍

I've been watching the recent weekly dev meeting video, and I also only see added value for including a composer.json file and making Backdrop core in packagist; no downsides.

Having said that, I need to stress that we should make sure that we never actually introduce any dependencies for Backdrop on composer; and I believe that we all agree on that; just wanted to make it clear.

Finally, I don't see any reason to wait for 1.14 for this to happen. The addition of this file does not affect the API or any portion of code in Backdrop, so I see it as an as simple task as adding a CHANGELOG.txt or README file; it can happen anytime in the 1.13.x release cycle.

[klonos]

@drupol care to file a PR for this?

[drupol]

I will :)

On Sat, Jun 15, 2019, 11:38 Gregory Netsas notifications@github.com wrote:

@drupol https://github.com/drupol care to file a PR for this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/backdrop/backdrop-issues/issues/2257?email_source=notifications&email_token=AAB5RCVYFCD6MQPRGLJXKVLP2S2ATA5CNFSM4CRMLM72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYUNWI#issuecomment-502351577, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB5RCRN45NMX2U7URJWTC3P2S2ATANCNFSM4CRMLM7Q .

[serundeputy]

I have a few ideas here.

1. There is no need to rush this.
2. We could/should start more conservatively.
   • I'll fork backdrop and add a composer.json to the fork and add that to packagist under the namespace backdrop/backdrop
   • this is how wordpress does it
3. I'd like to make the composer.json a little more useful than just getting backdrop on packagist (that is a good and helpful step, but I'd like to also tell composer where backdrop stores modules, themes, and layouts like https://github.com/drupal/drupal/blob/8.8.x/composer.json#L35
4. We could consider packaging a backdrop composer install w/ a site local drush that would prevent conflicts w/ people using drush for backdrop and drupal simultaneously (it seems to me that a person using composer to manage a project will likely want drush as well.).

[serundeputy]

@quicksketch seems I can not have two forks of backdrop under the @serundeputy namespace. Can i put a backdrop/backdrop fork in /backdrop-ops/backdrop-composer and then i'll have to request packagist permissions to the backdrop-ops namespace which will technically have access to all the repos there; which I think is no danger, but if you'd rather have it somewhere else just let me know.

[jenlampton]

no downsides.

As mentioned at the very end of the discussion, there is the downside of perception. If people see a composer.json file in core, they might think they need to use composer with Backdrop. Composer is a developer tool, and Backdrop is not made for developers -- it's made for people. For most people who have experience using composer on other projects -- this perception is likely to turn them away.

I wonder if we could put it into the core directory... Does it need to be in the project root @drupol ?

I need to stress that we should make sure that we never actually introduce any dependencies for Backdrop on composer;

yes, 100%.

I don't see any reason to wait for 1.14 for this to happen.

You must have missed that part of the meeting :) The reason to wait till 1.14 was so that we could get more opinions from more people. It didn't have anything to do with the technical challenge of adding the file.

[findlabnet]

... make sure that we never actually introduce any dependencies for Backdrop on composer;

In other words: make sure that we never actually unlock this gun to shoot yourself in the foot.

[klonos]

...there is the downside of perception.

Hmm, I have not thought of that.

...so that we could get more opinions from more people.

Nope, did not miss that bit of the meeting; just thought that we won't need 2 months for that. But I may be wrong, and there is no harm to wait. I can see how waiting more could cause losing "momentum" from people that want to contribute towards this now (like @drupol), but I can see your point too, so issue label changed accordingly 😉

[drupol]

Hi,

@jenlampton It must be at the root, yes.

@klonos I don't mind waiting :-)

[klonos]

I don't mind waiting :-)

At least I hope that the waiting in Backdrop is considerably shorter and less frustrating for you than in D7 😅

[drupol]

I don't mind waiting :-)

At least I hope that the waiting in Backdrop is considerably shorter and less frustrating for you than in D7 sweat_smile

I will keep my mouth shut :-)

[serundeputy]

I've created https://packagist.org/packages/backdrop/backdrop the upstream is: https://github.com/backdrop-ops/backdrop-composer

This is just a basic backdrop installation that you can use via:

composer create-project -s dev backdrop/backdrop

right now this needs the -s dev part as the releases are out of sync, but once 1.13.3 comes out I think the releases will sync up.

Composer Infra

I would like to scaffold out some more composer infrastructure:

• backdrop/backdrop

  • this exists: https://packagist.org/packages/backdrop/backdrop
  • this is the basis for the other composer packages

• backdrop/core

  • a git subtree of just the core directory so one can manage there codebase as just the custom code and pull in backdrop as a build dependency
  • this is not done yet

• backdrop/backdrop-composer-nested

  • an opinionated composer build with a nested web root
  • includes drush proper
    • includes backdrop-drush-extension

I'm looking into ways to get the drush's to cooperate and end up in locations they need to be.

toodo: scrape backdrop-contrib and file PRs against repos with composer.json files and an explanation to the maintainer(s) as to why they might choose to include this PR, and of course it is up to them to accept or deny the PR.

[stpaultim]

there is the downside of perception. If people see a composer.json file in core, they might think they need to use composer with Backdrop.

I understand the importance of perception and appreciate that we're paying attention to it. I was initially skeptical that the mere presence of a composer.json file would be a problem. Sophisticated developers would understand that it's presence does not necessarily imply a requirement. Novice developers/site builders would probably not notice it at all and/or not have a good enough understanding of how composer works to worry about it. I see files in modules all the time that I don't understand and just ignore them.

However, to the point made by @jenlampton - I think there is a class of Drupal developer that has been tripped up and frustrated by the changes in Drupal 8 and the requirement of composer has been one of the most visible and frustrating changes for them. Composer is frequently called out by some of these developers as the most frustrating parts of Drupal 8 development for them. These developers are an important audience for Backdrop and I think it's important to be sensitive to their frustration.

I'd love it if we could find a way to provide a composer file in core in a way that does not confuse any of these folks. If it were possible to keep the file out of root, that might help (as suggested, but apparently not possible).

Would it be possible to include the composer.json file in a sub-directory and then have developers move it to the root directory during install - either manually or through some automated process? Could the file be named composer(optional).json (or something else) and then renamed during install when asked for.

I'm in favor of including the composer.json file. But, I'm looking for creative ways of addressing the issue of perception and making it VERY clear that this is purely optional and not scaring away anyone that was frustrated by composer in Drupal 8.

[drupol]

there is the downside of perception. If people see a composer.json file in core, they might think they need to use composer with Backdrop.

Until the moment they open the file and see that there is nothing in the require section ?

I understand the importance of perception and appreciate that we're paying attention to it. I was initially skeptical that the mere presence of a composer.json file would be a problem. Sophisticated developers would understand that it's presence does not necessarily imply a requirement. Novice developers/site builders would probably not notice it at all and/or not have a good enough understanding of how composer works to worry about it. I see files in modules all the time that I don't understand and just ignore them.

Then just do the same with composer.json ? :-)

You cannot imagine how benefic it will be to have it at the root of the project. It will be easier to install locally, in docker, everywhere. Another example, we could even ship a docker instance that will always be up to date thanks to the semver mechanism in the composer file (backdrop-ops/backdrop: ^1 and we are done)

However, to the point made by @jenlampton - I think there is a class of Drupal developer that has been tripped up and frustrated by the changes in Drupal 8 and the requirement of composer has been one of the most visible and frustrating changes for them. Composer is frequently called out by some of these developers as the most frustrating parts of Drupal 8 development for them. These developers are an important audience for Backdrop and I think it's important to be sensitive to their frustration.

I hear the arguments, but I'm also seeing it being used everywhere. That doesn't mean that we HAVE to use it. It's just that big successful projects are using it. Look at Symfony, Laravel, Bolt CMS, etc etc... Composer is a great tool when it's used like it's meant to be, on my side, I couldn't live without it.

How are you going to do when something in Backdrop will need something that is done perfectly by a library which is available on Packagist ? Are you going to copy paste it ? Have you ever think about this case in the future ?

Having a composer.json file in the project is the first step toward something great that needs some directions. If one day we need to use composer, it needs to be well defined and explained somewhere.

Let's not forget that the "class" of developers that are frustrated are also preventing themselves of evolving and moving forward. Deny it or not, Composer is part of PHP and part of the way we should use projects.

That said, as you may have understood, I hate avoiding a pattern just because someone might not understand it. It's up to us as developers to keep learning and if we don't understand something, it's a reason to learn it, not to avoid it.

You can take the following as a comparison... There are people out there who doesn't want to learn English because it's not their mother tongue... if they work in I.T., English is a requirement for being successful. Reading and understanding English is a requirement for using Backdrop. It's not written elsewhere, but it's "normal". Just like using Composer for PHP projects should be when we need it.

Imagine, we could have a custom Composer plugin that install Backdrop modules just like Symfony does with their "recipes" in symfony/flex ! The possibilities are endless.

I'd love it if we could find a way to provide a composer file in core in a way that does not confuse any of these folks. If it were possible to keep the file out of root, that might help (as suggested, but apparently not possible).

No please no... then there's no point having it in Backdrop then.

The point of having that file is to have it at the root of the project and linked in Packagist.

Would it be possible to include the composer.json file in a sub-directory and then have developers move it to the root directory during install - either manually or through some automated process? Could the file be named composer(optional).json (or something else) and then renamed during install when asked for.

I'm in favor of including the composer.json file. But, I'm looking for creative ways of addressing the issue of perception and making it VERY clear that this is purely optional and not scaring away anyone that was frustrated by composer in Drupal 8.

Does this "frustrated by composer in Dx" is still valid nowadays ?

5 years ago I can understand, Composer was not what it was, but nowadays, do you really think that developper are more afraid by Composer or by the uber complexity of doing the least small thing in Drupal ?

[serundeputy]

I resemble many of the remarks made by @drupol .

I'd love it if we could find a way to provide a composer file in core in a way that does not confuse any of these folks. If it were possible to keep the file out of root, that might help (as suggested, but apparently not possible).

• I'd say for now let's just leave the file out of core all together and leave it in backdrop-ops/backdrop-composer
  • this gets us on packagist (https://packagist.org/packages/backdrop/backdrop)
  • enables a mechanism for developers wanting to leverage composer
  • does not frighten the composer phobic among us

We can then sit back and observe how useful it is to our community and if it is 80% plus we can add it to core; if it is not we can just leave it in backdrop-ops/backdrop-composer ... no harm, no foul and as far as I can tell everyone can be happy that way.

[jenlampton]

I love these ideas proposed by @stpaultim 👍

Would it be possible to include the composer.json file in a sub-directory and then have developers move it to the root directory during install - either manually or through some automated process?

If we do this, I'd prefer that the move was manual, as I don't want the file to be actively in use unless someone specifically wanted it there. We could add something into our README.md that mentions the file exists, and where, and that it can be moved if necessary.

Could the file be named composer(optional).json (or something else) and then renamed during install when asked for.

ooh! We could even do something like optional.composer.json with a note in the README.md that says to rename this file if desired. I like this approach better than hiding the file in a deeper directory, as it would be more obvious to people who do want to use it. They wouldn't need to read the readme to find it.

No please no... then there's no point having it in Backdrop then. The point of having that file is to have it at the root of the project and linked in Packagist.

... I just had another idea. What if we had the composer.json file removed by the packaging script? (see https://github.com/backdrop-contrib/project/issues/37) That would make it available to developers or those who install directly form Git, but would remove it for anyone who downloads it as a zip from the web.

big successful projects are using it. Look at Symfony, Laravel, Bolt CMS, etc etc...

I'm not sure any of these projects are fair comparisons for Backdrop CMS. Symfony and Laravel are intended for developers, not for end-users. Do we know if there is a composer.json shipped with WordPress?

How are you going to do when something in Backdrop will need something that is done perfectly by a library which is available on Packagist ?

We'll use the same rules we've always used and include the library as part of Backdrop. This is what we already do with core (see Ckeditor, jQuery, etc), and what we do with contrib (hence, no need for the Libraries module anymore). This makes for a much better end-user experience than any alternative. If that ever changes, we can re-evaluate.

Let's not forget that the "class" of developers that are frustrated are also preventing themselves of evolving and moving forward.

Careful, this is your opinion. Let's not insult anyone. Backdrop's principals on this matter are clear: not chasing popular trends in technology, but instead adopting common, proven, and learnable systems. Composer is very "hot right now" but has a long way to go before it becomes proven, or even common.

Does this "frustrated by composer in Dx" is still valid nowadays ?

Yes, more than ever.

do you really think that developper are more afraid by Composer

It's not that people are afraid of composer, it's that people have been forced into using it, and now have the first-hand experience with it, sometimes of it going very badly. There are plenty experienced developers who have learned to work with it, but prefer that they didn't have to.

We can then sit back and observe how useful it is to our community and if it is 80% plus we can add it to core

How will we tell when we reach that mark?

[olafgrabienski]

We'll use the same rules we've always used and include the library as part of Backdrop. This is what we already do with core (see Ckeditor, jQuery, etc), and what we do with contrib (...). This makes for a much better end-user experience than any alternative.

As end-user (site builder) I love this rule! I'm really happy that I don't have to download libraries and ask me e.g. if a link to a library is still valid, if it still points to the proper library release etc. Based on the rule, I get the release that core or module developers consider as appropriate, no matter what kind it is (newest vs. older, original vs. adapted and so on).

In contrast, I imagine that for developers it might be attractive to provide a library by writing a release number rule in a composer file. For a project like Backdrop, which AFAIK doesn't rely very much on third party libraries, I hope however that the effort to choose and provide libraries without a tool like Composer, isn't perceived as needless work. It doesn't 'only' make life of end-users more easy. In my opinion, it also allows better quality control.

[klonos]

Could the file be named composer(optional).json (or something else) and then renamed during install when asked for.

ooh! We could even do something like optional.composer.json with a note in the README.md that says to rename this file if desired. ...

Yeah, I really like that idea 👍 ...it clearly conveys the optional bit, and at the same time is kept in the root directory (so people that want to use it have easy access to it).

... I just had another idea. What if we had the composer.json file removed by the packaging script? That would make it available to developers or those who install directly form Git, but would remove it for anyone who downloads it as a zip from the web.

Also like 👍

big successful projects are using it. Look at Symfony, Laravel, Bolt CMS, etc etc...

I'm not sure any of these projects are fair comparisons for Backdrop CMS. Symfony and Laravel are intended for developers, not for end-users. ...

I agree (although out of those specific examples, I cannot see how much dev-targeting or not Bolt CMS is).

...Do we know if there is a composer.json shipped with WordPress?

It does not. I just downloaded a fresh copy of the current 5.2.2 from https://en-au.wordpress.org/download and no composer file there. No composer files in https://github.com/WordPress/WordPress either; they keep them in https://github.com/WordPress/wordpress-develop instead.

Let's not forget that the "class" of developers that are frustrated are also preventing themselves of evolving and moving forward. ...I hate avoiding a pattern just because someone might not understand it. It's up to us as developers to keep learning and if we don't understand something, it's a reason to learn it, not to avoid it.

I am personally very sensitive about this @drupol. It is the "elite" developers that have pushed me away from contributing to d.org. Yes, I have learned how to use composer in the meantime, but it took me many years (because I was not doing Drupal/Backdrop professionally till only recently). I still recall very nasty (although very "politely" worded) comments in d.org, where it was implied that some were "smarter" than others. It has happened to me personally, and I have seen it happening to other, less tech-savvy people.

As @jenlampton said, we need to be very careful with this; our community has so far been free of such comments/remarks/insinuations. Non-coders have been welcomed to contribute what they can; novice developers encouraged to start tinkering with the code, and to seek assistance when they hit roadblocks. I personally am very passionate about making Backdrop as user-friendly as possible ...every small, hidden corner of it 😄

Does this "frustrated by composer in Dx" is still valid nowadays ?

I don't know, "someone" in the Gitter chat said:

I don't like Drupal 8, not because of the features, but because of all the things around. You need to have at least a PHD in physics to create a custom block.

😄 ...I know that I may be taking that out of context, because you are referring to composer specifically here, but composer seems to be one of "the things around".

Also quoting @frednwright from Gitter:

p.s. goodbye Drupal. You were good to me. I climbed that steep learning curve for you. Then you ran down the hill into the ditch. Hello Backdrop! I think this is the start of a beautiful friendship...

It is this kind of bad UX/DX that I am trying to help avoid in Backdrop. Adding composer to the mix adds complexity - it does not help ...unless one already speaks "composer" of course, in which case, I am sure that they will also be able to figure out that they can use https://github.com/backdrop-ops/backdrop-composer or rename optional.composer.json and move along.

We can then sit back and observe how useful it is to our community and if it is 80% plus we can add it to core

How will we tell when we reach that mark?

Once we have #285 in place, perhaps we can have it check the codebase for renamed optional.composer.json to composer.json(?).

[kreynen]

Late to this party, but please consider adding composer create-project -s dev backdrop/backdrop to the project description on https://packagist.org/packages/backdrop/backdrop or using a forked Readme.md. By default, Packagist suggests composer require backdrop/backdrop which isn't all that useful.

[klonos]

@serundeputy can you please transfer maintainership of https://packagist.org/packages/backdrop/backdrop to either @herbdool or @quicksketch?

If possible to add multiple people, then let's please add all @backdrop/core-committers so to have redundancy.

[klonos]

An relevant update here: we have this new contrib project now: https://github.com/backdrop-contrib/composer_manager

Composer Manager allows Backdrop contributed modules to depend on PHP libraries managed via Composer.

Also, a thread has been started in Zulip to further discuss this and get some feedback from the community: https://backdrop.zulipchat.com/#narrow/stream/218635-Backdrop/topic/Composer