Best practices for modules including PHP libraries?

[jazzdrive3]

Would using composer within a contrib project be acceptable, for instance? Not to make it required for user to run composer, just for internal dependency management. That means the vendor folder would be checked in and part of what is downloaded.

This is mainly for larger contrib projects, of course.

Or would it make more sense to manually include the dependencies, and integrate with Backdrops built in registry?

This assumes that these libraries all have the appropriate license, and can be included in the repo.

[quicksketch]

Hi @jazzdrive3! A good question. At this point we don't even have a port of Libraries API, as for JS-driven modules (e.g. a Flexslider module or similar), we're encouraging modules to bundle the library.

That may not be appropriate for PHP libraries (though it depends on the library). The problem space is also fuzzy because Libraries API module isn't really suitable for use with Composer anyway, or at least I'm not familiar with it being used with Composer.

We have had a very long discussion on libraries and an official location over in https://github.com/backdrop/backdrop-issues/issues/159. In particular see this comment) that outlines a general plan. However, I'm not sure how that would fit with composer. Would using composer within a root libraries directory do the job?

It may be best to join that discussion rather than starting a separate thread here.

[Graham-72]

At this point we don't even have a port of Libraries API

Is https://github.com/backdrop-contrib/libraries of no help?

[kreynen]

Not sure if this is an appropriate place to start a should Backdrop move to semver in contrib to support Composer/Packagist discussion, but it seems related enough... so here it is.

Bundling is certainly a short term answer and was standard practice on GitHub until Composer and Packagist. Git submodules are another option, but these only make sense if the code lives in same directory as the project.

While Packagist started as a PHP Package Repository, it is now the place to go for all things. The repository locations, versions, potential forks, licensing for Fonts, WYSIWYG, and hundreds of js libraries can all be found there.

There are obvious downsides to committing a copy of another project's code into your project's repo. Packagist has already answered many of the questions about maintaining dependencies in truly distributed projects.

The discussion about Composer use in Drupal's contrib has moved to https://www.drupal.org/node/2551607. While any project can add a composer.json, that really doesn't get you very far if everything required for the project isn't available.

I've come around to accepting that Packagist.org as another acceptable single point of failure. Like GitHub, if Packagist was down it would be impacting the entire FOSS community. Not just Drupal or Backdrop.

The arguments against embracing Composer more directly in Backdrop would like be very similar to the arguments @davidbhayes made in http://wptavern.com/i-love-composer-i-love-wordpress-but-i-object-to-a-marriage, but I think these can all be addressed. One solution we've been talking about in the Drupal community for some time is a http://jqueryui.com/download/ style "Make My Drupal" site. This would essentially have been a web version of https://www.drupal.org/project/profiler_builder that used a form vs. the site itself to generate a .make and combine everything the user wanted into a single download. Basically packaging that bypasses the Drupal.org Packaging Whitelist. The challenge is that while the profile builder project includes potential locations of some libraries, there was no way for a Drupal project to indicate that it required a specific fork or patch of a library.

Packagist addresses that.

The big issue for D8's contrib and Backdrop to fully embrace Packagist is the need for semantic versioning. Because Backdrop doesn't rely on Drupal.org's packaging scripts, this may be much easier to achieve in Backdrop. Moving to semver probably doesn't meet the criteria for a 1.x feature, but I'd suggest following how this is playing out in D8 and consider this change in 2.x.

[quicksketch]

Because Backdrop doesn't rely on Drupal.org's packaging scripts, this may be much easier to achieve in Backdrop. Moving to semver probably doesn't meet the criteria for a 1.x feature, but I'd suggest following how this is playing out in D8 and consider this change in 2.x.

If we decided to go to semver for contrib, this would indeed be much, much easier for Backdrop. Not because of our existing tools, but because we currently only have one version of Backdrop to support. So if we decided to adopt semver, we would simply ignore "1.x-" prefixes as if they weren't there at all. Modules compatible with Backdrop 2.x in the future could simply require a Backdrop version greater than 2.0.0 to indicate their support level. The real stickiness is getting over functionally-equivalent versions that are compatible with two major releases of Backdrop.

However, the technical packaging support for semver is much less complicated than the implications around how Contrib needs to function. I'll be watching the Drupal issue for that now that I know it exists: https://www.drupal.org/node/1612910. The Drupal folks are also looking at release channels, which may be supported by semver eventually as a solution.

Anyway, back to @jazzdrive3's question, now that I've read through all these related issues. We might consider porting https://www.drupal.org/project/composer_manager to Backdrop for modules interested in maintaining dependencies via Composer. I'd like to take a look at that module and see how it manages its libraries, and if it would be compatible with our existing ideas for core library support from #159.

[jazzdrive3]

Composer Manager works great for something like Drupal 8, which seems to assume a certain level of aptitude, but I'm not sure it would fit with Backdrop without a drastically different approach.

As it is right now, if a module requires Composer Manager, it also requires Composer on the server (of course), and then it requires at least two drush commands be run. If we could build a UI around the process, or automate it....but then is a Composer UI really something that the project wants to take on?

If Backdrop wants to leave it to the realm of average end users to be able to upload and install modules, I'm not sure something like Composer Manager would be a great idea. This was the main reason I wanted to bring up the question, as it seems just using Composer internally for each contrib, and then checking in the vendor folder for each repo would mitigate this, but it leaves the potential for a lot of bloat and duplication.

An alternative would be something like mentioned in #159, perhaps with each library having its own module that can then be defined as a dependency. But then that leaves a lot of extra modules.

[quicksketch]

Thanks @jazzdrive3 for the thoughtful input. After reviewing Composer Manager, I think you may be right. Although the UI for Composer Manager looks quite nice, there does seem to be a heavy command-line component still. I hadn't realized that it just parses composer.json files from each module to make one "master" composer.json file, on which the composer command must be run. That indeed sounds a bit more complicated than I would desire for developer experience.

On the other hand, if we adopted a basic Libraries API-like approach to external libraries, perhaps you could use Composer to manage your "libraries" directory if desired. Or if that's over-the-top, just downloading the libraries into the right directory could do the job (as Drupal users are doing currently). The only trouble is that PHP libraries are becoming increasingly granular, managing them manually may soon become too complicated to do without a dedicated tool.

[deviantintegral]

Composer manager does the whole merging process because composer only supports a single composer.json and vendor directory. Otherwise you could have a situation where multiple contrib modules require the same library, and you end up with duplicate class errors. It doesn't handle conflicting dependencies well (last version according to module weights wins), so you can still have issues if modules require different major versions of the same library.

There's the Composer Merge Plugin which can probably replace much of what composer manager does now, but it's not a UI.

As far as a UI, it's basically all of the same problems that led to authorize.php. Totally possible to do, but no one has done it yet. And for webhosts where you can't use authorize.php and need to ftp / rsync files, you would need some sort of external web UI or local UI to build /vendors and composer.lock.

Perhaps shipping a .command file which would work on OS X, and a .bat file which works with XAMPP would be close enough?

[quicksketch]

Hi @deviantintegral! Thanks for joining in here. :)

Perhaps shipping a .command file which would work on OS X, and a .bat file which works with XAMPP would be close enough?

This is sort of the root of the whole problem. We don't want a command-line component at all. Or if anything, it should be supplemental, not required. We're working towards searchable/installable modules directly from the UI in #870. But in truth that's only the "browser" component. The installation ability for modules has always existed for the entire D7 cycle. But the additional complexity of libraries or dependencies handled through the UI has not yet been solved.

I suppose you're right that we could use authorize.php in the same way to jump through all the hoops of building the composer.json file and then executing composer to download the dependencies. But then we've got a system complicated enough that installing a module not through the UI is much more complicated.

[LionsAd]

@quicksketch Note: You don't need to run the composer commandline runtime - composer has a perfectly suitable PHP library and as long as you use the tarball functionality it should also perfectly work on shared hosting.

[jenlampton]

Based on how the composer workflow is evolving in Drupal 8, I think it would be better to stick to our plan of including libraries directly in the modules that need them for now. Only once it's been determined how to create both a good user and developer experience at the same time, should we consider moving to a composer based workflow for Backdrop.

Until then, I'd love to see hook_library_info() adapted so that PHP libraries can be added, much like the Libraries API for Drupal 7. See https://github.com/backdrop/backdrop-issues/issues/2911

[jlfranklin]

Ugh. IMHO, one of the benefits of Backdrop over Drupal 8 was the lack of Composer.

[jenlampton]

@jlfranklin that's likely because you are an advanced Backdrop/Drupal user. I've talked to countless people who won't move to Drupal 8 because of the composer requirement alone. For people who don't use the command line it's a huge deterrent.