Define aims and outstanding technical tasks for distro support and Features-style config packages in core

[nedjo]

Some interrelated issues with Drupal 7:

• On its own, a core Drupal install does practically nothing. Few if any sites run just the core.
• Rather than becoming more accessible, Drupal site development is increasingly costly. Typical site development costs range into the tens of thousands of dollars.
• Most custom development essentially repeats, with small variations, work that has been completed and paid for tens of thousands of times before.
• Alternatives to custom development like Drupal Gardens are limited, lacking a lot of the advantages of Drupal, and tied to particular corporate service offerings.
• With few exceptions, Drupal distributions seem to have low adoption rates.
• Despite early efforts like Kit and later ones like Debut and the interoperability aims in the Open App Standard in practice, distro developers went their own ways, developing features that worked only within the confines of their own distro. A big missed potential of Drupal 7 was interoperable features.

The closest that Drupal 7 came to interoperability was the Panopoly feature set, which has been adopted by several distributions including Open Atrium 2. But, while it's a valuable contribution, Panopoly comes at significant costs. Far from being slim and basic, the base install of Panopoly adds dozens of modules to Drupal core and pushes it well past what's available in typical shared hosting.

It would be great to address these problems from the start in Backdrop. Steps to do so could include:

• Define clear standards for interoperable feature development.
• Ship core with several workable features that:
  • provide out of the box usable website solutions,
  • can be the basis for other, more specialized features and distributions,
  • model the standards for interoperable features.

[quicksketch]

Hi @nedjo! Thanks for opening this issue. This is a largely "meta" conversation, so I'm not clear on what's actionable here. What do you mean by "clear standards for interoperable feature development"?

Ship core with several workable features that:

Oh, you mean like an "image gallery" feature? What things do you have in mind? Sorry for the questions, I'm trying to figure out what's being described here.

[nedjo]

What do you mean by "clear standards for interoperable feature development"?

I mean a set of defined patterns (naming conventions and so on) and in-core implementations that will ensure that Features-module style bundles of exported configuration are mutually compatible across sites and between distributions More below.

This is a largely "meta" conversation, so I'm not clear on what's actionable here.

What I'd like to work up here is a general direction that we can then dig into implementing. Here's a bit more background and detail.

Currently Drupal 7 ships with two install profiles ('standard' and 'minimal') that each does some initial configuration to turn Drupal the framework into a CMS. But these profiles are monolithic (I can't for example choose to use just the 'article' content type from 'standard') and fixed (I can't easily alter or extend any of the configuration).

In practice, Drupal 7 distros almost exclusively are built on a different basis--the Features module. But there is no established pattern for making features work nicely with each other. For some details, see my article Tips for Writing Interoperable Drupal Distributions (p. 124 - 135) and an earlier blog post on Apps and interoperability. Basically, it's a dog's breakfast out there. Even minor differences can produce needless but deep incompatibility between different features and, hence, between distros. See this inventory of role names and this list of conflicting components (different features trying to manage the same piece of configuration).

I'm hopeful that we can address these issues in Backdrop by doing things right in the first place. That means:

• Define some simple patterns (naming conventions, etc.) as to how to ensure exported configuration bundles ("features") are interoperable.
• As part of the CMI work, ensure we support in core what's needed for basic Features-like functionality. Not the Features module UI elements or the ability to generate features--just the ability to (a) package a bundle of configuration suited to a given use case and (b) override/customize configuration provided by another module. See this relevant wiki page on the CMI and Drupal distros.
• Include in core one or more "base" features that provide the sort of configuration that would be needed on most sites. This could include a small, standardized set of:
  • user roles, like 'administrator', 'editor', 'contributor'.
  • text formats, like 'full HTML'.
  • vocabularies, like 'tags'.
  • field base definitions (but not instances), like field_image or field_tags.
• Include in core several more features that meet the most commonly needed elements. These would require the base feature(s). Examples might include:
  • News: a 'news' content type and associated fields, permissions, etc.
  • Blog: a 'blog' content type and associated fields, permissions etc.
  • Event: an 'event' content type and associated fields, permissions etc. (hypothetically assuming that a 'date' field type is in core).
• Offer, at install time, different mixes of these features to meet different site types. A "Blog" site template/profile would have the blog feature, while a 'campaign' one might include 'news' and 'event'.

Of course, we would need to be careful not to:

• bloat core with excessive modules needed to support more complex types of features or
• overbuild core features.

Advantages:

• For the most basic sites, Backdrop is a usable CMS out of the box with several meaningful variations at install time.
• Distro developers have a clear model and can build directly off of what's in core, which itself adheres closely to the established feature patterns. The 'base' feature(s) in core provide built-in compatibility across distros. The core 'event' feature may provide just a content type and little more, but it can be extended in various directions. One distro might include an event calendar (using fullcalendar, for example), while another might include both base 'event' and 'event_calendar' as dependencies for 'event_registration'.

Is there support for this direction? Cautions? Pitfalls? Suggested changes?

This is an area of Backdrop planning and development I'm keen to coordinate and contribute to.

[sun]

FWIW, that was still a bit hard to digest, and I'm not sure whether I interpreted it correctly. Please correct me and clarify where I'm wrong. I'd like to clarify on a few aspects:

1. CMI actually resolves >90% of all of that. In the end, that was the exact architectural goal for CMI in the first place.
2. CMI transfers all configuration into flat files, which means that any module/theme/profile/distro can supply + install new configuration.
   However:
3. CMI has no protection or whatsoever against two (or more) extensions that are trying to set up the exact same configuration (e.g., a news.module setting up a "news" node type, and a "openpublish" profile/distro attempting to do the exact same, too).
4. CMI further has no concept of configuration "bundles" (like feature modules). All configuration is (reasonably) owned and controlled by the respective modules. For this reason, all configuration filenames are namespaced by module name; i.e., node.what.ever is owned and controlled by Node module.
   That said, "Feature modules" are still possible with CMI. The only difference is that they lose data authority the moment they are installed.
5. The fundamental concept of apps/features introduces a new authority to configuration that is tough to make sense of.
   1. An app/feature adds configuration, but does it own (and maintain) it?
   2. Is the module that actually owns the config allowed to modify it? (updates/upgrades...)
   3. And in case two apps/features attempt to introduce the exact same configuration, who wins?
6. In past CMI efforts, all of these aspects were discussed to some extent. Even with the improved architecture and clearly separated configuration and data authority, no one was able to come up with an architectural concept that would allow a "third-party" to not only bundle multiple config objects into a single unit (while retaining API simplicity), but additionally, to introduce a sane concept of "co-ownership" of data.

Again, just trying to provide some clarifications and backstory. This is a very complex topic and I'd love to brainstorm about potential solutions. :)

[nedjo]

@sun: Thanks for the background!

Yes, the technical issues here mostly come down to deciding who provides or owns particular pieces of configuration (an image field, for example). A derivative problem is avoiding conflicting dependencies, when two feature sets each provide a base element (like a role or field definition) that multiple other features will require. The non-technical issue is: how can we prevent parallel configuration that produces needless clutter or incompatibility.

For now in this ticket, beyond brainstorming, I think we can aim to:

• Flag outstanding technical problems and open tickets for each.
• Define aims for Backdrop--what do we aim to ship with in terms of install-time features and/or install profiles? Probably this part can be a pull request on the documentation repository.

I'll change the ticket title to reflect these aims.

[nedjo]

I've been musing about distros in Drupal and engaging with the Joomla development community. Relevant material:

• DrupalCon presentation by Jeff Eaton on install profiles and distros in Drupal 8.
• Blog post on configuration management and Features in Drupal 8.

Why do distros still form only a tiny fraction of Drupal sites, with only three Drupal 7 distros having install bases of more than 1,000 sites, and only one with 2,000 or more? By comparison, there are over 1,100 modules with install bases of more than 1,000 sites. In my case, a edge-case module I threw together mostly in an afternoon and haven't looked at in years has 32 times the install base of a distro I've spent thousands of hours on over the past three years. Ouch!

There are lots of reasons, but a key one I think we can focus on fixing in Backdrop:

• Distros in Drupal must be a starting point. They can't be installed as part of or after Drupal core. Instead, you must know about, download, and install the distro separately. You can't add a distro to an existing site--at least, not without a lot of custom work.

Here's what I'm thinking. When you install Backdrop, it looks something like Pantheon's installer. That is, you can install a bare-bones blank slate site (roughly equivalent to Drupal 7's "minimal"). Or, you can install one of a curated list of install profiles that together meet the most common site use cases.

On the back end, a small subset of these install profiles ship with core and use only core modules and themes. But (assuming internet access and file write access, similar to what's in the Apps module) the installer also connects to a service that provides a list of selected install profiles (distros). If the admin installing the site selects one of these contributed distros, the installer downloads and installs the relevant code, then proceeds with the install, including any steps added by the distro.

And, ideally, it would also be possible to install a distro after installing the site. This is what a current Typo3 initiative aims for:

Up until now the introduction and government packages had to be delivered as part of a whole TYPO3 installation as there was no easy way to provide out of the box installations without delivering the core with it. The new distribution management aims at making it possible to deliver and install full packages as normal extension via the extension manager. Therefor making maintenance of existing and creation of new packages much easier.

On the community end, we establish criteria that distros must meet for inclusion in the core distro service and have an approval process. Criteria could include:

• meeting interoperability standards
• Drupal import support (based on Migrate in core?).

As a result, distros become a Backdrop mainstay rather than an afterthought.

Whaddya think?

[quicksketch]

As a result, distros become a Backdrop mainstay rather than an afterthought.

I don't know if this is a viable option for our initial rollout of Backdrop 1.0. Considering we're attempting to tackle a problem that Drupal has failed repeatedly to deliver, but not for lack of trying.

In terms of our goals for Backdrop: being a product that is usable directly, I'm more inclined to take Backdrop the other direction: Remove the "minimal" install profile and make the "standard" profile the only option out-of-box. And make that standard profile actually into a useful set up with basic functionality: i.e. a blog and an image gallery. If you're a power-user that doesn't want anything, then you can install using the minimal profile through a less prominent option (e.g. the command line, a setting in settings.php, or a query string option, ?profile=minimal). The minimal option is useful for testing, so I don't think we'd remove it, but asking this question up-front can immediately ruin a users first experience.

Asking users to make decisions up-front during installation gives a sense of worry that they can't add those features later. And if they could make those decisions later (probably by enabling a module), then we should guide them to that as an option rather than asking them any questions at all up-front.

We don't yet know what the landscape looks like for D8/Backdrop modules, but my guess is that "feature packages" will be come much more common with the availability of CMI. If you can already package together content types and their fields, permissions, menus, and other configuration into a single module, doesn't that solve many of the problems distributions aimed to provide in the first place?

We're already working with limited resources and have our work cut out for us in 1.0 (http://handbook.backdropcms.org/versions/roadmap/). A significant investment on a feature that has yet to prove its value despite years of attempts in the Drupalsphere is not appealing.

[kreynen]

I think there are several reasons distributions aren't more popular and I agree w/ @nedjo that if this was addressed in Backdrop up front it could really impact how Backdrop is used... which could be a good thing.

One reason distributions aren't more popular is that by the time many people realize a distribution exists for the features they have been trying to find and configure themselves, moving to a distribution can be more confusing and time confusing than any benefit. I recently wrote https://drupal.org/project/profile_switcher and https://drupal.org/project/profile_status_check to make it easier to move a standard Drupal install to a distribution, then back off the modules the user had in sites/all/modules that can no be found in profile/[PROFILE NAME]/modules. Some modules like Media or OG require drush rr to move from sites/all to profiles/, so even with these modules it is probably still too difficult to move a site started w/ a standard install profile to a distribution for most site builders.

Another reason distributions aren't as popular is some people believe they are as less stable or secure than Drupal core… and they are right. Distributions are second class citizens in Drupal. OpenPublish still ships w/ the 7.16 and should be unpublished like any other project that allows users to download code with know security issues. The 1000+ sites running that are being told Drupal core isn't secure, but they can't easily update because core is so tied to the distribution. Even the popular Commerce distribution didn't update from 1.x to 2.x in Drupal 7… forget moving from a D6 distribution to D7. I don't know of any distribution that facilitated that.

Lists of "one click" install options like what Pantheon and Acquia Cloud are doing has definitely made it more likely someone might start a site from a distribution, but this could have been addressed on Drupal.org as well. If Drupal core wasn't so focused on being a CMS, but instead a packaged a Drupal CMS distribution as one of many distributions using the same process there wouldn't be this perception of Drupal being one thing and distributions being this other thing.

@quicksketch jumping to an image gallery as a feature that he'd like to include is like history repeating itself since several of the first Feature examples were image galleries. Of course a few developers named their Feature image_gallery when they exported and immediately created namespace conflicts. When I pushed DevSeed how they were going to resolve the namespace issue, their response was that everyone would eventually use Feature Servers and/or the namespace issue would work itself out. Obviously neither of those happened.

I think it might be possible to have our cake and eat it too on this one. Asking yourself 'Will something else included in the distribution make this a dependency?' can be used to determine when something needs to be included in core vs. a the Backdrop CMS distribution.

Views? Yes WYSIWYG? Maybe Image Gallery? Unlikely

Taking this approach shouldn't limit what's included in what most users install when using Backdrop. This is the approach used with CiviCRM. https://github.com/civicrm/civicrm-core is packaged a few different ways by the core team https://github.com/civicrm/civicrm-core/tree/master/distmaker/dists. Then there are a few other variations of that. There's a standalone version packaged for CiviDesk and then some changes are back ported into the LTS release https://github.com/CiviCRM42. Site builders would never download and attempt to install civicrm-core from github.

While I'd like to see this happen, one of the things I like about working with distributions is that communities form around the distribution of users with similar needs. Since Backdrop is still defining itself and how a Backdrop community might interact with the existing Drupal community, I don't know if having niche communities collaborating around OpenOutreach, CMDrupal, or CiviCRM would help or hurt.

[nedjo]

I posted a relevant Drupal 7 project this week that's similar to what I was suggesting above: https://www.drupal.org/sandbox/nedjo/2429233. The basic approach:

• Offer a curated list of distributions and offers them for installation at install time.
• Download the one that's selected and extract to the profiles directory, then hand off the rest of the install process to that distro.

I'm interested in doing the same for Backdrop, and would be particularly interested if this could be considered for a future core release.

Also on the topic of distros....

Digging into preparing for distros in Drupal 8, I've run into some unexpected roadblocks that may affect Backdrop distros as well.

I've described the issues in a couple of blog posts and a drupal.org issue.

In a nutshell: the CMI focus on the staging problem has created a number of barriers to distros, the main one arising from site configuration "living" on the site rather than in the providing modules.

I've begun work on a couple of relevant Drupal 8 modules: Configuration Packager as a replacement for Features and Configuration Synchronizer for importing configuration updates from modules and themes. These might be relevant to Backdrop, though they would need a lot of adapting.

[quicksketch]

the CMI focus on the staging problem has created a number of barriers to distros, the main one arising from site configuration "living" on the site rather than in the providing modules.

Right, the fact that config is stored in a central location instead of spread across modules makes module control of config more difficult. We understood that when taking it on, and expected that a contrib module would provide functionality to actually "bundle together" a collection of configuration to make a "feature". It would be nice if "Features" actually continued to live on, but did only what it was supposed to do in the first place, though I can understand wanting to avoid that particular name just because of the baggage.

One particular difference we have in our implementation (if I recall correctly) is that Backdrop still has a concept of "default" configuration. Views, Image Styles, and Layouts at least, all support "Reverting" configuration back into their default state. Unfortunately that a per-module implementation, not a universal system, but it could be extended to things like node types and (with a lot of work) fields.

I'm interested in doing the same for Backdrop, and would be particularly interested if this could be considered for a future core release.

Streamlining the installation process has been an important goal for us. We actually eliminated the selection of both an install profile (by hiding the "minimal" install) and the database type (but that was a side effect of only supporting MySQL). In my experience, it's quite common for new users to choose what essentially amounts to the "wrong" profile or database right off the bat, and that can be a very negative experience for the user, causing them to eliminate Drupal (or Backdrop) as an option for their site.

However, with that said, I'm quite open to including the functionality to support distributions out of the box, as long we we don't present users with multiple (potentially "wrong") choices by default.

[quicksketch]

Pulling a quote from your blog post:

In Drupal 8, module-provided configuration is imported once and once only--when the module is installed. The assumption is that, from that point onward, the configuration is "owned" by the site.

Backdrop includes a "module" key for several types of config files. In the event that a module provided the config in the first place, some modules allow you to revert those back to their default.

/**
 * Revert the changes made by users to a default image style.
 *
 * @param style
 *   An image style array.
 * @return
 *   Boolean TRUE if the operation succeeded.
 */
function image_default_style_revert($style) {
  image_style_flush($style);
  config('image.style.' . $style['name'])->delete();
  config_install_default_config($style['module'], 'image.style.' . $style['name']);
  return TRUE;
}

The way these modules keep track of whether a module is in the "default" state is by setting a flag in the config file. The "default" configuration provided by the module has this config value set to a constant indicating it's in a "default" state. Once the user makes any changes, that flag is set to be in an "overridden" state. Then UI elements allow that particular config file to be reverted, via a mechanism like the function above.

It's not super-clean and it's not even consistent at this point. One of the things I really wanted to do before release was to make Views, Layouts, and Image Styles all use the same keys for keeping track of whether a config was in its default or overridden state, but shipping was more important. I'd love to see a consistent mechanism for handing default/overridden/custom configuration.

[nedjo]

I'm quite open to including the functionality to support distributions out of the box, as long we we don't present users with multiple (potentially "wrong") choices by default.

Agreed with the aim here. The way I've written Distribution Installer is to require the profiles directory be writable at install time. (It might be better to reuse the FTP code used for in-browser module uploading, but I haven't done so yet. In D8 I'm not sure if that code's working, though I imagine it is in Backdrop.) One simple approach would be: a page to select a distro to install is presented only if the profiles directory is writable. So it becomes an option that's simple to invoke but not a default.

In the event that a module provided the config in the first place, some modules allow you to revert those back to their default.

Nice. I considered using a flag to track the default status of config items. For Configuration Synchronizer I opted for using a snapshot for comparison.

There are a bunch of distinct issues here. Probably I should try to put together a Backdrop equivalent of the issue META: Required functionality for Drupal 8 distributions. I'd need to look more closely at Backdrop code first.

[klonos]

Quite a few years later, I am wondering where we are with distribution support in https://github.com/backdrop-contrib.

Does one need to specify a .info file with type = distribution or something to have it be packaged? It certainly would not make any sense for an entire distro to be shown in the Project Browser, unless it was meant to be run as a "recipe", to download all required modules - but distros are more than a collection of modules. Nevertheless, we need a way to list them in b.org.

(reviving this old issue because we had a question re a distro: #3998)