META: Reduce boilerplate in composer.json and circle.yml

[stevector]

Although they are both less than 100 lines, both composer.json and circle.yml have a lot of complexity that we are likely to copy into other repos for WordPress, Drupal 7, and many Drupal distributions like Lightning and commerce. To make the maintenance of a growing number of Composer examples more sustainable, we should reevaluate dependencies and places code can be shared. I will make individual issues for specific topics like

• abstracting dependencies included for tests into a common testing package as we have done with WordPress: https://github.com/pantheon-systems/pantheon-wordpress-upstream-tests
• Removing installer-paths config from composer.json and moving it to a package (since we are already relying an a Pantheon-specific package to provide Quicksilver installer-paths)
• Reducing the need for repeated references to drupal-scaffold.
• Refactoring circle.yml for the coming CircleCI 2.0 (which will allow for common global dependencies used in many Pantheon repos like Terminus, Terminus plugins, and CGR to be pre-installed via Docker file)

[greg-1-anderson]

Related to #14

[greg-1-anderson]

A couple of words on goals.

Ultimately, for source code repositories, we would like to support:

• Github
• Bitbucket
• Gitlab

For CI services, we would like to support:

• Circle CI
• Travis CI
• Gitlab
• Jenkins

And, of course, for each of these, we want to support:

• Drupal 8
• Drupal 7
• WordPress

That's plenty of permutations, and it doesn't even count the distributions!

For all of those service, it is beneficial to move code/scripts that will not be customized by the user out of composer.json / circle.yml (or equivalent) and into a Terminus plugin or Quicksilver script or an external repository that can be cloned, as suggested in the OP.

In instances where end-user customization is necessary, it's better to put it in composer.json rather than circle.yml, as there will be multiple variants of the CI script file (for different services), but all will use the same composer.json file.

I'm strongly in favor of this effort as a whole. Let's discuss each potential improvement in its own issue, as suggested. We should be cautious about optimizations that introduce complexity, and do so only where necessary.

[stevector]

Let's discuss each potential improvement in its own issue

Agreed.

We should be cautious about optimizations that introduce complexity, and do so only where necessary.

Double agreed.

I put my thoughts on testing the permutations over in https://github.com/pantheon-systems/terminus-build-tools-plugin/issues/5

[greg-1-anderson]

Summary of discussion on reducing boilerplate.

• Our aspirational goal is to have a single project to require (per cms / framework) that makes any drupal-composer/drupal-project-like project (e.g. lightning, contenta, et. al.) work on Pantheon.
  • "Pick your base project, then run composer require pantheon-systems/pantheon-components-d8 to make it run on Pantheon."
  • The Terminus Build Tools Plugin will then be usable with any Composer-managed Drupal 8 distribution; it will just need to add in one more composer require after scaffolding the required site (which will be drupal-composer/drupal-project by default).
• We will use an incremental strategy to get there:
  • In the short term, we will continue to advise folks to use this repository, and will duplicate it as necessary to support other distributions.
  • To reach our eventual goal, we will include our "components" project as a dependency of this project, and migrate as many of the things in this project into the dependency project as possible.
  • We are done once everything has been migrated into the dependent project.
• There are a number of things that we can migrate right away (or remove):
  • The components we need / want from the require section of composer.json:
  • drush/drush n.b. CSE would like the Drush version to be strongly specified
  • webflo/drupal-core-strict
  • drupal/config_direct_save
  • drupal/simple-block
  • console/config.yml: This file specifies where the Drupal root is; however, this is no longer necessary, as Drupal Console can find the webroot without this configuration entry that was needed by older versions. Leaving this file in place is nominally useful, though, as it serves as a placeholder to signal users that they may customize their own Drupal Console settings. All the same, we could just remove this file.
  • drush/reinstall.drush.inc: We could probably drop this file in place with our pantheon-components project. The only function of this file is to make settings.php writable when Drush is used to re-install Drupal.

There are some items that will need additional thought / work to migrate.

composer.json items Some items in composer.json cannot be relocated to a dependency's composer.json file. For some of these, we can just presume that the parent project (drupal-composer/drupal-project) will have some strategy for the setting that we will simply adopt (e.g. minimum-stability, prefer-stable, etc.); we can document what values we recommend, but ultimately it would be up to the project-in-charge to decide what values should be used.

Drupal Scaffold

Currently, our example project includes the configuration settings that the drupal-scaffold tool uses to determine which files to scaffold. Currently, drupal-scaffold checks only the root-level composer.json for this file list. We could allow drupal-composer/drupal-project to scaffold the Drupal 8 files from the standard drupal.org repository, and then fix up the Pantheon customizations / additional files as a separate step handled by our project.

rvtraveller/qs-composer-installer

Composer installers need to be included from the top-level composer.json file. We'd probably need to make our pantheon-components repository a Composer installer, and have it either include the qs-composer-installer functionality, or find some way to forward the necessary calls to our event hooks to a child component.

Installation steps

The Terminus Build Tools Plugin currently reads the drush site-install command to run from the extras section of the root composer.json project. If we made a generic project with all of our necessary additions, then we'd need to be able to generate the install command ourselves. At the moment, the only part that needs to be customized is the installation profile to install. We could add some logic to the build tools to find the profile by some metric, and add an option to specify the profile so that the user could override the decision the tool would make.

Configuration export

The Terminus Build Tools Plugin exports configuration at the end of the build:create:project command; it currently finds the command to run from the extra section of composer.json.

Commit Comment Scripts These scripts are currently referenced from the circle.yml script et. al. to add a comment back to the PR that caused a multidev environmnet to be created. These scripts could be moved into the Terminus Build Tools Plugin.

PR Workflow

We wouldn't need to support the PR workflow as part of an MVP for our pantheon-components project; we could start off with the PR workflow as being something supported by the example drops-8 composer repository. Ultimately it would be nice if the PR workflow could be added to any project. Some / all of these operations could become part of the Pantheon platform instead.

• Build Steps: The Terminus Build Tools Plugin runs composer build-assets to allow folks to add their additional build steps (e.g. less or sass compile). We might need another way to do this.
• Composer install: composer install is currently part of the build-assets step. We could presume that composer install is always necessary and just run it.
• Prepare for Pantheon: Presently, the .gitignore for the canonical repository is set up such that the vendor directory and other build results are not committed. However, when the project is committed to the Pantheon repository, we would like to "repair" the .gitignore file so that build assets are added and committed to the repository. We cannot simply force-add everything in the build step; this would work for building, but would not be good for committing from the Pantheon dashboard.
• GitHub Pushback: This is already in a separate project, which is installed via rvtraveller/qs-composer-installer.

Behat Tests

Presumably behat test templates are provided by the project that the user selects to install (contenta, lightning et. al.)

[greg-1-anderson]

Looking at the composer installer documentation again, it looks like composer installers can perhaps be included from a sub-project, although the info in extras often must be in the top-level composer.json. I have not confirmed.

[stevector]

This package looks like a promising boilerplate reducer: https://github.com/hussainweb/drupal-composer-helper