Patch not executed with new Drupal 8.8 composer structure

ucola commented 4 years ago

Hi all

In Drupal 8.8 it is recommended a installation like this https://github.com/drupal-composer/drupal-project

That means that composer runs outside the Webroot. Drupal is inside a subfolder (i.e. web). The main problem, the composer runs outside this folder and the patch cannot be applyed, because all patches (from drupal.org) are core/XXX and not web/core/XXX.

So my question, is it possible to take care about the new recommanded Drupal Composer structure? That means is needed to put a path where the "webroot" is. Drupal itselfs have implement this for scaffold module like bellow. So the scaffold module know where the webroot is.

"extra": {
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        }
}

https://github.com/drupal-composer/drupal-project/blob/8.x/composer.json

cweagans commented 4 years ago

This plugin changes directories appropriately to the location that the composer package was installed to. Give it a try and if it doesn't work, please post a complete composer.json that reproduces the problem that you're having.

ucola commented 4 years ago

Thank you for your answer. We try it and it was not working. Our folder structure is like this (new recommended Drupal 8.8 Composer structure). |- .composer.json |- .composer.lock |- ... |- web ---|- core ---|- libraries ---|- modules ---|- ... ---|- index.php

How you see is the composer.json outside the webroot (web). So that means if you try to execute a patch it is not working, because the path of the patches are allways inside webroot. Bellow you see an example of the new composer.json. Inside this I have put a patch https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch but it is not possible to execute this patch.

{
    "name": "drupal-composer/drupal-project",
    "description": "Project template for Drupal 8 projects with composer",
    "type": "project",
    "license": "GPL-2.0-or-later",
    "authors": [
        {
            "name": "",
            "role": ""
        }
    ],
    "repositories": [
        {
            "type": "composer",
            "url": "https://packages.drupal.org/8"
        }
    ],
    "require": {
        "php": ">=7.0.8",
        "composer/installers": "^1.2",
        "cweagans/composer-patches": "^1.6.5",
        "drupal/console": "^1.0.2",
        "drupal/core": "^8.8.0",
        "drupal/core-composer-scaffold": "^8.8.0",
        "drush/drush": "^9.7.1 | ^10.0.0",
        "vlucas/phpdotenv": "^4.0",
        "webflo/drupal-finder": "^1.0.0",
        "zaporylie/composer-drupal-optimizations": "^1.0"
    },
    "require-dev": {
        "drupal/core-dev": "^8.8.0"
    },
    "conflict": {
        "drupal/drupal": "*"
    },
    "minimum-stability": "dev",
    "prefer-stable": true,
    "config": {
        "sort-packages": true
    },
    "autoload": {
        "classmap": [
            "scripts/composer/ScriptHandler.php"
        ],
        "files": ["load.environment.php"]
    },
    "scripts": {
        "pre-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "pre-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::checkComposerVersion"
        ],
        "post-install-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ],
        "post-update-cmd": [
            "DrupalProject\\composer\\ScriptHandler::createRequiredFiles"
        ]
    },
    "extra": {
        "composer-exit-on-patch-failure": true,
        "patches": {
            "drupal/shield": {
                "Exclude/Include Pages from Shield Protection (D8 Feature Port), https://www.drupal.org/project/shield/issues/2822720": "https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch"
            }
        },
        "patchLevel": {
            "drupal/core": "-p2"
        },
        "drupal-scaffold": {
            "locations": {
                "web-root": "web/"
            }
        },
        "installer-paths": {
            "web/core": ["type:drupal-core"],
            "web/libraries/{$name}": ["type:drupal-library"],
            "web/modules/contrib/{$name}": ["type:drupal-module"],
            "web/profiles/contrib/{$name}": ["type:drupal-profile"],
            "web/themes/contrib/{$name}": ["type:drupal-theme"],
            "drush/Commands/contrib/{$name}": ["type:drupal-drush"]
        }
    }
}

cweagans commented 4 years ago

Can you also do a composer install -vvv and paste the output either here or in a gist?

ucola commented 4 years ago

of corse

this is with new composer structure

composer install -vvv
Reading ./composer.json
Loading config file /home/XXX/.composer/auth.json
Loading config file ./composer.json
Checked CA file /usr/local/share/certs/ca-root-nss.crt: valid
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): git branch --no-color --no-abbrev -v
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): git describe --exact-match --tags
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): git log --pretty="%H" -n1 HEAD
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): hg branch
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): fossil branch list
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): fossil tag list
Executing command (/home/XXX/www/XXX.ch/tmp/some-dir): svn info --xml
Reading /home/XXX/.composer/composer.json
Loading config file /home/XXX/.composer/auth.json
Loading config file /home/XXX/.composer/composer.json
Loading config file /home/XXX/.composer/auth.json
Reading /home/XXX/.composer/auth.json
Reading /home/XXX/www/XXX.ch/tmp/some-dir/vendor/composer/installed.json
Reading /home/XXX/.composer/vendor/composer/installed.json
Loading plugin Composer\Installers\Plugin
Loading plugin cweagans\Composer\Patches
Loading plugin Drupal\Console\Composer\Plugin\Extender
Loading plugin Drupal\Composer\Plugin\Scaffold\Plugin
Loading plugin zaporylie\ComposerDrupalOptimizations\Plugin
Required tags were not explicitly set so the zaporylie/composer-drupal-optimizations set default based on project's composer.json content.
extra.commerce-drupal-optimizations.require.symfony/symfony: '>3.4'
Loading plugin Grasmash\ComposerConverter\Composer\Plugin
Running 1.7.1 (2018-08-07 09:39:23) with PHP 7.2.27 on FreeBSD / 11.3-RELEASE-p3
Reading ./composer.lock
Gathering patches for root package.
Removing package drupal/shield so that it can be re-installed and re-patched.
  - Removing drupal/shield (1.3.0-rc1)
Deleting web/modules/contrib/shield - deleted
> pre-install-cmd: DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Reading ./composer.lock
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.
Resolving dependencies through SAT
Looking at all rules.

Dependency resolution completed in 0.009 seconds
Analyzed 393 packages to resolve dependencies
Analyzed 1372 rules to resolve dependencies
Package operations: 1 install, 0 updates, 0 removals
Installs: drupal/shield:1.3.0-rc1
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
Found 1 patches for drupal/shield.
  - Installing drupal/shield (1.3.0-rc1): Reading /home/XXX/.composer/cache/files/drupal/shield/cc6375cb2d9f37f2229da0654b5d5e04039b6646.zip from cache
Loading from cache
 Extracting archiveExecuting command (CWD): unzip -qq  'web/modules/contrib/shield/19fdd73610fde6c0684b1803c347cda3.zip' -d '/home/XXX/www/XXX.ch/tmp/some-dir/vendor/composer/84e8e632'
    REASON: Required by the root package: Install command rule (install drupal/shield 1.3.0-rc1)

  - Applying patches for drupal/shield
    https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch (Exclude/Include Pages from Shield Protection (D8 Feature Port), https://www.drupal.org/project/shield/issues/2822720)
Downloading https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch
patch '-p1' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
Executing command (CWD): patch '-p1' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
patch: unrecognized option `--no-backup-if-mismatch'
usage: patch [-bCcEeflNnRstuv] [-B backup-prefix] [-D symbol] [-d directory]
             [-F max-fuzz] [-i patchfile] [-o out-file] [-p strip-count]
             [-r rej-name] [-V t | nil | never | none] [-x number]
             [-z backup-ext] [--posix] [origfile [patchfile]]
       patch <patchfile

patch '-p0' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
Executing command (CWD): patch '-p0' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
patch: unrecognized option `--no-backup-if-mismatch'
usage: patch [-bCcEeflNnRstuv] [-B backup-prefix] [-D symbol] [-d directory]
             [-F max-fuzz] [-i patchfile] [-o out-file] [-p strip-count]
             [-r rej-name] [-V t | nil | never | none] [-x number]
             [-z backup-ext] [--posix] [origfile [patchfile]]
       patch <patchfile

patch '-p2' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
Executing command (CWD): patch '-p2' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
patch: unrecognized option `--no-backup-if-mismatch'
usage: patch [-bCcEeflNnRstuv] [-B backup-prefix] [-D symbol] [-d directory]
             [-F max-fuzz] [-i patchfile] [-o out-file] [-p strip-count]
             [-r rej-name] [-V t | nil | never | none] [-x number]
             [-z backup-ext] [--posix] [origfile [patchfile]]
       patch <patchfile

patch '-p4' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
Executing command (CWD): patch '-p4' --no-backup-if-mismatch -d 'web/modules/contrib/shield' < '/tmp//5e4f6da822da1.patch'
patch: unrecognized option `--no-backup-if-mismatch'
usage: patch [-bCcEeflNnRstuv] [-B backup-prefix] [-D symbol] [-d directory]
             [-F max-fuzz] [-i patchfile] [-o out-file] [-p strip-count]
             [-r rej-name] [-V t | nil | never | none] [-x number]
             [-z backup-ext] [--posix] [origfile [patchfile]]
       patch <patchfile

   Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch

  [Exception]                                                                                                                                  
  Cannot apply patch Exclude/Include Pages from Shield Protection (D8 Feature Port), https://www.drupal.org/project/shield/issues/2822720 (ht  
  tps://www.drupal.org/files/issues/2019-06-12/2822720-51.patch)!

and this is with old structure. you see, here the patch is working. the versions of drupal/shield are exactly the same.

composer install -vvv
Reading ./composer.json
Loading config file /home/XXX/.composer/auth.json
Loading config file ./composer.json
Checked CA file /usr/local/share/certs/ca-root-nss.crt: valid
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): git branch --no-color --no-abbrev -v
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): git describe --exact-match --tags
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): git log --pretty="%H" -n1 HEAD
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): hg branch
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): fossil branch list
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): fossil tag list
Executing command (/home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot): svn info --xml
Reading /home/XXX/.composer/composer.json
Loading config file /home/XXX/.composer/auth.json
Loading config file /home/XXX/.composer/composer.json
Loading config file /home/XXX/.composer/auth.json
Reading /home/XXX/.composer/auth.json
Reading /home/XXX/www/XXX.ch/devucola8.XXX.ch/webroot/vendor/composer/installed.json
Reading /home/XXX/.composer/vendor/composer/installed.json
Loading plugin Composer\Installers\Plugin
Loading plugin cweagans\Composer\Patches
Loading plugin DrupalComposer\DrupalScaffold\Plugin
Loading plugin Drupal\Console\Composer\Plugin\Extender
Loading plugin OomphInc\ComposerInstallersExtender\Plugin
Loading plugin Wikimedia\Composer\MergePlugin
Loading plugin Grasmash\ComposerConverter\Composer\Plugin
Running 1.7.1 (2018-08-07 09:39:23) with PHP 7.2.27 on FreeBSD / 11.3-RELEASE-p3
Reading ./composer.lock
Gathering patches for root package.
Removing package drupal/shield so that it can be re-installed and re-patched.
  - Removing drupal/shield (1.3.0-rc1)
Deleting modules/contrib/shield - deleted
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Reading ./composer.lock
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.
Resolving dependencies through SAT
Looking at all rules.

Dependency resolution completed in 0.009 seconds
Analyzed 435 packages to resolve dependencies
Analyzed 1156 rules to resolve dependencies
Package operations: 1 install, 0 updates, 0 removals
Installs: drupal/shield:1.3.0-rc1
Gathering patches for root package.
Gathering patches for dependencies. This might take a minute.
Found 1 patches for drupal/core.
Found 1 patches for sngrl/php-firebase-cloud-messaging.
Found 1 patches for drupal/shield.
  - Installing drupal/shield (1.3.0-rc1): Executing command (CWD): git --version

    Cloning to cache at '/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/'
Executing command (/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/): git rev-parse --git-dir
Executing command (/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/): git rev-parse --quiet --verify '8.x-1.3-rc1^{commit}'
Executing command (/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/): git rev-parse --git-dir
Executing command (/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/): git remote -v
Executing command (/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/): git remote set-url origin 'https://git.drupalcode.org/project/shield.git' && git remote update --prune origin
Cloning 8.x-1.3-rc1 from cache
Executing command (CWD): git clone --no-checkout '/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/' 'modules/contrib/shield' --dissociate --reference '/home/XXX/.composer/cache/vcs/https---git.drupalcode.org-project-shield.git/' && cd 'modules/contrib/shield' && git remote set-url origin 'https://git.drupalcode.org/project/shield.git' && git remote add composer 'https://git.drupalcode.org/project/shield.git'
Executing command (modules/contrib/shield): git branch -r
Executing command (modules/contrib/shield): git checkout '8.x-1.3-rc1' -- && git reset --hard '8.x-1.3-rc1' --
    REASON: Required by the root package: Install command rule (install drupal/shield 1.3.0-rc1)

  - Applying patches for drupal/shield
    https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch (Exclude/Include Pages from Shield Protection (D8 Feature Port), https://www.drupal.org/project/shield/issues/2822720)
Downloading https://www.drupal.org/files/issues/2019-06-12/2822720-51.patch
Testing ability to patch with git apply. This command may produce errors that can be safely ignored.
git -C 'modules/contrib/shield' apply --check -v '-p1' '/tmp//5e4f707915f4c.patch'
Executing command (CWD): git -C 'modules/contrib/shield' apply --check -v '-p1' '/tmp//5e4f707915f4c.patch'
Checking patch README.md...

Checking patch config/install/shield.settings.yml...
Checking patch config/schema/shield.schema.yml...

Checking patch shield.module...

Checking patch shield.services.yml...

Checking patch src/Form/ShieldSettingsForm.php...

Checking patch src/ShieldMiddleware.php...

git -C 'modules/contrib/shield' apply '-p1' '/tmp//5e4f707915f4c.patch'
Executing command (CWD): git -C 'modules/contrib/shield' apply '-p1' '/tmp//5e4f707915f4c.patch'

> post-package-install: Drupal\Core\Composer\Composer::vendorTestCodeCleanup
    Processing drupal/shield
Package zendframework/zend-diactoros is abandoned, you should avoid using it. Use laminas/laminas-diactoros instead.
Package zendframework/zend-escaper is abandoned, you should avoid using it. Use laminas/laminas-escaper instead.
Package zendframework/zend-feed is abandoned, you should avoid using it. Use laminas/laminas-feed instead.
Package zendframework/zend-stdlib is abandoned, you should avoid using it. Use laminas/laminas-stdlib instead.
Generating autoload files

is strange, looks like the path is change correctly to execute the patch. On both I use 1.6.7 from your module

cweagans commented 4 years ago

What OS are you running? And what version of patch do you have?

ucola commented 4 years ago

both examples are on same server, with same composer-paches version. one is working the other not.

os: FreeBSD 11.3 package: cweagans/composer-patches 1.6.7

cweagans commented 4 years ago

https://github.com/cweagans/composer-patches/issues/182 https://github.com/cweagans/composer-patches/issues/159

It's a freebsd issue, though I'm not entirely sure how one is working and one is not.

ucola commented 4 years ago

for me is strange, why is one version working and the other not (on same serve with same release)?

I try right now and remove --no-backup-if-mismatch and now is working. can you give me a hint for a solution?

cweagans commented 4 years ago

i have no idea why it's working in one install but not another. it could be any number of things, but it's not a problem with this plugin.

things to check:

patch --version in both places
echo $PATH in both places
check the composer-patches version in both places
any other differences between the two projects that you can think of

FreeBSD in general has been a weird thing to support for this project. Unfortunately, it's probably not something that's going to be well supported in the future.

ucola commented 4 years ago

i dosen't understand it, because this is the same server, only another folder....

patch --version on both > patch 2.0-12u11 FreeBSD
echo $PATH on both > /usr/local/bin:/usr/local/bin:/usr/bin:/bin:/home/XXX/npm_modules/node_modules/.bin:/home/XXX/drush:/home/XXX/.composer/vendor/bin
composer-patches both > 1.6.7
differences > on one we have .composer.json outside webroot at the other one inside webroot

Did we need the --no-backup-if-mismatch parameter?

cweagans commented 4 years ago

it was important to not have .bak files all over the place. not sure how freebsd handles it. the two issues i linked above have more info about freebsd issues specifically. i don't think this is a composer-patches plugin issue though -- it's a difference in how patch works on different platforms.

ucola commented 4 years ago

thank you for your answer, how I see is this still an issue with freebsd. I have contact my hosting provider to update the patch module.

https://www.drupal.org/project/lightning/issues/2911843#comment-12560493

but another question, is not possible to add a quick check which OS is used and if freebsd execute patch command without --no-backup-if-mismatch?

cweagans commented 1 year ago

I missed it before, but it looks like a patch was being applied to drupal/shield, which is not required by the original composer.json - which is why that wouldn't be patched.

main has better support for different versions of patch, including BSD Patch.

cweagans / composer-patches

Patch not executed with new Drupal 8.8 composer structure #300