Build fails on OSX

[jasonruyle]

I'm using a late 2009 Mac, so am forced to use the boot2docker setup. Everything else is up to date (OS, docker-machine, docker).

Running docker-machine up works, things are built. docker-compose exec -u buildkit civicrm civibuild create dmaster --url http://localhost:8080

This seems to run fine, but then hits an error with the Drupal install.

There are issues with Mac/Docker/Drupal with file/folder permissions. One way others have solved this is to allow for the primary user to have the default macOS user permission. The user would need 501:20 uid/gid.

Do you have any suggestions on how we might apply those permissions to user buildkit?

Here is the log info:

+++ chmod u-w sites/default/settings.php
+++ amp datadir sites/default/files /buildkit/app/private/dmaster/default
Update data directory: sites/default/files
PHP Fatal error:  Uncaught RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php:43
Stack trace:
#0 phar:///buildkit/bin/amp/src/Amp/Command/DatadirCommand.php(40): Amp\Permission\External->applyDirPermission('write', 'sites/default/f...')
#1 phar:///buildkit/bin/amp/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(252): Amp\Command\DatadirCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 phar:///buildkit/bin/amp/vendor/symfony/console/Symfony/Component/Console/Application.php(889): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 phar:///buildkit/bin/amp/vendor/symfony/console/Symfony/Component/Console/Application.php(193): Symfony\Component\Console\Application->doRunCommand(Object(Amp\Command\DatadirCommand), Object(Symfony\Component\Console\Inpu in phar:///buildkit/bin/amp/src/Amp/Permission/External.php on line 43

Fatal error: Uncaught RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php on line 43

RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php on line 43

Call Stack:
    0.0007     353672   1. {main}() /buildkit/bin/amp:0
    0.0094     543320   2. require('phar:///buildkit/bin/amp/bin/amp') /buildkit/bin/amp:10
    0.0154     844208   3. Amp\Application::main() phar:///buildkit/bin/amp/bin/amp:18
    0.0821    3100448   4. Amp\Application->run() phar:///buildkit/bin/amp/src/Amp/Application.php:59

[michaelmcandrew]

hey there @jasonruyle,

Thanks for the submitting the issue. Have a look here for some discussion. One way of solving this if you want to get up and running is to ask amp to choose a different permission schema:

  [4] "worldWritable": Set loose, generic permissions [chmod 1777] (discouraged)

should fix it.

This setting is defined in/buildkit/.amp/services.yml on the civicrm container.

Not sure how familiar you are with Docker, but there are various approaches you could take to overwriting that file. One is to edit the Dockerfile and rebuild the image. Another is to mount an edited file from the host system.

That will get you up and running but it is more of a patch.

A better solution requires a bit more head scratching. Patch/scratch welcome :)

[michaelmcandrew]

PS. It might be that the patch is to buildkit or amp

[jasonruyle]

Hi @michaelmcandrew - Thanks for the response. I'll try and update the permissions to start, but have a feeling its coming from buildkit or amp.

I did attempt to change the permissions on the system, assigning uid/gid, etc. I also tried to update the Dockerfile with something I use for other development I do with Drupal.

If I find a solution I'll post to here.

[michaelmcandrew]

Did you try the solution I suggested above?

You should be able to fix it by changing the amp configuration.

As mentioned, that is defined in /buildkit/.amp/services.yml on the civicrm container. Change the [perm_type](https://github.com/michaelmcandrew/civicrm-buildkit-docker/blob/master/civicrm/amp.services.yml#L5 to worldWritable, ensure the change is propogated to the container and it should work.

[jasonruyle]

Did a full image rebuild, built new container and docker-compose exec with the update to amp.services.yml (worldWritable) and have the same issue.

I think its more then just making the folder/files world writable. I'm guessing it has to do with OSX specific user priv.

Something I have to do in other projects for Drupal is to run the following in the Dockerfile:

RUN usermod --non-unique --uid 1000 www-data \
  && groupmod --non-unique --gid 1000 www-data \
  && chown -R www-data:www-data /app/web

Others have suggested the numbers are 501:20 (wodby/docker4drupal)

But I'm not sure what step something like this can be run for this install.

[michaelmcandrew]

Hey there,

If you go the same error, i.e.

PHP Fatal error:  Uncaught RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php:43

Then you are still using setFacl.

If you were using worldWritable (see below) then I would except to see a different error.

   [0] "none": Do not set any special permissions for the web user
  [1] "linuxAcl": Set tight, inheritable permissions with Linux ACLs [setfacl] (recommended)
         In some distros+filesystems, this requires extra configuration.
         eg For Debian-based distros: https://help.ubuntu.com/community/FilePermissionsACLs
  [2] "osxAcl": Set tight, inheritable permissions with OS X ACLs [chmod +a] (recommended)
  [3] "custom": Set permissions with a custom command
  [4] "worldWritable": Set loose, generic permissions [chmod 1777] (discouraged)

I don't think it makes sense to go down the route of trying to match UIDs across the host and container.

[chrisfromredfin]

Hi there - like @jasonruyle I've edited civicrm/amp.services.yml and changed it, and then I re-ran

docker-compose exec -u buildkit civicrm civibuild create dmaster --url http://localhost:8080

But alas, I still have the same error. I looked for perm_type elsewhere. Should it be one of these that needs to change?

⟩ grep -R 'perm_type' *
civicrm/amp.services.yml:    perm_type: worldWritable
publish/civicrm/php7.1/amp.services.yml:    perm_type: linuxAcl
publish/civicrm/php7.0/amp.services.yml:    perm_type: linuxAcl
publish/civicrm/php5.6/amp.services.yml:    perm_type: linuxAcl
publish/civicrm/php7.2/amp.services.yml:    perm_type: linuxAcl
publish/templates/civicrm/amp.services.yml.twig:    perm_type: linuxAcl

[michaelmcandrew]

Can you copy and paste the error messages you are receiving?

On Mon, 26 Mar 2018, 16:43 Chris Wells, notifications@github.com wrote:

Hi there - like @jasonruyle https://github.com/jasonruyle I've edited civicrm/amp.services.yml and changed it, and then I re-ran

docker-compose exec -u buildkit civicrm civibuild create dmaster --url http://localhost:8080

But alas, I still have the same error. I looked for perm_type elsewhere. Should it be one of these that needs to change?

⟩ grep -R 'perm_type' * civicrm/amp.services.yml: perm_type: worldWritable publish/civicrm/php7.1/amp.services.yml: perm_type: linuxAcl publish/civicrm/php7.0/amp.services.yml: perm_type: linuxAcl publish/civicrm/php5.6/amp.services.yml: perm_type: linuxAcl publish/civicrm/php7.2/amp.services.yml: perm_type: linuxAcl publish/templates/civicrm/amp.services.yml.twig: perm_type: linuxAcl

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/michaelmcandrew/civicrm-buildkit-docker/issues/15#issuecomment-376212678, or mute the thread https://github.com/notifications/unsubscribe-auth/AAM0L1EC3OcXKkqbVLLWMPeznuW6uf4Gks5tiQy6gaJpZM4S3Mst .

[chrisfromredfin]

First I edited the file:

~/src/civicrm-buildkit-docker · (master±)
⟩ git diff
diff --git a/civicrm/amp.services.yml b/civicrm/amp.services.yml
index eb3f764..d8a02a0 100644
--- a/civicrm/amp.services.yml
+++ b/civicrm/amp.services.yml
@@ -2,7 +2,7 @@ parameters:
     version: 2
     db_type: mysql_dsn
     mysql_dsn: 'mysql://root:buildkit@mysql:3306'
-    perm_type: linuxAcl
+    perm_type: worldWritable
     perm_user: www-data
     hosts_type: file
     httpd_type: apache24

Then made sure I was up to date:

~/src/civicrm-buildkit-docker · (master±)
⟩ docker-compose up -d
Starting civicrmbuildkitdocker_mysql_1   ... done
Starting civicrmbuildkitdocker_maildev_1 ... done
Starting civicrmbuildkitdocker_phpmyadmin_1 ... done
Starting civicrmbuildkitdocker_civicrm_1    ... done

Then I tried the exec:

~/src/civicrm-buildkit-docker · (master±)
⟩ docker-compose exec -u buildkit civicrm civibuild create dmaster --url http://localhost:8080
[[Download dmaster (type 'drupal-demo' in '/buildkit/build/dmaster')]]
Already downloaded dmaster
[[Install dmaster/default (type 'drupal-demo' in '/buildkit/build/dmaster')]]
[[Execute /buildkit/app/config/drupal-demo/install.sh]]
++ source /buildkit/app/config/drupal-demo/install.sh
+++ amp_install
+++ _amp_install_cms
+++ echo '[[Setup MySQL and HTTP for CMS]]'
[[Setup MySQL and HTTP for CMS]]
+++ cvutil_assertvars _amp_install_cms CMS_ROOT SITE_NAME SITE_ID TMPDIR
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ mktemp.php ampvar
+++ local amp_vars_file_path=/tmp/ampvaruXc7tr
+++ local amp_name=cmsdefault
+++ '[' default == default ']'
+++ amp_name=cms
+++ '[' -n http://localhost:8080 ']'
+++ amp create -f --root=/buildkit/build/dmaster --name=cms --prefix=CMS_ --url=http://localhost:8080 --output-file=/tmp/ampvaruXc7tr --perm=admin
+++ source /tmp/ampvaruXc7tr
++++ CMS_URL=http://localhost:8080
++++ CMS_ROOT=/buildkit/build/dmaster
++++ CMS_DB_DSN='mysql://dmastercms_z7dus:h8zzYwveO9IvAjFf@mysql:3306/dmastercms_z7dus?new_link=true'
++++ CMS_DB_USER=dmastercms_z7dus
++++ CMS_DB_PASS=h8zzYwveO9IvAjFf
++++ CMS_DB_HOST=mysql
++++ CMS_DB_PORT=3306
++++ CMS_DB_NAME=dmastercms_z7dus
++++ CMS_DB_ARGS='--defaults-file='\''/buildkit/.amp/my.cnf.d/my.cnf-b2d48571322fabfa35c9e82f9b2db8df'\'' dmastercms_z7dus'
+++ rm -f /tmp/ampvaruXc7tr
+++ _amp_install_civi
+++ echo '[[Setup MySQL for Civi]]'
[[Setup MySQL for Civi]]
+++ cvutil_assertvars _amp_install_civi CMS_ROOT SITE_NAME SITE_ID TMPDIR
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ mktemp.php ampvar
+++ local amp_vars_file_path=/tmp/ampvartvKpcu
+++ local amp_name=cividefault
+++ '[' default == default ']'
+++ amp_name=civi
+++ amp create -f --root=/buildkit/build/dmaster --name=civi --prefix=CIVI_ --skip-url --output-file=/tmp/ampvartvKpcu --perm=super
+++ source /tmp/ampvartvKpcu
++++ CIVI_URL=
++++ CIVI_ROOT=/buildkit/build/dmaster
++++ CIVI_DB_DSN='mysql://dmasterciv_sxplw:FQukXyiL72hgE4hS@mysql:3306/dmasterciv_sxplw?new_link=true'
++++ CIVI_DB_USER=dmasterciv_sxplw
++++ CIVI_DB_PASS=FQukXyiL72hgE4hS
++++ CIVI_DB_HOST=mysql
++++ CIVI_DB_PORT=3306
++++ CIVI_DB_NAME=dmasterciv_sxplw
++++ CIVI_DB_ARGS='--defaults-file='\''/buildkit/.amp/my.cnf.d/my.cnf-6eff11cff96136843aaee64f3625e4f5'\'' dmasterciv_sxplw'
+++ rm -f /tmp/ampvartvKpcu
+++ _amp_install_test
+++ echo '[[Setup MySQL for Test]]'
[[Setup MySQL for Test]]
+++ cvutil_assertvars _amp_install_test CMS_ROOT SITE_NAME SITE_ID TMPDIR
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ mktemp.php ampvar
+++ local amp_vars_file_path=/tmp/ampvarYqTlTL
+++ local amp_name=testdefault
+++ '[' default == default ']'
+++ amp_name=test
+++ amp create -f --root=/buildkit/build/dmaster --name=test --prefix=TEST_ --skip-url --output-file=/tmp/ampvarYqTlTL --perm=super
+++ source /tmp/ampvarYqTlTL
++++ TEST_URL=
++++ TEST_ROOT=/buildkit/build/dmaster
++++ TEST_DB_DSN='mysql://dmastertes_4lfia:hb1Itq88cf1B8mSV@mysql:3306/dmastertes_4lfia?new_link=true'
++++ TEST_DB_USER=dmastertes_4lfia
++++ TEST_DB_PASS=hb1Itq88cf1B8mSV
++++ TEST_DB_HOST=mysql
++++ TEST_DB_PORT=3306
++++ TEST_DB_NAME=dmastertes_4lfia
++++ TEST_DB_ARGS='--defaults-file='\''/buildkit/.amp/my.cnf.d/my.cnf-beb43bca07892bf062c4fb86391d710b'\'' dmastertes_4lfia'
+++ rm -f /tmp/ampvarYqTlTL
+++ drupal_install
+++ drupal7_install
+++ cvutil_assertvars drupal7_install CMS_ROOT SITE_ID CMS_TITLE CMS_DB_USER CMS_DB_PASS CMS_DB_HOST CMS_DB_NAME ADMIN_USER ADMIN_PASS CMS_URL
+++ _cvutil_assertvars_back=ehxB
+++ set +x
++++ _drupal_multisite_dir http://localhost:8080 default
++++ '[' default == default ']'
++++ echo default
+++ DRUPAL_SITE_DIR=default
++++ cvutil_build_hostport mysql 3306
++++ local host=mysql
++++ local port=3306
++++ '[' -z 3306 ']'
++++ echo mysql:3306
+++ CMS_DB_HOSTPORT=mysql:3306
+++ pushd /buildkit/build/dmaster
+++ '[' -f sites/default/settings.php ']'
+++ rm -f sites/default/settings.php
+++ drush site-install -y --db-url=mysql://dmastercms_z7dus:h8zzYwveO9IvAjFf@mysql:3306/dmastercms_z7dus --account-name=admin --account-pass=FcvUQnBzDjjV --account-mail=admin@example.com '--site-name=CiviCRM Sandbox on Drupal' --sites-subdir=default
You are about to create a sites/default/settings.php file and DROP all tables in your 'dmastercms_z7dus' database. Do you want to continue? (y/n): y
No tables to drop.                                                                                                      [ok]
Starting Drupal installation. This takes a few seconds ...                                                              [ok]
Installation complete.  User name: admin  User password: FcvUQnBzDjjV                                                   [ok]
+++ chmod u+w sites/default
+++ chmod u+w sites/default/settings.php
+++ cvutil_inject_settings /buildkit/build/dmaster/sites/default/settings.php drupal.settings.d
+++ local FILE=/buildkit/build/dmaster/sites/default/settings.php
+++ local NAME=drupal.settings.d
+++ cvutil_assertvars cvutil_inject_settings PRJDIR SITE_NAME SITE_TYPE SITE_CONFIG_DIR SITE_ID SITE_TOKEN PRIVATE_ROOT FILE NAME
+++ _cvutil_assertvars_back=ehxB
+++ set +x
+++ local TMPFILE=/buildkit/app/tmp/drupal-demo/dmaster/default.settings.tmp
+++ cvutil_makeparent /buildkit/app/tmp/drupal-demo/dmaster/default.settings.tmp
++++ dirname /buildkit/app/tmp/drupal-demo/dmaster/default.settings.tmp
+++ parent=/buildkit/app/tmp/drupal-demo/dmaster
+++ '[' '!' -d /buildkit/app/tmp/drupal-demo/dmaster ']'
+++ cat
+++ sed 's/^<?php//'
+++ cat
+++ cat
+++ chmod u-w sites/default/settings.php
+++ amp datadir sites/default/files /buildkit/app/private/dmaster/default
Update data directory: sites/default/files
PHP Fatal error:  Uncaught RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php:43
Stack trace:
#0 phar:///buildkit/bin/amp/src/Amp/Command/DatadirCommand.php(40): Amp\Permission\External->applyDirPermission('write', 'sites/default/f...')
#1 phar:///buildkit/bin/amp/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(252): Amp\Command\DatadirCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#2 phar:///buildkit/bin/amp/vendor/symfony/console/Symfony/Component/Console/Application.php(889): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#3 phar:///buildkit/bin/amp/vendor/symfony/console/Symfony/Component/Console/Application.php(193): Symfony\Component\Console\Application->doRunCommand(Object(Amp\Command\DatadirCommand), Object(Symfony\Component\Console\Inpu in phar:///buildkit/bin/amp/src/Amp/Permission/External.php on line 43

Fatal error: Uncaught RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php on line 43

RuntimeException: setfacl: sites/default/files: Operation not supported
 in phar:///buildkit/bin/amp/src/Amp/Permission/External.php on line 43

Call Stack:
    0.0003     353352   1. {main}() /buildkit/bin/amp:0
    0.0057     543000   2. require('phar:///buildkit/bin/amp/bin/amp') /buildkit/bin/amp:10
    0.0089     843888   3. Amp\Application::main() phar:///buildkit/bin/amp/bin/amp:18
    0.0617    3100128   4. Amp\Application->run() phar:///buildkit/bin/amp/src/Amp/Application.php:59

[jasonruyle]

Same problems as myself. I tried moving to different locations on my system. I tried changing buildkit permissions, etc, and nothing.

Maybe try in Dockerfile:

RUN usermod --non-unique --uid 1000 www-data \
  && groupmod --non-unique --gid 1000 www-data \
  && chown -R www-data:www-data /buildkit/app/web

The final path is probably wrong. I wasn't able to get this working, but in other Dockerfile I run for Drupal installation the above snippet has worked.

[veda-consulting]

Just a quick note that I'm seeing the same thing on Mac OS

[jasonruyle]

The other deployment systems for docker and drupal usually have a way to find the id-u (which I think defaults to 502 on mac. I'm not sure where this would have to be passed for permission purposes in buildkit. I wrote the sample above, but I'm guessing it would be more like:

RUN usermod -u 501 www-data
RUN usermod -G staff www-data

[michaelmcandrew]

@veda-consulting - makes sense - this issue affects all macs. I wouldn't expect this to work on any mac until this issue is resolved.

Until someone comes along that wants to dig into the guts (which will require an understanding of Mac, Docker and buildkit) the below is a fairly decent workaround that allows people to get it up and running.

I am not a mac owner so can't go through the motions myself but the approach I outlined above does make sense to me and something similar worked for kcristiano.

@jasonruyle and @chrisfromredfin. You missed the part that rebuilds the image that I mentioned here:

Not sure how familiar you are with Docker, but there are various approaches you could take to overwriting that file. One is to edit the Dockerfile and rebuild the image. Another is to mount an edited file from the host system.

and here:

ensure the change is propagated to the container and it should work.

Just changing the Dockerfiles is not enough. You need a step that creates an image/container from your new Dockerfile or need to get the amp config on to your container some other way.

Getting more familiar with docker (specifically how a Dockerfile docker-compose.yml files works is going to help you a lot in getting this working. I can't do it for you, but I can give you a couple of pointers in the right direction.

The basic steps are:

1. Make the change in the amp.services.yml file of the civicrm directory. (I think you have already done that)
2. Ensure that change is propogated to the container. There are mutiple ways to do this, e.g.
   1. rebuild the image. One approach is to change this line in the docker-compose.yml file to reference your local build rather than my image.
   2. Another quick and dirty way to get things up and running would be to edit the file on the running container (there are multiple ways to do this)

Have a look at the following resources on Dockerfiles and docker-compose.yml.

• https://docs.docker.com/engine/reference/builder/
• https://docs.docker.com/compose/

Let me know if you get stuck on the above and I will try and help.

aside for @jasonruyle: the setfacl error that is being reported is due to the fact that setfacl does not exist on a Mac. No amount of fiddling with user ids is going to change that fact.

Extended OS ACL is nice but I think the medium/long term solution is to create a new permssions schema for buildkit that does not rely on 'linuxAcl' or "osxAcl". Most other cloud native apps do this so we should be able to too.

[michaelmcandrew]

Hey there,

After a bit more discussion and thinking on this subject, I've created a new branch perm-none that runs apache as the buildkit user and switches to amp's none permission scheme. I'm hopeful that this will solve the problems for mac users.

I'm trying it out locally on my linux box. would be great to get a bit of feedback from on macs.

If you want to try this out, please:

1. switch to the new branch locally
2. rebuild the containers with the following command (from your docker-compose-buildkit directory). docker-compose up -d --build (this step will take a while)
3. remove the amp volume with something like docker volume rm civicrmbuildkitdocker_amp (you can find the exact name for the volume you need to remove with docker volume ls. It should be based on the name of the directory you cloned this repo to and end with _amp)
4. start up the containers as normal.

Let me know if this works for you / you have any questions.

[veda-consulting]

@michaelmcandrew that seems to work for me. I'm getting a few issues which seem to be path related but thats different to the original post on this issue.

[jasonruyle]

Glad someone got it going with the update and appreciate you taking time out of your day to look at this stuff. I did a git clone to new area, make sure the volume, etc was gone and checked out the new branch perm-none.

Got to the following and failed. Not sure what there is to do at this point. I think this is the correct setup for Mac users, probably just my system that is the problem.

Build process completed successfully
Installing '/usr/local/lib/php/extensions/no-debug-non-zts-20151012/xdebug.so'
install ok: channel://pecl.php.net/xdebug-2.6.0
configuration option "php_ini" is not set to php.ini location
You should add "zend_extension=/usr/local/lib/php/extensions/no-debug-non-zts-20151012/xdebug.so" to php.ini
Removing intermediate container ab5fab48db43
 ---> 942fcbc170c7
Step 5/28 : RUN a2enmod rewrite
 ---> Running in 185433ee7c35
Enabling module rewrite.
To activate the new configuration, you need to run:
  service apache2 restart
Removing intermediate container 185433ee7c35
 ---> 51b4e2b56824
Step 6/28 : RUN useradd --home-dir /buildkit --create-home buildkit
 ---> Running in 0b427efe52c3
Removing intermediate container 0b427efe52c3
 ---> 70f6bcc72d45
Step 7/28 : COPY sudo /etc/sudoers.d/buildkit
 ---> ddd432840762
Step 8/28 : USER buildkit
 ---> Running in 61f5f929620f
Removing intermediate container 61f5f929620f
 ---> df96625fa514
Step 9/28 : WORKDIR /buildkit
Removing intermediate container b55282539b70
 ---> c01954b86fdf
Step 10/28 : ENV PATH="/buildkit/bin:${PATH}"
 ---> Running in 61c0db9e674a
Removing intermediate container 61c0db9e674a
 ---> c3e2b72fc053
Step 11/28 : RUN git clone https://github.com/civicrm/civicrm-buildkit.git buildkit-tmp
 ---> Running in 74ae4f79848b
Cloning into 'buildkit-tmp'...
Removing intermediate container 74ae4f79848b
 ---> e7b82ca9a715
Step 12/28 : RUN mv buildkit-tmp/* buildkit-tmp/.git* .
 ---> Running in cc1c32262c91
mv: cannot create hard link './bin/cividist' to './app/backdrop.settings.d': Operation not permitted
mv: cannot create hard link './bin/pdepend-civi-csv' to './app/config': Operation not permitted
mv: cannot create hard link './bin/mkpasswd.php' to './app/civicrm.settings.d': Operation not permitted
mv: cannot create hard link './bin/phpunit-xml-cleanup' to './app/wp-config.d': Operation not permitted
mv: cannot create hard link './bin/codecept' to './app/drupal.settings.d': Operation not permitted
mv: cannot create hard link './bin/givi' to './app/civitest.settings.d': Operation not permitted
mv: cannot create hard link './bin/gitify' to './app/backdrop.settings.d/post.d': Operation not permitted
mv: cannot create hard link './bin/releaser' to './app/config/extdir': Operation not permitted
mv: cannot create hard link './bin/fetch-universe' to './app/config/wp-empty': Operation not permitted
mv: cannot create hard link './bin/find-symlink-tree-orphans' to './app/config/hr15': Operation not permitted
mv: cannot create hard link './bin/civi-download-tools' to './app/config/dist': Operation not permitted
mv: cannot create hard link './bin/pidlockfile.php' to './app/config/backdrop-empty': Operation not permitted
mv: cannot create hard link './bin/regmv' to './app/config/drupal8-clean': Operation not permitted
mv: cannot create hard link './bin/phpunit-each' to './app/config/drupal-clean': Operation not permitted
mv: cannot create hard link './bin/phpcs-civi' to './app/config/l10n': Operation not permitted
mv: cannot create hard link './bin/civibuild' to './app/config/hr17': Operation not permitted
mv: cannot create hard link './bin/civilint' to './app/config/drupal8-empty': Operation not permitted
mv: cannot create hard link './bin/extract-url' to './app/config/drupal-demo': Operation not permitted
mv: cannot create hard link './bin/civihydra' to './app/config/joomla-demo': Operation not permitted
mv: cannot create hard link './bin/find-stale-builds' to './app/config/prevem': Operation not permitted
mv: cannot create hard link './bin/timeout.php' to './app/config/symfony': Operation not permitted
mv: cannot create hard link './bin/civi-test-run' to './app/config/drupal-empty': Operation not permitted
mv: cannot create hard link './bin/mktemp.php' to './app/config/wp-case': Operation not permitted
mv: cannot create hard link './bin/phpcbf-civi' to './app/config/backdrop-clean': Operation not permitted
mv: cannot create hard link './bin/pdepend-to-csv' to './app/config/joomla-empty': Operation not permitted
mv: cannot create hard link './src/create-civicase-examples.php' to './app/config/drupal-clean42': Operation not permitted
mv: cannot create hard link './src/civibuild.app.sh' to './app/config/drupal-case': Operation not permitted
mv: will not create hard link './src/views' to directory './app/config/drupal8-demo'
mv: cannot create hard link './src/releaser_json.php' to './app/config/backdrop-demo': Operation not permitted
mv: cannot create hard link './src/civibuild.settings.php' to './app/config/distmgr': Operation not permitted
mv: cannot create hard link './src/civibuild.defaults.sh' to './app/config/cxnapp': Operation not permitted
mv: cannot create hard link './src/cividist-aggregate-json.php' to './app/config/empty': Operation not permitted
mv: cannot create hard link './src/civicrm.settings.php' to './app/config/wmff': Operation not permitted
mv: cannot create hard link './src/civibuild.compute-defaults.sh' to './app/config/hr16': Operation not permitted
mv: will not create hard link './src/help' to directory './app/config/wp-demo'
mv: cannot create hard link './src/civibuild.aliases.sh' to './app/config/starterkit': Operation not permitted
mv: will not create hard link './src/Civi' to directory './app/config/messages'
mv: cannot create hard link './src/civibuild.lib.sh' to './app/config/universe': Operation not permitted
mv: will not create hard link './src/command' to directory './app/config/doctrine'
mv: cannot create hard link './src/civibuild.parse.sh' to './app/config/hrdemo': Operation not permitted
mv: will not create hard link './src/drush' to directory './app/config/hr15/drupal.settings.d'
mv: will not create hard link './tests/phpunit' to directory './app/civicrm.settings.d/pre.d'
mv: cannot create hard link './vagrant/README.md' to './app/civitest.settings.d/pre.d': Operation not permitted
ERROR: Service 'civicrm' failed to build: The command '/bin/sh -c mv buildkit-tmp/* buildkit-tmp/.git* .' returned a non-zero code: 1

[michaelmcandrew]

I wonder if you guys are using the AUFS storage driver? If so, try switching to Overlay2 and trying again. See http://markshust.com/2017/03/02/making-docker-mac-faster-overlay2-filesystem for a bit more information.

[veda-consulting]

@michaelmcandrew @jasonruyle to confirm, my setup (which works) is as follows

Parvezs-Mac-mini:.ssh parvezsaleh$ docker info | grep "Storage Driver" Storage Driver: overlay2

[michaelmcandrew]

@veda-consulting thanks for the feedback.

@jasonruyle - let me know if this change works for you as well and I will pull in the changes to the master branch and close this issue.

[jasonruyle]

I'm on aufs. That is the issue. Here is what I did with my legacy OSX (boot2docker), but parts might help others as well. Ran docker info to make sure I was on overlay2.

Have to admit, still not able to access, but the install went correct. This is the solution for mac users.

docker-machine create -d virtualbox --engine-storage-driver overlay2 osxdefault
docker-machine env osxdefault
eval $(docker-machine env osxdefault)
docker info
docker-compose up -d --build
docker-compose exec -u buildkit civicrm civibuild create dmaster --url http://localhost:8080

[michaelmcandrew]

I've created a new release with a updated docker image and docker-compose.yml: https://github.com/michaelmcandrew/civicrm-buildkit-docker/releases/tag/0.6.0.

@jasonruyle - i am closing this issue for now since it feels like it is fixed but give me a shout if you are still having issues and I'll try and help get you up and running.

[totten]

Just giving a report-back after doing a new run on OSX... and it (mostly) worked! Hoowa!

So far, I've run into two snags.

First snag -- After doing a fresh git clone and docker-compose up -d, amp was still configured with perm_type=linuxAcl.

• It's possible that my local docker environment retained some unseen artifact (image or volume) from a few weeks ago -- and I was too ignorant to clean it up? Or perhaps the downloaded image was out-of-date?
• I switched from image: michaelmcandrew/civicrm to build: civicrm. This built a new image, but it didn't immediately make a difference. Again that might have been my docker-inexperience. The first build probably initialized .amp, and then those values stayed around. So perhaps I could've cleared that.
• Regardless, I hopped into a bash console, ran amp config:set --perm_type=none
• Then builds started to work. Both CLI and HTTP. Whoohoo!

Second snag -- unit tests are markedly slower than when running locally. To benchmark, I ran a subset of the test suite (~500 tests):

time env CIVICRM_UF=UnitTests phpunit4 tests/phpunit/CRM/Core/AllTests.php

This was going slow, so I aborted and applied Testing Performance Trick #1 -- run mysql on ram disk:

$ git diff docker-compose.yml
diff --git a/docker-compose.yml b/docker-compose.yml
index 5dd6d8b..8facc7e 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -28,8 +28,8 @@ services:
     image: mysql:5.7
     environment:
       MYSQL_ROOT_PASSWORD: buildkit
-    volumes:
-      - mysql:/var/lib/mysql
+    tmpfs:
+      - /var/lib/mysql
     restart: always

   phpmyadmin:

Did a docker-compose down and docker-compose up -d. Stepped into the docker-compose exec -u root mysql bash and verified it was using the ramdisk. Then restarted the tests.

Alas, still slow. But this time I waited for a full benchmark. Comparison:

• OS X, Docker 17.12, civicrm-buildkit-docker, PHP 7.0, MySQL 5.7
  • Real execution time: 19m44.911s
• OS X, Nix 2.0, bknix, PHP 5.6, MySQL 5.7
  • Real execution time: 5m35s

I guess it's not quite a fair comparison because (on OSX) all docker containers run inside a Linux VM, which could be starved of resources. Checking the Docker GUI, that VM had 4 CPUs and 2 GB RAM. So I bumped up to 4 GB RAM. Shutdown/restarted all Docker services and ran the tests again. But it still took 19m36.352s.

I can't think of a simple, high-level reason for it to be so much slower. 4 CPU's and 4 GB seems pretty ample. During the last test, there was plenty of unused RAM (1.4 GB per /proc/meminfo). Virtualized CPUs can be a little slower, but you'd expect 1.2x not 4x. PHP 7 should be faster than PHP 5.6. The filesystem could be a lot slower, but I can't see how that matters once you get phpunit started. (The I/O should be mostly upfront loading *.php files and mysql requests; the former is cached in ram by php, and the latter is stored in ram.) Maybe the virtual networking adds a lot of lag to the php-mysqld communication? It's an open mystery.

Even so, most of the time folks only need to run a couple unit-tests, and the download process (especially when using images) is pretty slick, so it seems like an improvement for a typical contributor.

[chrisfromredfin]

Still not working for me, but I can confirm @totten's methodology for snag 1 works. After running the docker-compose up -d, I bash'd in and confirmed that /buildkit/.amp/services.yml did indeed still say perm_type: linuxAcl. So, trying the build, of course it failed. bash'd in again, ran amp config:set --perm_type=none and then tried the build again. Great success!

Unlike totten, I feel fairly confident I didn't have anything leftover from a previous build. I did the following to ensure a clean build:

1. stopped & removed every docker container on my system
2. deleted every docker image on my system
3. removed /build/* (which for me was just build/dmaster) from the repo folder.

(I was following this: http://blog.baudson.de/blog/stop-and-remove-all-docker-containers-and-images )

I am running currently off master (which contains just that one additional commit for always restart).

So, I can see right there on line 91 of the civicrm Dockerfile that the amp.services.yml is copied. https://github.com/michaelmcandrew/civicrm-buildkit-docker/blob/master/civicrm/Dockerfile

So... why isn't it actually being copied? Or is something else changing the setting back? After we copy it in should we then EXEC amp config:set --perm_type=none

[michaelmcandrew]

@chrisfromredfin and @totten

You missed deleting the volumes that were created by docker-compose.

We create volumes for stuff that we want to persist beyond containers and I created one for the ~/.amp directory since that contains apache vhost definitions that we want to persist.

But it also contains the ~/.amp/services.yml file. We would prefer to use the container version of this file but since they are in the same folder, this is not possible.

@totten might be an idea to 'unixify' the .amp directory so that logs, data and config are stored in seperate folders

[chrisfromredfin]

Yes, totally did. I assumed container removal did that (I'm a docker n00b but I'm getting there). So started clean again, to mimic what a newcomer would do (so I removed images, containers AND volumes now :) ), and can confirm this all worked without a hitch on my Mac!