Drupal installation and update is extremely slow

[Niklan]

Issue Type

• Support Request

Your Environment

Vagrant 1.9.3
VirtualBox 5.0.36_Ubuntur114008
ansible 2.3.0.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides
  python version = 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]

Your OS

• Linux (4.4.0-72-generic)

DISTRIB_ID=neon
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="KDE neon Developer Edition"
NAME="KDE neon"
VERSION="Developer Edition"
ID=neon
ID_LIKE="ubuntu debian"
PRETTY_NAME="KDE neon Developer Edition"
VERSION_ID="16.04"
HOME_URL="http://neon.kde.org/"
SUPPORT_URL="http://neon.kde.org/"
BUG_REPORT_URL="http://bugs.kde.org/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

Summary

When I try to vagrant up with drush make or compser build and installation, this process take a long time. Drush make is complete around 30-40 minutes, composer throw error about 1 hour working. This is very annying. But not so critical. If i disable build and installation Drupal by DrupalVM, I can copy and paste clean core and install site in 1-2 minutes.

But drush up -y executes around 60 minutes on Drupal 8. Holy... why is it so long? I bet this is problem with folder syncing, I uses default settings almost at all, NFS for synced folders and so on.

I noticed that when this commands (drush up, compser create-project) executes so long, they create (i think from archive which they download it) file by file very slow. So, it's looks like sync problem. I tried to increase memory limit and cpu's cores, but nothing changes. Everything else works very fast.

By the now it's again hanging on such operation:

My $10 DigitalOcean droplet with other projects on it do same operation for 10-20 seconds!

P.s. Example above finally finished in 90 minutes! And with error: 2006 MySQL server has gone away

[thom8]

@Niklan I'd recommend running any composer commands on the host machine, this way it also can leverage the host cache.

[Niklan]

@thom8 but whats about drush commands? Above i posted example, simple upgrade from 8.3.0 to 8.3.1 took 90 minutes! Same update on DigitalOcean, which droplet is less pewerful took 10-20 seconds, not more.

[thom8]

You should also be able to run drush commands on the host if you have it installed.

Could also try running drush in debug mode to see what the hold up is -- drush -dvy up

[thom8]

Still would recommend using drupal composer then adding DrupalVM as a dependency.

composer require --dev geerlingguy/drupal-vm

All from the host machine.

[geerlingguy]

@Niklan - A few things:

1. Have you read through the Linux setup docs? (http://docs.drupalvm.com/en/latest/getting-started/installation-linux/) - seems like you're okay there, but just want to make sure you've seen them.
2. If Drupal is mounted on a really slow mount, then it would make anything (even a normal page load on the site) super slow. Can you try using rsync or something else for the synced folder type and see if that fixes it?
3. If #2 works, you could consider doing a reverse-mounted folder.

Finally, does your computer have a hard drive or SSD? And how much RAM do you have on your computer? If it's less than a few GB, that could be an issue too.

[Niklan]

@geerlingguy

1. Yes, I installed everything, and also installed required vagrant plugins.
2. All works very fast, the problem only starts when I try to install or upgrade Drupal core. Everything else is super fast.
3. Well, I will try rsync and see what happens, and write results here.

I have HDD, but I don't think this is the main problem. For example, on dockerized drupal, which also uses vagrant inside it, drupal 8 upgrades very fast, no more than a minute. It's hard to belive that on same computer, but using drupalvm this operation is 90 times slower. I mean, this slowing only appears on Drupal VM with Drupal update and installation processes. I have 8GB RAM, and VM has 2GB of RAM, also while working I always have around of free 1-2GB RAM.

[Niklan]

I tried rsync, Drupal updates from 8.3.0 to 8.3.1 in 2-5 seconds.

I also trying to install and build Drupal with DrupalVM by Drush Make and Composer. Both works very fast now. But I have new problem now. I set:

vagrant_synced_folders:
  - local_path: ./drupal
    destination: /var/www/drupal
    type: rsync
    create: true

drupal directory created as expected, system marks it as network folder as when i using NFS, but there is nothing here, folder just empty. Files only available by vagrant ssh for now. I think this is third about reverse-mounted folder, but how can I do it?

And is it safe to change NFS to rsync on existed project? Which folder has proiority, local or in vagrant. Sying, i update same file in vagrant and local, but what happens, which will be using?

[geerlingguy]

@Niklan - Yeah, then it looks like it's definitely related to the shared folder mount. I wonder if it wasn't actually using NFS in the first place... even using native shared folders (which are horribly slow), a Drupal install takes < 5 minutes (< 1 minute if using rsync or NFS on my own computer), and something like updb / cr takes maybe ~1-2 minutes, depending on the site.

There are many articles on the Internet talking about strategies for making Vagrant performance better (e.g. https://stefanwrobel.com/how-to-make-vagrant-performance-not-suck, and some articles on my own blog)—and most recommend rsync if you can use it (it's one-way only, as you've found out—no files inside the VM get put back on your drive), or NFS if not, since it's usually much faster than the alternatives.

I haven't run a VM on a system with a spinning hard drive in a few years, so I don't know just how big a delta that would have vs an SSD, but judging by how a lot of other things are much faster on SSD, maybe there's something with your HDD that is causing the issue when using NFS or native shared folders.

One final thing: I noticed you're running VirtualBox 5.0.x—that version is not guaranteed to work well with Drupal VM (You need 5.1.x if at all possible), and has been known to have weird issues. Maybe try upgrading and see if that helps? (First do a vagrant destroy to kill the VM).

[Niklan]

@geerlingguy I tried to update VirtualBox to 5.1.20, destroyed previous box with vagrant destroy, and now I can't run new one. I've got this message:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["startvm", "46e0b557-b569-4fe6-bef4-8af490a54f0f", "--type", "headless"]

Stderr: VBoxManage: error: The virtual machine 'drupal8.dev' has terminated unexpectedly during startup with exit code 1 (0x1)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component MachineWrap, interface IMachine

[oxyc]

https://github.com/geerlingguy/drupal-vm/issues/1019#issuecomment-266720253

[Niklan]

@oxyc thank you, thats helped me

[Niklan]

Now installs starts and crash on NGINX checking.

TASK [geerlingguy.nginx : Ensure nginx is installed.] **************************
fatal: [baikalsvet]: FAILED! => {"cache_update_time": 1492784835, "cache_updated": false, "changed": false, "failed": true, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"     install 'nginx'' failed: E: Sub-process /usr/bin/dpkg returned an error code (1)\n", "stderr": "E: Sub-process /usr/bin/dpkg returned an error code (1)\n", "stderr_lines": ["E: Sub-process /usr/bin/dpkg returned an error code (1)"], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nnginx is already the newest version (1.10.3-0+xenial0).\nThe following packages were automatically installed and are no longer required:\n  python-crypto python-ecdsa python-httplib2 python-jinja2 python-markupsafe\n  python-paramiko python-pkg-resources python-setuptools python-six\n  python-yaml sshpass\nUse 'sudo apt autoremove' to remove them.\n0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.\n2 not fully installed or removed.\nAfter this operation, 0 B of additional disk space will be used.\nSetting up nginx-full (1.10.3-0+xenial0) ...\r\nJob for nginx.service failed because the control process exited with error code. See \"systemctl status nginx.service\" and \"journalctl -xe\" for details.\r\ninvoke-rc.d: initscript nginx, action \"start\" failed.\r\n\u001b[0;1;31m*\u001b[0m nginx.service - A high performance web server and a reverse proxy server\r\n   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)\r\n   Active: \u001b[0;1;31mfailed\u001b[0m (Result: exit-code) since Fri 2017-04-21 14:34:02 UTC; 4ms ago\r\n     Docs: man:nginx(8)\r\n  Process: 9891 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; \u001b[0;1;31m(code=exited, status=1/FAILURE)\u001b[0m\r\n  Process: 9888 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)\r\n\r\nApr 21 14:34:01 baikalsvet nginx[9891]: nginx: [emerg] listen() to [::]:80, ...)\r\nApr 21 14:34:01 baikalsvet nginx[9891]: nginx: [emerg] listen() to 0.0.0.0:8...)\r\nApr 21 14:34:01 baikalsvet nginx[9891]: nginx: [emerg] listen() to [::]:80, ...)\r\nApr 21 14:34:02 baikalsvet nginx[9891]: nginx: [emerg] listen() to 0.0.0.0:8...)\r\nApr 21 14:34:02 baikalsvet nginx[9891]: nginx: [emerg] listen() to [::]:80, ...)\r\nApr 21 14:34:02 baikalsvet nginx[9891]: nginx: [emerg] still could not bind()\r\nApr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;39mnginx.service: Control process exite...=1\u001b[0m\r\nApr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;31mFailed to start A high performance w...r.\u001b[0m\r\nApr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;39mnginx.service: Unit entered failed state.\u001b[0m\r\nApr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;39mnginx.service: Failed with result 'e...'.\u001b[0m\r\nHint: Some lines were ellipsized, use -l to show in full.\r\ndpkg: error processing package nginx-full (--configure):\r\n subprocess installed post-installation script returned error exit status 1\r\ndpkg: dependency problems prevent configuration of nginx:\r\n nginx depends on nginx-full (<< 1.10.3-0+xenial0.1~) | nginx-light (<< 1.10.3-0+xenial0.1~) | nginx-extras (<< 1.10.3-0+xenial0.1~); however:\r\n  Package nginx-full is not configured yet.\r\n  Package nginx-light is not installed.\r\n  Package nginx-extras is not installed.\r\n nginx depends on nginx-full (>= 1.10.3-0+xenial0) | nginx-light (>= 1.10.3-0+xenial0) | nginx-extras (>= 1.10.3-0+xenial0); however:\r\n  Package nginx-full is not configured yet.\r\n  Package nginx-light is not installed.\r\n  Package nginx-extras is not installed.\r\n\r\ndpkg: error processing package nginx (--configure):\r\n dependency problems - leaving unconfigured\r\nErrors were encountered while processing:\r\n nginx-full\r\n nginx\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "nginx is already the newest version (1.10.3-0+xenial0).", "The following packages were automatically installed and are no longer required:", "  python-crypto python-ecdsa python-httplib2 python-jinja2 python-markupsafe", "  python-paramiko python-pkg-resources python-setuptools python-six", "  python-yaml sshpass", "Use 'sudo apt autoremove' to remove them.", "0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.", "2 not fully installed or removed.", "After this operation, 0 B of additional disk space will be used.", "Setting up nginx-full (1.10.3-0+xenial0) ...", "Job for nginx.service failed because the control process exited with error code. See \"systemctl status nginx.service\" and \"journalctl -xe\" for details.", "invoke-rc.d: initscript nginx, action \"start\" failed.", "\u001b[0;1;31m*\u001b[0m nginx.service - A high performance web server and a reverse proxy server", "   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)", "   Active: \u001b[0;1;31mfailed\u001b[0m (Result: exit-code) since Fri 2017-04-21 14:34:02 UTC; 4ms ago", "     Docs: man:nginx(8)", "  Process: 9891 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; \u001b[0;1;31m(code=exited, status=1/FAILURE)\u001b[0m", "  Process: 9888 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)", "", "Apr 21 14:34:01 baikalsvet nginx[9891]: nginx: [emerg] listen() to [::]:80, ...)", "Apr 21 14:34:01 baikalsvet nginx[9891]: nginx: [emerg] listen() to 0.0.0.0:8...)", "Apr 21 14:34:01 baikalsvet nginx[9891]: nginx: [emerg] listen() to [::]:80, ...)", "Apr 21 14:34:02 baikalsvet nginx[9891]: nginx: [emerg] listen() to 0.0.0.0:8...)", "Apr 21 14:34:02 baikalsvet nginx[9891]: nginx: [emerg] listen() to [::]:80, ...)", "Apr 21 14:34:02 baikalsvet nginx[9891]: nginx: [emerg] still could not bind()", "Apr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;39mnginx.service: Control process exite...=1\u001b[0m", "Apr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;31mFailed to start A high performance w...r.\u001b[0m", "Apr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;39mnginx.service: Unit entered failed state.\u001b[0m", "Apr 21 14:34:02 baikalsvet systemd[1]: \u001b[0;1;39mnginx.service: Failed with result 'e...'.\u001b[0m", "Hint: Some lines were ellipsized, use -l to show in full.", "dpkg: error processing package nginx-full (--configure):", " subprocess installed post-installation script returned error exit status 1", "dpkg: dependency problems prevent configuration of nginx:", " nginx depends on nginx-full (<< 1.10.3-0+xenial0.1~) | nginx-light (<< 1.10.3-0+xenial0.1~) | nginx-extras (<< 1.10.3-0+xenial0.1~); however:", "  Package nginx-full is not configured yet.", "  Package nginx-light is not installed.", "  Package nginx-extras is not installed.", " nginx depends on nginx-full (>= 1.10.3-0+xenial0) | nginx-light (>= 1.10.3-0+xenial0) | nginx-extras (>= 1.10.3-0+xenial0); however:", "  Package nginx-full is not configured yet.", "  Package nginx-light is not installed.", "  Package nginx-extras is not installed.", "", "dpkg: error processing package nginx (--configure):", " dependency problems - leaving unconfigured", "Errors were encountered while processing:", " nginx-full", " nginx"]}

[geerlingguy]

@Niklan - It looks like:

Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.

Can you run those commands to check the logged output? Maybe a setting is missing or incorrect?

[yannickoo]

I just experienced the same issue and I could fix it with the help of this article.

I executed this code inside the VM here before running vagrant provision from outside again and it works 🎉

sudo systemctl stop apache2.service
sudo apt-get install nginx
sudo systemctl start apache2.service

[Niklan]

The problem with nginx was in new Drupal VM 4.4.3 version.

I create ne project with 4.4.2 and same configs, but problems in not presented here.

I upgraded VirtualBox to 5.1.20 and this is not helped me. NFS is still very slow and rsync very fast. If I stay with rsync, what the best way to work with core, connect via ssh? And how to upload files? FTP? It is not possible to set local_path as the main folder, not destinaton?

[Niklan]

If i tried to install with thouse configs:

vagrant_hostname: drupal8.dev
vagrant_machine_name: drupal

vagrant_synced_folders:
  - local_path: ./drupal
    destination: /var/www/drupal
    type: rsync
    create: true

drupal_build_composer_project: false
drupal_install_site: false

All installs, but when I SSH to /var/www/drupal, I haven't any permission

/var/www/drupal$ touch test.txt
touch: cannot touch 'test.txt': Permission denied

[Niklan]

Another try

All installs well, I copied my project to ./drupal and look how vagrant sync files. It was extremly fast, so this is not NFS problem i think. Maybe NFS is slow VM -> host, but host -> VM works without any delay.

And also, Drupal VM 4.4.3 is realy broken. It's fails on NGINX, also even apache web server is set.

Summarizing:

• NFS host to VM works without delay, so this is not the problem for the main issue. How can I test sync from VM to host?
• Drupal VM 4.4.4 looks like broken. I thinks it's better to create separate issue for that?

[oxyc]

You are correct, the current release is broken but 4.4.4 will be released soon.

For optimal performance, it's best to develop on the host like @thom8 suggested earlier. This is a slightly different setup and I think it works best with requiring Drupal VM as a composer dependency. You do this by setting up a new Drupal project, using eg drupal-composer/drupal-project as an example. Require your modules etc in the composer.json and then require Drupal VM as a dev dependency. There are docs on how to set this up.

Now that you have Drupal VM as a dependency and all the files on your host machine you can set up rsync. You could use multiple synced folder types so you get the performance of rsync for the code base, and the convenience of two-way sync from NFS for Drupal's configuration sync directory.

  - local_path: .
    destination: /var/www/drupal
    type: rsync
    create: true
    excluded_paths:
      - private
      - .git
      - web/sites/default/files
      - tmp
      # Exclude the second synced folder.
      - config/drupal
  # Use a slower but two-way sync for configuration sync directory.
  - local_path: config/drupal
    destination: /var/www/drupal/config/drupal
    type: "nfs"
    create: true

Now when you modify the codebase on your host, the files will be rsynced to the VM and read in native speed. When you export your configuration, they will be written to the NFS synced config/drupal (you need to configure the path in settings.php) and be available on your host as well.

---

This issue sounds very odd to me though. It shouldn't be that slow. Maybe we can fix it somehow... Is it equally slow if you run drush up -y within the VM when using NFS? If it is, do as @thom8 suggested, run it in debug mode and check what's holding it up drush -dvy up.

[oxyc]

I wonder if it's something you can tune in the NFS settings. I read a post that suggested using async as an NFS option. Vagrant docs also provide this as an example for linux, maybe worth trying

config.vm.synced_folder ".", "/vagrant",
  nfs: true,
  linux__nfs_options: ['rw','no_subtree_check','all_squash','async']

You can set that option for your vagrant_synced_folder like so:

vagrant_synced_folders:
  - local_path: .
    destination: /var/www/drupalvm
    type: nfs
    create: true
    options_override:
      linux__nfs_options: ['rw','no_subtree_check','all_squash','async']

[Niklan]

I don't know what happening, but Drupal VM is completly broken for me now. I can't even start new project.

Clean Drupal VM 4.4.4 folder. I just create this file config.yml with this content:

vagrant_hostname: drupalvm.dev
vagrant_machine_name: drupalvm

vagrant_synced_folders:
  - local_path: ./drupal
    destination: /var/www/drupal
    type: nfs
    create: true

drupal_build_composer_project: false
drupal_install_site: false

drupal_core_path: /var/www/drupal

Installtion and provision is always ending with error:

TASK [geerlingguy.drupal-console : Run Drupal Console init.] *******************
fatal: [drupalvm]: FAILED! => {"changed": true, "cmd": "php /usr/local/bin/drupal init", "delta": "0:00:00.024360", "end": "2017-04-24 11:43:45.218653", "failed": true, "rc": 1, "start": "2017-04-24 11:43:45.194293", "stderr": "", "stderr_lines": [], "stdout": "Could not find Drupal in the current path.", "stdout_lines": ["Could not find Drupal in the current path."]}       

Files are not syncing, they are seems do not have permissions to do that.

/var/www/drupal$ ls -la
total 0
/var/www/drupal$ touch test.txt
touch: cannot touch 'test.txt': No such file or directory

If i trying to acces any of the pages, dashboard, adminer and other I got this:

I checked sites-enabled and there seems all okay.

What is going on? Honestly it's not first time I'm trying to use Drupal VM, year ago I have same issues on diffirent system.

[oxyc]

Drupal Console installation is currently broken and you'll need to remove it from installed_extras until it's fixed upstream in the ansible-role-drupal-composer role. https://github.com/geerlingguy/drupal-vm/issues/1335

Files are not syninc, they are seems do not have permissions to do that.

I haven't seen that error before. I wonder if your host NFS installation is broken somehow. What does the VM entries look like in your host machines /etc/exports?

Did you check the Linux docs? Specifically, I'm wondering if it could be a firewall issue.

Make sure the vboxnet interfaces are not being blocked by your system firewall. For Fedora (and many flavors of Linux), check out this guide for more: Get Vagrant's NFS working under Fedora 20.

[Niklan]

Inside clean VM. Just installed.

# /etc/exports: the access control list for filesystems which may be exported
#   to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#

I haven't any firewalls enabled at all, I handle such things on router.

[oxyc]

There are no entries at all? Do you have NFS installed?

sudo apt-get install nfs-kernel-server

[oxyc]

If you do could you post the debug log from running vagrant up (no need for the provision step)

vagrant up --debug &> vagrant.log

[Niklan]

$ sudo apt-get install nfs-kernel-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
nfs-kernel-server is already the newest version (1:1.2.8-9ubuntu12.1).
The following packages were automatically installed and are no longer required:
  python-crypto python-ecdsa python-httplib2 python-jinja2 python-markupsafe python-paramiko python-pkg-resources python-setuptools python-six python-yaml sshpass
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.

[Niklan]

vagrant up --debug &> vagrant.log

https://pastebin.com/JNVZeuPE

[oxyc]

Ah that exports was from within the VM, then yes it should be empty.

What's the result of ls -la /var/www?

[Niklan]

$ ls -la /var/www
total 16
drwxr-xr-x  4 root root 4096 Apr 24 12:05 .
drwxr-xr-x 12 root root 4096 Apr 24 12:04 ..
drwxrwxr-x  2 1000 1000 4096 Apr 24 12:21 drupal
drwxr-xr-x  2 root root 4096 Apr 24 12:05 html

Files is now syncing, but I can't access any site. They return 404 not found.

F.e. drupalvm.dev - 404, adminer.drupalvm.dev - 403, dashboard 404 and so on.

[oxyc]

Well this is just getting stranger.

1. So the group shouldn't be 1000. Could you post the log from running vagrant provision? I'm specifically interested in the drupalvm.www tasks near the end.

2. touch: cannot touch 'test.txt': No such file or directory isn't a permission issue, it sounds more like an NFS issue. Do you have an encrypted home, or any special type of filesystem on your host (not ext3-4?)

3. So if you create a file on the host, does it appear in /var/www/drupal? Or in /vagrant?

4. Please post the /etc/exports from your host machine just in case there's something that sticks out. Only need the section for this VM.

[Niklan]

1. https://pastebin.com/JFTPmThg
2. No, I have single partition with EXT4 filesystem without any encryption.
3. Yes, files created on host is appear in /var/www/drupal immediately. In /vagrant I see them too, but inside /drupal folder ofc, also with config.yml and such files from root folder of the project. And currently created vagrant.log file.

4. # VAGRANT-BEGIN: 1000 5614f1c4-862b-4709-b58a-1b3710869948
   "/home/nikita/Projects/local/baikal.dev/drupal" 192.168.88.88(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,fsid=1958351571)
   "/home/nikita/Projects/local/baikal.dev" 192.168.88.88(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,fsid=3803141642)
   # VAGRANT-END: 1000 5614f1c4-862b-4709-b58a-1b3710869948

[oxyc]

For now could you remove drupalconsole as a dependency, and post the log from a full re-provision?

installed_extras:
  - adminer
  # - drupalconsole
  - drush
  - mailhog
  - pimpmylog
  - varnish

[Niklan]

https://pastebin.com/g0ziF5pw

[oxyc]

Do things work now?

[Niklan]

Yes, thank you so much. Site is copied and started.

But why Drupal Console installation failure breaks entire web-server?

I will return later to the main problem of issue, if some update coming soon to D8 core, or when I will have more free time, I will download 8.3.0 and try to update it with debug of Drush.

[oxyc]

See #133. When there's a failure, the entire provisioning process stops. Unfortunately in cases like these...

At least we now have some debugging information and know for sure that NFS is running :) When you do have time, I'll try and help more to see if we can figure out why it's so slow. On no system should it take 90min.

[Niklan]

I installed clear Drupal VM with this config.yml:

vagrant_hostname: drupal8.dev
vagrant_machine_name: drupal8

vagrant_synced_folders:
  - local_path: ./drupal
    destination: /var/www/drupal
    type: nfs
    create: true

drupal_build_composer_project: false
drupal_install_site: false

drupal_core_path: /var/www/drupal

installed_extras:
  - adminer
  # - drupalconsole
  - drush
  - mailhog
  - pimpmylog
  - varnish

Copied drupal 8.3.0 core into ./drupal folder and install the site. All was super fast. Then I vagran ssh into VM and naviagte to drupal folder cd /var/www/drupal and then runs 'drush -dvy up`.

This is debug info: https://pastebin.com/GKrMARdE

As you can see, after Calling drush_delete_dir(/var/www/drupal/drupal-8.3.1, 1) [11.96 sec, 28.92 МБ] it react at Calling is_readable(/tmp/download_file8T3IYC) [815.15 sec, 28.92 МБ] — 813 seconds for deleting directory?

And funny moment, this is failed update, site not working anymore because of PHP Fata error during update. Folder ./drupal is now looks like this:

As for me, looks like problem is not in NFS, its more about curl \ temp folder.

P.s. can I exclude .git folder from sync via NFS? I don't need it in VM.

[oxyc]

Thanks, that information is going to help a lot I bet! I'll see if I can dig something up later as I'm working on something else right now. This might be something that others have more experience with though. @geerlingguy any idea what could cause this:

Executing: wget -q --timeout=30 -O /tmp/download_file8T3IYC https://ftp.drupal.org/files/projects/drupal-8.3.1.tar.gz Calling is_readable(/tmp/download_file8T3IYC) [815.15 sec, 28.92 МБ] [debug] Calling is_writable(/home/vagrant/.drush/cache/download) [815.16 sec, 28.92 МБ] [debug] Calling rename(/tmp/download_file8T3IYC, /home/vagrant/.drush/cache/download/https---ftp.drupal.org-files-projects-drupal-8.3.1.tar.gz) [815.16 sec, 28.93 МБ] [debug] Error while sending QUERY packet. PID=26470 Statement.php:59 [815.17 sec, 28.94 МБ]

Doesn't seem like it could be be NFS because /tmp isn't even in a shared directory.

[oxyc]

Derp I misread the meaning of the debug logs a bit.

Could you try:

vagrant_synced_folders:
  - local_path: ./drupal
    destination: /var/www/drupal
    type: nfs
    create: true
    options_override:
      nfs_udp: false

https://github.com/composer/composer/issues/1787#issuecomment-31518184

[thom8]

@Niklan I would also recommend upgrading to Virtualbox 5.1.x before debugging much further, there's a number of bug fixes in the latest release.

more info here -- https://github.com/geerlingguy/drupal-vm/issues/1276#issuecomment-291017885

[Niklan]

@thom8, I'm already use latest version of virtualbox, I manually updated it. So all debug informations above from 5.1.20 version of it.

[Niklan]

@oxyc I did this, run vagran provision, and again try to update drupal from 8.3.0 to 8.3.1. There is result: https://pastebin.com/fzEPFhnk — nothing is changed.

I looked at /cat/exposts:

# VAGRANT-BEGIN: 1000 bd305377-c8ab-4cb7-b88a-53f0339482c8
"/home/nikita/Projects/local/drupal8.dev/drupal" 192.168.88.88(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,fsid=1101004324)
"/home/nikita/Projects/local/drupal8.dev" 192.168.88.88(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,fsid=3212763298)
# VAGRANT-END: 1000 bd305377-c8ab-4cb7-b88a-53f0339482c8

Seems like vagrant provision don't update mount points at all. Is there way to do this without rebuild entire VM? Look like I still have one record from the old VM, which was destroyed. I need it to remove manualy or with special NFS tools? Cuz umount -f -l PATH can't handle it.

[thom8]

@Niklan vagrant reload will reprovision the mounts.

However, if there's any unlinked config in there you might need to these remove manually.

When I've run into this before, I halt all my VMs then clear everything in /etc/exports and then vagrant up.

[thom8]

This is more likely a vagrant issue so it might be a good idea to check the vagrant issue queue too -- https://github.com/mitchellh/vagrant/issues

To check if it's isolated to DrupalVM, you could also try another project -- https://github.com/thom8/drupal8-vagrant

If you can replicate it here it's more likely an upstream issue with vagrant / vbox.

[Niklan]

@thom8 vagrant reload didn't help, then halt VM and remove lines from /etc/exports manually and then vagrant up, he is created them again, but they the same as before, don't see any references to nfs_udp: false. I thinks this is somehow must be reflected in exports.

# VAGRANT-BEGIN: 1000 bd305377-c8ab-4cb7-b88a-53f0339482c8
"/home/nikita/Projects/local/drupal8.dev/drupal" 192.168.88.88(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,fsid=1101004324)
"/home/nikita/Projects/local/drupal8.dev" 192.168.88.88(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000,fsid=3212763298)
# VAGRANT-END: 1000 bd305377-c8ab-4cb7-b88a-53f0339482c8

All the same.

I will check your vagrant envoirment later.

[thom8]

I may be wrong but I'm pretty sure with NFS you can't mount a subdirectory of another mount, at least not on the same machine.

[thom8]

Ahh I am wrong, looks like it works..

but in another project where I'm doing this it gets created as : "/path/to/mount" "/path/to/mount/subdir" 192.168.88.88 -alldirs -mapall=501:20

Not sure if that would make a difference.

Anyhow, the above doesn't mount the codebase twice so this might be the source of the performance issues.

[geerlingguy]

@Niklan - Another possibility — are you using a VPN, or are you behind a proxy server? Sometimes that can have an effect on NFS or other networking bits.

[Niklan]

@geerlingguy no, I don't using VPN at all. Files with drush download very fast, they extracting very slow via tmp folder, I don't know where is main bottleneck for this. Installation of new modules is also very fast, maybe because they have a little amount of files, but the core update or installation is always hang. It's not looks more as NFS problem, because files syncing very fast, at least host -> VM direction I can't see any delay, they syncing immediately, seems same for VM -> host syncing. It's slow down on extraction of files from archive with core inside VM, this operation doing file by file movent.

[thom8]

@Niklan as a workaround would you consider doing any code changes directly on the host rather than inside the VM.

I don't have this issue but always run drush/composer commands on the codebase from the host machine, it's generally lots faster + I can share my drush/composer cache between projects :)

[Niklan]

Hi everyone, I'm back to this issue. Now I have build new PC, which is much powerful than previous one. More RAM, more cores, more MHz and so on. But still uses HDD with 7200 speed. Problem is still present. When update in progress one of the cores always high, others just do nothing.

So this is not CPU, RAM problem. So for now I think only HDD or Vagrant (VB? NFS?) is the main problem. I sure SSD solve the problem (I will buy it later this summer, I don't see reasons for this now, all except update works extremly fast), but this is not solution for all. Anyway I don't think the hour or 1.5h for core update is normal even for HDD, extracting and copying core files done in seconds outside vagrant, sync is work momentally, but in vagrant this things work extremly slow.

I will try @thom8 vagrant box later and will see if this problem is presented here too.