Monolithic Docker provider

tomjn commented 2 years ago

Based on the work from @pentatonicfunk, I didn't have access to their github repo for this so here's a branch

Checks

[ ] I've updated the changelog.
[x] I've tested this PR
[ ] This PR is for the develop branch not the stable branch.
[ ] This PR is complete and ready for review.

update-docs[bot] commented 2 years ago

Thanks for opening this pull request! Make sure CHANGELOG.md gets updated with this change, additionally any docs that need updated can be found at https://github.com/Varying-Vagrant-Vagrants/varyingvagrantvagrants.org

GitHub
GitHub - Varying-Vagrant-Vagrants/varyingvagrantvagrants.org: The VVV docs and website
The VVV docs and website. Contribute to Varying-Vagrant-Vagrants/varyingvagrantvagrants.org development by creating an account on GitHub.

pentatonicfunk commented 2 years ago

@tomjn so i've tested this branch, and it worked fine with docker you can view the generated log here: https://gist.github.com/pentatonicfunk/ae8a5d54a568dc65eeb77c7a9545ca40 , see if anything noticeable we should fix / improve.

verified, vvv.test, phpmyadmin and one.wordpress.test

tested wp-cli

hendrawan@ubuntu-builder-dump:~/vvv-local$ vagrant ssh
vagrant@vvv:~$ cd /srv/www/wordpress-one/public_html/
vagrant@vvv:/srv/www/wordpress-one/public_html$ wp --help
vagrant@vvv:/srv/www/wordpress-one/public_html$ wp user list
+----+------------+--------------+------------------+---------------------+---------------+
| ID | user_login | display_name | user_email       | user_registered     | roles         |
+----+------------+--------------+------------------+---------------------+---------------+
| 1  | admin      | admin        | admin@local.test | 2022-09-13 16:33:32 | administrator |
+----+------------+--------------+------------------+---------------------+---------------+

tested php-debug

vagrant@vvv:/srv/www/wordpress-one/public_html$ switch_php_debugmod xdebug
 * Making sure log/php/xdebug-remote.log is readable and present
 * Enabling 'xdebug' for PHP v7.4
 * Restarting PHP FPM services so that the change takes effect
 * Restarting PHP 7.4 FastCGI Process Manager php-fpm7.4                                                                                                                                                                                                                                                                                                                                                                                                                             [ OK ]
 ✔ PHP Debug mod switch to xdebug on all available PHP versions complete.
vagrant@vvv:/srv/www/wordpress-one/public_html$ switch_php_debugmod none
Xdebug: [Log Files] File '/var/log/php/xdebug-remote.log' could not be opened.
 ✘ Disabling active PHP v7.4 debug mod: 'xdebug'
 * Restarting PHP FPM services so that the change takes effect
 * Restarting PHP 7.4 FastCGI Process Manager php-fpm7.4                                                                                                                                                                                                                                                                                                                                                                                                                             [ OK ]
 ✔ All PHP Debug mods are now turned off.

now couple of things that we might wanna improve

ntpdate
```
default: 13 Sep 16:34:08 ntpdate[29157]: Can't adjust the time of day: Operation not permitted
default:  - clock synchronisation failed
```
we might wanna exclude this for docker, as this isn't available in docker. I'll check tomorrow see if can patch the provision to skip it for docker
- mysq/debian.cnf this file https://github.com/Varying-Vagrant-Vagrants/VVV/pull/2632/files#diff-8362b383445f6f81f9502cee31bd037fccbfe2ed2fa4073050a5d908635fa36e, we discussed in the past. and i still dont know why without it docker acted funky, i will try to debug this more, see if we can remove this requirement as its not the case on other provider

VVV_DOCKER anything we should do about this ?
MacOS specific provision test Since changes also has specific MacOS commands, should i give it a test ? im thinking to use scaleway, they provide mac mini m1 server. Not sure if we can integrate it into Github Actions ?

Gist
vvv-docker-1.log
GitHub Gist: instantly share code, notes, and snippets.

GitHub
Monolithic Docker provider by tomjn · Pull Request #2632 · Varying-Vagrant-Vagrants/VVV
Based on the work from @pentatonicfunk, I didn't have access to their github repo for this so here's a branch Checks I've updated the changelog. I've tested this PR This PR is ...

tomjn commented 2 years ago

VVV_DOCKER was what I used when I tried to run VVV provisioners with a docker config to indicate if the provisioner was in a docker environment, and if so to skip certain things

MacOS specific provision test

I can do this later in the week, I've 2 days off

mysq/debian.cnf

Lets keep it, but we need to make sure it only happens when docker is the provider, I don't know what impact this will have on non-docker users

ntpdate

agreed, we can skip this if we're running on docker

pentatonicfunk commented 2 years ago

what do you think is the best to detect that we are running in docker here ? was thinking to add a provision helpers function

function vvv_is_in_docker() {
   if [ ! -f /.dockerenv ] && [ ! -f /run/.containerenv ]; then
       return 0
   fi
   return 1
}

or do you think adding env in Vagrantfile would be better something like VVV_PROVIDER ?, though im not sure it would work in trigger since it doesn't support env option in run

tomjn commented 2 years ago

A vvv_provider function or an environment variable set from the vagrant file sounds ideal at the moment

Ideally we would be able to tell from the environment itself if a step is necessary without knowing the providers, eg detecting folders or systemd rather than docker itself

On Wed, 14 Sep 2022 at 14:30, Hendrawan Kuncoro @.***> wrote:

what do you think is the best to detect that we are running in docker here ? was thinking to add a provision helpers function

function vvv_is_in_docker() { if [ ! -f /.dockerenv ] && [ ! -f /run/.containerenv ]; then return 0 fi return 1 }

or do you think adding env in Vagrantfile would be better something like VVV_PROVIDER ?, though im not sure it would work in trigger since it doesn't support env option in run

— Reply to this email directly, view it on GitHub https://github.com/Varying-Vagrant-Vagrants/VVV/pull/2632#issuecomment-1246768930, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAOLZYY3YVHREZZNZPCGA3V6HHOTANCNFSM6AAAAAAQJ2265A . You are receiving this because you were mentioned.Message ID: @.***>

tomjn commented 2 years ago

@pentatonicfunk I tried to test this, I changed provider: docker started up the docker engine, and ran vagrant up --provision. I ran into problems almost immediatley with networking and port 80 being in use.

Is there more to getting this running than the standard vagrant up? Are you trying to use sudo? To my knowledge applications can only bind to ports below 1000 on MacOS if they're the root user, but sudo vagrant is an eldritch horror to be avoided at all costs

tomjn commented 2 years ago

We also need to resolve the docker image problem, we need either a generic image or one we can generate ourselves given steps

tomjn commented 2 years ago

❯ vagrant up --provision       
__ __ __ __
\ V\ V\ V / v3.11 Ruby:2.7.6, Path:"/Users/tomjn/dev/vvvdocker"
 \_/\_/\_/  git::docker-provider(3ee80e47)

Platform: darwin19 shell:/bin/zsh vagrant-goodhosts shared_db_folder_disabled
Vagrant: v2.3.0, docker: v??

Docs:       https://varyingvagrantvagrants.org/
Contribute: https://github.com/varying-vagrant-vagrants/vvv
Dashboard:  http://vvv.test

Bringing machine 'default' up with 'docker' provider...
==> default: Running action triggers before up ...
==> default: Running trigger: VVV Setup docker local network before up...
    default: Running local: Inline script
    default: bash -c 'sudo ifconfig lo0 alias 192.168.56.4/24'
==> default: Creating and configuring docker networks...
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 80 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 80, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.

pentatonicfunk commented 2 years ago

We also need to resolve the docker image problem, we need either a generic image or one we can generate ourselves given steps

yea for sure, it should be straightforward, this is the image that i've been using: https://github.com/pentatonicfunk/docker-vagrant-ubuntu-base-images/blob/main/20.04/Dockerfile we can maybe provide under VVV instead

for the failed provision. NO i dont use sudo for vagrant @tomjn is it maybe you already have another setup that using 192.168.56.4 ? i think i had similar issue in the past, even on linux, i tested vvv with VB, then destroy, after that switch to docker, it couldn't create the network, so its not mac specific. I think destroy still residue the VB network and need to be manually removed.

GitHub
docker-vagrant-ubuntu-base-images/Dockerfile at main · pentatonicfunk/docker-vagrant-ubuntu-base-images
Vagrant compatible Ubuntu docker images. Contribute to pentatonicfunk/docker-vagrant-ubuntu-base-images development by creating an account on GitHub.

tomjn commented 2 years ago

vagrant global-status shows no active VMs or vagrant environments other than the docker environment which says preparing but never goes anywhere. I don't believe it even reaches the stage where it can do docker related things because of this. I can find no applications using port 80 on my machine

Of note I'm on an M1 machine so it's not possible to install VirtualBox, and Parallels is unlaunched

pentatonicfunk commented 2 years ago

ah yea, no VB on M1 the port mapping is part of docker command

# Docker use image.
  config.vm.provider :docker do |d|
    d.image = 'pentatonicfunk/vagrant-ubuntu-base-images:20.04'
    d.has_ssh = true
    if Vagrant::Util::Platform.platform == 'darwin19'
      # Docker in mac need explicit ports publish to access
      d.ports = [ "#{vvv_config['vm_config']['private_network_ip']}:80:80" ]
      d.ports += [ "#{vvv_config['vm_config']['private_network_ip']}:443:443" ]
      d.ports += [ "#{vvv_config['vm_config']['private_network_ip']}:3306:3306" ]
      d.ports += [ "#{vvv_config['vm_config']['private_network_ip']}:8025:8025" ]
      d.ports += [ "#{vvv_config['vm_config']['private_network_ip']}:1025:1025" ]
    end
  end

so at least we know it attempt to execute docker command. its just can't bind to 80, so the most likely culprit is 80 already bind / opened by something, but you confirmed there is none

any chance you would try this ? see if maybe it shows something

❯ lsof -i -P | grep -i "listen" | grep -i ":80"
com.docke  1396 hendrawan  233u  IPv4 0x74827ad49f2952c9      0t0  TCP subdir.wordpress.test:80 (LISTEN)
com.docke  1396 hendrawan  236u  IPv4 0x74827ad49d6627f9      0t0  TCP subdir.wordpress.test:8025 (LISTEN)

tomjn commented 2 years ago

launchd       1           root   19u  IPv6 0x1883043d49fd0667      0t0    TCP localhost:8021 (LISTEN)
launchd       1           root   20u  IPv4 0x188304507a810717      0t0    TCP localhost:8021 (LISTEN)
launchd       1           root   34u  IPv6 0x1883043d49fd0667      0t0    TCP localhost:8021 (LISTEN)
launchd       1           root   35u  IPv4 0x188304507a810717      0t0    TCP localhost:8021 (LISTEN)
prl_naptd   936           root   25u  IPv4 0x1883045072433177      0t0    TCP *:8000 (LISTEN)

pentatonicfunk commented 2 years ago

alright, let me try m1 on scaleway, see if i can replicate

pentatonicfunk commented 2 years ago

i was able to provision on M1

though it stopped in this step, probably need to enable new VM framework experimental on docker

but confirmed that docker container successfully created

tomjn commented 2 years ago

hmm the dubious ownership issue should have been resolved in 3.9, I'm still no closer to identifying what has taken port 80 and have made no further progress

pentatonicfunk commented 2 years ago

yea, that port 80 is annoying. since mojave, MacOS should allow binding lower ports without lifted privilege https://news.ycombinator.com/item?id=18302380

Now in 10.14, any application, regardless of the user's privileges, can bind to typical reserved ports, such as 22, 80, 443, etc. etc.

so shouldn't be about sudo thing

there is sample container in here: https://docs.docker.com/get-started/ that binding 80 port, see if its working on your setup. if it doesn't work, then we can pin down the issue is between docker and macos it self, not vvv / vagrant related.

docker run -d -p 80:80 docker/getting-started

MacOS Mojave 10.14 no longer enforces privileged ports | Hacker News

Docker Documentation
Orientation and setup
Get oriented on some basics of Docker and install Docker Desktop.

tomjn commented 2 years ago

I can run Altis Local Server fine which relies on a Traefik proxy container, though I can confirm those containers or any others were not running.

I can get past this error if I use sudo vagrant up but then run into another problem:

vvvdocker on  docker-provider via ⍱ v2.3.0 
❯ sudo vagrant up
┌-──────────────────────────────────────────────────────────────────────────────┐
│                                                                               │
│  ⚠ DANGER DO NOT USE SUDO ⚠                                                   │
│                                                                               │
│ ! ▄▀▀▀▄▄▄▄▄▄▄▀▀▀▄ !  You should never use sudo or root with vagrant.          │
│  !█▒▒░░░░░░░░░▒▒█    It causes lots of problems :(                            │
│    █░░█░▄▄░░█░░█ !                                                            │
│     █░░█░░█░▄▄█    ! We're really sorry but you may need to do painful        │
│  !  ▀▄░█░░██░░█      cleanup commands to fix this.                            │
│                                                                               │
│  If vagrant does not work for you without sudo, open a GitHub issue instead   │
│  In the future, this warning will halt provisioning to prevent new users      │
│  making this mistake.                                                         │
│                                                                               │
│  ⚠ DANGER SUDO DETECTED!                                                      │
│                                                                               │
│  In the future the VVV team will be making it harder to use VVV with sudo.    │
│  We will require a config option so that users can do data recovery, and      │
│  disable sites and the dashboard.                                             │
│                                                                               │
│  DO NOT USE SUDO, use ctrl+c/cmd+c and cancel this command ASAP!!!            │
│                                                                               │
└───────────────────────────────────────────────────────────────────────────────┘

 ⚠ DANGER VAGRANT IS RUNNING AS ROOT/SUDO, DO NOT USE SUDO ⚠

__ __ __ __
\ V\ V\ V / v3.11 Ruby:2.7.6, Path:"/Users/tomjn/dev/vvvdocker"
 \_/\_/\_/  git::docker-provider(3ee80e47)

Platform: darwin19 shell:/bin/sh vagrant-goodhosts shared_db_folder_disabled
Vagrant: v2.3.0, docker: v??

Docs:       https://varyingvagrantvagrants.org/
Contribute: https://github.com/varying-vagrant-vagrants/vvv
Dashboard:  http://vvv.test

Bringing machine 'default' up with 'docker' provider...
==> default: Running action triggers before up ...
==> default: Running trigger: VVV Setup docker local network before up...
    default: Running local: Inline script
    default: bash -c 'sudo ifconfig lo0 alias 192.168.56.4/24'
==> default: Creating and configuring docker networks...
Traceback (most recent call last):
    126: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/batch_action.rb:86:in `block (2 levels) in run'
    125: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:203:in `action'
    124: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:203:in `call'
    123: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/environment.rb:631:in `lock'
    122: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:217:in `block in action'
    121: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:248:in `action_raw'
    120: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
    119: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
    118: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
    117: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
    116: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
    115: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/delayed.rb:19:in `call'
    114: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
    113: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/trigger.rb:32:in `call'
    112: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
    111: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
    110: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
    109: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
    108: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
    107: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
    106: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
    105: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
    104: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
    103: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
    102: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
    101: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
    100: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     99: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine.rb:18:in `call'
     98: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     97: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
     96: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
     95: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
     94: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
     93: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
     92: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     91: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     90: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     89: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     88: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     87: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
     86: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
     85: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
     84: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
     83: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
     82: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     81: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     80: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     79: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
     78: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
     77: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
     76: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
     75: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
     74: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     73: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     72: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     71: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/provision.rb:80:in `call'
     70: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     69: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     68: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     67: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
     66: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
     65: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
     64: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
     63: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
     62: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     61: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     60: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     59: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/env_set.rb:19:in `call'
     58: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     57: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     56: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     55: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_build_dir.rb:19:in `call'
     54: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     53: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_sync_folders.rb:23:in `call'
     52: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     51: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_nfs_valid_ids.rb:14:in `call'
     50: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     49: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/synced_folders/nfs/action_cleanup.rb:25:in `call'
     48: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     47: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/synced_folder_cleanup.rb:28:in `call'
     46: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     45: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_nfs_settings.rb:15:in `call'
     44: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     43: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_networks.rb:364:in `call'
     42: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     41: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/login.rb:32:in `call'
     40: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     39: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/build.rb:24:in `call'
     38: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     37: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
     36: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
     35: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
     34: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
     33: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
     32: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     31: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
     30: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     29: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/env_set.rb:19:in `call'
     28: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     27: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_port_warning.rb:11:in `call'
     26: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     25: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_port_checker.rb:14:in `call'
     24: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     23: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/forwarded_ports.rb:28:in `call'
     22: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     21: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_forwarded_port_collision_params.rb:36:in `call'
     20: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
     19: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:44:in `call'
     18: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/environment.rb:631:in `lock'
     17: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:45:in `block in call'
     16: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:101:in `handle'
     15: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:281:in `with_forwarded_ports'
     14: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:281:in `each'
     13: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:285:in `block in with_forwarded_ports'
     12: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:124:in `block in handle'
     11: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:293:in `call_port_checker'
     10: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_forwarded_port_collision_params.rb:31:in `block in call'
      9: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:276:in `port_check'
      8: from /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/is_port_open.rb:17:in `is_port_open?'
      7: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:632:in `tcp'
      6: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:227:in `foreach'
      5: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:227:in `each'
      4: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:642:in `block in tcp'
      3: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:137:in `connect'
      2: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:56:in `connect_internal'
      1: from /opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:1214:in `connect_nonblock'
/opt/vagrant/embedded/lib/ruby/2.7.0/socket.rb:1214:in `__connect_nonblock': Can't assign requested address - connect(2) for 127.0.0.1:2222 (Errno::EADDRNOTAVAIL)

vvvdocker on  docker-provider via ⍱ v2.3.0 took 3s 
❯

tomjn commented 2 years ago

I'm getting that stack trace now wether I use sudo or not, which is frustrating. Can someone else confirm if it works for them? At this point any data is useful, MacOS or not.

pentatonicfunk commented 2 years ago

okay one thing that we haven't tried --debug and finally i was able to replicate your 80 port conflict issue you had on scaleway. But its because i was having docker run -d -p 80:80 docker/getting-started running

....
 INFO warden: Calling IN action: #<VagrantPlugins::DockerProvider::Action::PrepareNFSValidIds:0x00007f906f0625c8>
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "ps", "-a", "-q", "--no-trunc"]
....
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "network", "ls", "--format={{.Name}}"]
....
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "network", "inspect", "bridge", "host", "none"]
....
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "network", "ls", "--format={{.Name}}"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "network", "inspect", "bridge", "host", "none"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "network", "ls", "--format={{.Name}}"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
DEBUG preparenetworks: Creating network vagrant_network_192.168.56.0/24
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "network", "create", "vagrant_network_192.168.56.0/24", "--subnet", "192.168.56.0/24"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO environment: Released process lock: docker-network-create
...
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "ps", "-a", "-q", "--no-trunc"]
....
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "ps", "-a", "-q", "--no-trunc"]
...
 INFO subprocess: Starting process: ["/usr/local/bin/docker", "inspect", "6bc0b9f6a13a8bdf340c3b71c387c3743b1acc5b910717fe1bf040804a0c572e"]
DEBUG subprocess: Command not in installer, not touching env vars.
 INFO subprocess: Command not in installer, restoring original environment...
DEBUG subprocess: Selecting on IO
DEBUG subprocess: Waiting for process to exit. Remaining to timeout: 32000
DEBUG subprocess: Exit status: 0
 INFO warden: Calling IN action: #<Vagrant::Action::Builtin::HandleForwardedPortCollisions:0x00007f904f736b30>
DEBUG environment: Attempting to acquire process-lock: fpcollision
DEBUG environment: Attempting to acquire process-lock: dotlock
 INFO environment: Acquired process lock: dotlock
 INFO environment: Released process lock: dotlock
 INFO environment: Acquired process lock: fpcollision
 INFO handle_port_collisions: Detecting any forwarded port collisions...
DEBUG handle_port_collisions: Extra in use: {"80"=>#<Set: {"*"}>}
DEBUG handle_port_collisions: Remap: {}
DEBUG handle_port_collisions: Repair: true
 INFO environment: Released process lock: fpcollision
ERROR warden: Error occurred: Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 80 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 80, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.
 INFO warden: Beginning recovery process...
 INFO warden: Calling recover: #<Vagrant::Action::Builtin::HandleForwardedPortCollisions:0x00007f904f736b30>
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
ERROR warden: Error occurred: Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 80 is already in use
on the host machine.

....
ERROR vagrant: /opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:129:in `block in handle'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:285:in `block in with_forwarded_ports'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:281:in `each'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:281:in `with_forwarded_ports'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:101:in `handle'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:45:in `block in call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/environment.rb:631:in `lock'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_forwarded_port_collisions.rb:44:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_forwarded_port_collision_params.rb:36:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/forwarded_ports.rb:28:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_port_checker.rb:14:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_port_warning.rb:11:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/env_set.rb:19:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/build.rb:24:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/login.rb:32:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_networks.rb:364:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_nfs_settings.rb:15:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/synced_folder_cleanup.rb:28:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/synced_folders/nfs/action_cleanup.rb:25:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/prepare_nfs_valid_ids.rb:14:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_sync_folders.rb:23:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine_build_dir.rb:19:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/init_state.rb:15:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/provision.rb:80:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/destroy_build_image.rb:43:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_box.rb:25:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/plugins/providers/docker/action/host_machine.rb:18:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/handle_box.rb:25:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:127:in `block in finalize_action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/call.rb:53:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/trigger.rb:32:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builtin/delayed.rb:19:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/warden.rb:48:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/builder.rb:180:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `block in run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/util/busy.rb:19:in `busy'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/action/runner.rb:101:in `run'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:248:in `action_raw'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:217:in `block in action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/environment.rb:631:in `lock'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:203:in `call'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/machine.rb:203:in `action'
/opt/vagrant/embedded/gems/2.3.0/gems/vagrant-2.3.0/lib/vagrant/batch_action.rb:86:in `block (2 levels) in run'
 INFO interface: error: Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 80 is already in use
on the host machine.

To fix this, modify your current project's Vagrantfile to use another
port. Example, where '1234' would be replaced by a unique host port:

  config.vm.network :forwarded_port, guest: 80, host: 1234

Sometimes, Vagrant will attempt to auto-correct this for you. In this
case, Vagrant was unable to. This is usually because the guest machine
is in a state which doesn't allow modifying port forwarding. You could
try 'vagrant reload' (equivalent of running a halt followed by an up)
so vagrant can attempt to auto-correct this upon booting. Be warned
that any unsaved work might be lost.
Vagrant cannot forward the specified ports on this VM, since they
would collide with some other application that is already listening
on these ports. The forwarded port to 80 is already in use
on the host machine.

....
 INFO interface: Machine: error-exit ["Vagrant::Errors::ForwardPortCollision", "Vagrant cannot forward the specified ports on this VM, since they\nwould collide with some other application that is already listening\non these ports. The forwarded port to 80 is already in use\non the host machine.\n\nTo fix this, modify your current project's Vagrantfile to use another\nport. Example, where '1234' would be replaced by a unique host port:\n\n  config.vm.network :forwarded_port, guest: 80, host: 1234\n\nSometimes, Vagrant will attempt to auto-correct this for you. In this\ncase, Vagrant was unable to. This is usually because the guest machine\nis in a state which doesn't allow modifying port forwarding. You could\ntry 'vagrant reload' (equivalent of running a halt followed by an up)\nso vagrant can attempt to auto-correct this upon booting. Be warned\nthat any unsaved work might be lost."]

after stopped that getting-started container, and then docker container prone + docker network prune i was able to run provision again, but still stopped at that dubious git thing, also annoying since i was able to help setup this completely on my mac on the past and my friend's mac couple of weeks ago 😓

pentatonicfunk commented 2 years ago

so about detected dubious ownership

i found weird thing, so if i do this manually it works

root@vvv:/srv/provision/extensions/core# sudo -EH -u "vagrant" git pull origin master
From https://github.com/Varying-Vagrant-Vagrants/vvv-utilities
 * branch            master     -> FETCH_HEAD
Already up to date.

but if i put it to sh script, it doesn't

#ttt.sh
function noroot() {
  sudo -EH -u "vagrant" "$@";
}

noroot git pull origin "${BRANCH}" -q

-----

root@vvv:/srv/provision/extensions/core# bash ttt.sh
fatal: detected dubious ownership in repository at '/srv/provision/extensions/core'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/provision/extensions/core

pasting here, just in case anybody had any ideas. i need to setup my working env

Update

another interesting behavior

#ttt.sh
function noroot() {
  sudo -EH -u "vagrant" "$@";
}

#noroot ls -al
noroot git pull origin master

root@vvv:/srv/provision/extensions/core# bash ttt.sh
fatal: detected dubious ownership in repository at '/srv/provision/extensions/core'
To add an exception for this directory, call:

    git config --global --add safe.directory /srv/provision/extensions/core

BUT, enabling ls -al made it work

#ttt.sh
function noroot() {
  sudo -EH -u "vagrant" "$@";
}

noroot ls -al
noroot git pull origin master

root@vvv:/srv/provision/extensions/core# bash ttt.sh
total 24
drwxr-xr-x  32 vagrant vagrant 1024 Sep 29 22:19 .
drwxr-xr-x   3 root    root      96 Sep 29 21:34 ..
-rw-r--r--   1 vagrant vagrant   57 Sep 29 21:34 .codacy.yml
drwxr-xr-x  15 vagrant vagrant  480 Sep 29 22:16 .git
-rw-r--r--   1 vagrant vagrant   39 Sep 29 21:34 .gitignore
-rw-r--r--   1 vagrant vagrant 1594 Sep 29 21:34 CONTRIBUTING.md
-rw-r--r--   1 vagrant vagrant 1107 Sep 29 21:34 LICENSE
-rw-r--r--   1 vagrant vagrant 1380 Sep 29 21:34 README.md
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 graphviz
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 memcached-admin
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 mongodb
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 nvm
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 opcache-gui
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 opcache-status
drwxr-xr-x   3 vagrant vagrant   96 Sep 29 21:34 php
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php56
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php70
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php71
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php72
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php73
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php74
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php80
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 php81
drwxr-xr-x   5 vagrant vagrant  160 Sep 29 21:34 phpmyadmin
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 svn-folder-upgrade
drwxr-xr-x  10 vagrant vagrant  320 Sep 29 21:34 tideways
drwxr-xr-x   7 vagrant vagrant  224 Sep 29 21:34 tls-ca
drwxr-xr-x   5 vagrant vagrant  160 Sep 29 21:34 trusted-hosts
-rw-r--r--+  1 vagrant vagrant   97 Sep 29 22:19 ttt.sh
drwxr-xr-x   5 vagrant vagrant  160 Sep 29 21:34 webgrind
drwxr-xr-x   4 vagrant vagrant  128 Sep 29 21:34 wpcli-dev
drwxr-xr-x   3 vagrant vagrant   96 Sep 29 21:34 yarn
From https://github.com/Varying-Vagrant-Vagrants/vvv-utilities
 * branch            master     -> FETCH_HEAD
Already up to date.

what is happening 😿

tomjn commented 2 years ago

Is there a vagrant user in the container that also owns all the mounted folders?

I rarely run vagrant with the debug flag as it has so much unnecessary information that it makes parsing anything useful out of it very difficult.

I also don’t have the getting started container running. I had Altis local server with traefik but I shut that down then restarted the docker engine with no difference. If we can’t run traefik or other equivalent proxies then this eliminates docker + vagrant as a viable option for many people

Is forwarding the ports absolutely necessary for provisioning? Can that be a separate task?

pentatonicfunk commented 2 years ago

Is there a vagrant user in the container that also owns all the mounted folders? all dir in /srv owned by vagrant
vagrant@vvv:~$ cd /srv/
vagrant@vvv:/srv$ ls -al
total 8
drwxr-xr-x  1 root    root    4096 Sep 29 22:41 .
drwxr-xr-x  1 root    root    4096 Sep 29 22:41 ..
drwxr-xr-x  2 vagrant vagrant   64 Sep 29 21:29 certificates
drwxr-xr-x  9 vagrant vagrant  288 Sep 29 21:18 config
drwxr-xr-x  5 vagrant vagrant  160 Sep 26 15:02 database
drwxr-xr-x 13 vagrant vagrant  416 Sep 29 21:34 provision
drwxr-xr-x  5 vagrant vagrant  160 Sep 29 21:33 www
this noroot and git dubious issue didn't happen on my prev mac mini setup, no clue what happen here, could be docker side, or somewhere else

Is forwarding the ports absolutely necessary for provisioning? in the past i tried without port forward, it didn't work, i believe localhost worked, though our setup is vvv.test so its unable to understand localhost

pentatonicfunk commented 2 years ago

so turns out in my prev setup there is gitconfig in home vagrant

root@vvv:/srv/provision/extensions/core# cat /home/vagrant/.gitconfig
[safe]
    directory = /srv/provision/extensions/core
    directory = /srv/www/default/dashboard
    directory = /srv/www/default/opcache-statu
    directory = /srv/www/default/opcache-status
    directory = /srv/www/wordpress-one
    directory = /srv/www/wordpress-two
    directory = /srv/www/wordpress-eight
    directory = /srv/www/wordpress-five
    directory = /srv/www/wordpress-four
    directory = /srv/www/wordpress-multi-subdir
    directory = /srv/www/wordpress-multi-subdir-five
    directory = /srv/www/wordpress-multi-subdir-four
    directory = /srv/www/wordpress-multi-subdir-three
    directory = /srv/www/wordpress-multi-subdir-two
    directory = /srv/www/wordpress-multi-subdom
    directory = /srv/www/wordpress-multi-subdom-five
    directory = /srv/www/wordpress-multi-subdom-four
    directory = /srv/www/wordpress-multi-subdom-three
    directory = /srv/www/wordpress-multi-subdom-two
    directory = /srv/www/wordpress-nine
    directory = /srv/www/wordpress-seven
    directory = /srv/www/wordpress-six
    directory = /srv/www/wordpress-ten
    directory = /srv/www/wordpress-three
    directory = /srv/www/wordpress-two
    directory = /srv/www/site-reports

meanwhile on my current setup, there is no gitconfig

root@vvv:/srv/provision/extensions/core# ls -al /home/vagrant/.gitconfig
ls: cannot access '/home/vagrant/.gitconfig': No such file or directory

i wonder is VVV supposed to maintain this gitconfig safe directory, or is it automatically added by something else ?

tomjn commented 2 years ago

It’s not something VVV adds it may be something git is adding

tomjn commented 1 year ago

@pentatonicfunk my suspicion is that the port 80 issue is related to Parallels, which is very unfortunate if that's the case.

Is there no way to remove the port 80 forward requirement? Even if it means changing URLs to use a different port or some other manual workaround? I'd like to get this to a point where I can actually start it, even if it's not a final setup

tomjn commented 1 year ago

Noting that if I comment out the ports, it does create the container and start provisioning.

Is it possible Traefik could be used to handle the domain/routing?

tomjn commented 1 year ago

@pentatonicfunk with https://developer.hashicorp.com/vagrant/docs/provisioning/docker it should be possible to use a generic ubuntu image, run an additional step to install missing packages and add the needed vagrant users, and others

This page details how to use a dockerfile https://developer.hashicorp.com/vagrant/docs/providers/docker/basics#dockerfiles

and networking info: https://developer.hashicorp.com/vagrant/docs/providers/docker/networking

Docker - Provisioning | Vagrant | HashiCorp Developer
Docker - Provisioning | Vagrant | HashiCorp Developer
The Vagrant Docker provisioner can automatically install Docker, pull Docker containers, and configure certain containers to run on boot.

Basic Usage - Docker Provider | Vagrant | HashiCorp Developer
Basic Usage - Docker Provider | Vagrant | HashiCorp Developer
The Docker provider in Vagrant behaves just like any other provider. If you are familiar with Vagrant already, then using the Docker provider should be intuitive and simple.

Networking - Docker Provider | Vagrant | HashiCorp Developer
Networking - Docker Provider | Vagrant | HashiCorp Developer
The Vagrant Docker provider supports using the private network using the `docker network` commands.

msaggiorato commented 1 year ago

Ok, got it running on an M1 here, and did some "fancy" (not really) port mapping technique which should allow port mapping to work even if the default ones are used by parallels or something else.

What works or was fixed in this approach:

Provisioning
Docker version parser detected
Port Mapping when defaults are used.
HTTP (VVV default works)

What doesnt:

HTTPS (due to port mapping i assume)
IP Assignment does not work with Docker Desktop on mac ( see here ). That's why @pentatonicfunk had to do those complex ifconfig commands.
Due to the above, proper domain mapping without port forwarding cannot happen.

Ideas that I still don't fully know would work or fully tested:

Do not do domain mapping, and resort to lando's domains that point to 127.0.0.1?
Tweak nginx/mailhog/mariadb config to consider other ports than the native ones (80 or 443)? We would need to feed the "available ports" into the provisioner to tweak the ports used on provision time. Potentially on startup as well (in case these change across up/halts)?
Considering other docker engine providers for macos different to docker Desktop? I'd say this is highly unlikely to fix problems. Alternatives would be Podman (they have a desktop offering now here, Colima. These may or may not work due to licensing problems with Docker Desktop. That aside, they all have the same problem in MacOS that docker containers cannot properly get host accessible IPs assigned (based on what I read).
How are we doing on the Tart provider? Would work there be less conflictive than this?
Is libvirt even an option for VM management?

tomjn commented 1 year ago

How are we doing on the Tart provider? Would work there be less conflictive than this?

Not quickly at all

Is libvirt even an option for VM management?

the instructions for how to install it on MacOS are far beyond the technical capabilities of almost our entire userbase. The vagrant plugin part is fine, but even I had trouble figuring out where to find the libvirt install instructions nevermind running what I found

tomjn commented 1 year ago

What I had hoped is that we would use the same solution docker based systems use, aka a proxy such as Traefik or something based on Traefik

msaggiorato commented 1 year ago

Traefik would work, but i don't see the benefit of having traefik here. I may be missing something but afaik:

We'd need to do port mapping if port 80 is not available on the host.
Lacking the ability to have a dedicated IP address is very limiting for the current architecture, that's why I think we'd need to allow publicly exposed services to work transparently on other ports (also to avoid them getting confused), specially PHP. I think it's doable with some configuration tweaks.
Not sure how SSL certs work with ports in the mix. Should work, but never tried them.

msaggiorato commented 1 year ago

the instructions for how to install it on MacOS are far beyond the technical capabilities of almost our entire userbase. The vagrant plugin part is fine, but even I had trouble figuring out where to find the libvirt install instructions nevermind running what I found

Was just going to share a quick research I did and I also hit the same wall, was hopeful for a minute there! Hahaha, we can get vagrant to run the installation of the plugin for us, yet it's uncharted territory, so i'd avoid it.

tomjn commented 1 year ago

Was just going to share a quick research I did and I also hit the same wall, was hopeful for a minute there! Hahaha, we can get vagrant to run the installation of the plugin for us, yet it's uncharted territory, so i'd avoid it.

The plugin is trivial, it's the libvirt part that's nightmarish. Imagine if we asked people to compile virtualbox from scratch.

Traefik would work, but i don't see the benefit of having traefik here. I may be missing something but afaik:

We'd need to do port mapping if port 80 is not available on the host. Lacking the ability to have a dedicated IP address is very limiting for the current architecture, that's why I think we'd need to allow publicly exposed services to work transparently on other ports (also to avoid them getting confused), specially PHP. I think it's doable with some configuration tweaks. Not sure how SSL certs work with ports in the mix. Should work, but never tried them.

Not really because while this PR had lots of issues with Parallels connecting to the container, I had Altis Local Server and Docker compose based environments running with zero issues. The issues I had with port 80 were unique and isolated to this PR because of the hard requirement that was imposed on having a container with an IP, and the networking commands needed to make that work.

For the record, I don't care about that! None of the Altis Local server containers are directly accessible on port 80, Traefik takes care of that, and proxies everything where it needs to go. I can also see the need for a dedicated IP address for goodhosts, but who said we needed goodhosts for docker? Goodhosts is needed for Parallels Hyper-V and VirtualBox to do something that Traefik would give us for free.

These are all non-issues for Docker based environments, I don't see why we need to take that and bend over backwards to make it fit the vagrant VM model.

tomjn commented 1 year ago

I do not expect VVV in a container to use goodhosts, or have a public IP that can be connected to, those are extra requirements and I don't know where they came from.

I expect to go to vvv.test and see the VVV dashboard, and to provision a site and visit that site by its URL by typing it in a browser address bar.

msaggiorato commented 1 year ago

This PR will now try to bind port 80, 443, etc to localhost. That works fine most of the time. No traefik needed in regards to that.

I fail to see how traefik solves the hostfile needing to point domains to 127.0.0.1 or ::1, which in order to avoid having people do it we use goodhosts for it.

All that traefik will do is point domains to containers based on domain names, which in a monolithic container makes zero sense.

On altis, wp-env, vip-env, lando, etc traefik is used to point to the right php container based on domain. In our current situation with a monolithic container that would be useless if we can properly set up nginx to handle traffic on the proper domain.

The way lando works is because there's an actual public domain that always resolve to 127.0.0.1 (explained here)

tomjn commented 1 year ago

Noting that I tested the PR and it provisioned but I could not load vvv.test

tomjn commented 1 year ago

This PR will now try to bind port 80, 443, etc to localhost. That works fine most of the time. No traefik needed in regards to that.

For me it works none of the time. And in the event it can't map the default ports it has to map alternatives and that has major knock on effects, requiring changes to provisioning, and all installed WP sites, as well as breaking all the links in docs and terminal output by requiring the addition of port numbers where none was present previously. It's a complete non-starter.

In my tests, the only port that was actually mapped was 2222:22:

In the future we might want to be non-monolithic, but even in standalone setups they might just be a database container and an everything else container. Traefik or Traefik type containers solve a problem that nobody has solved in this PR, exposing a domain name on port 80 for HTTP or for HTTPS, and doing it reliably and repeatably.

I fail to see how traefik solves the hostfile needing to point domains to 127.0.0.1 or ::1, which in order to avoid having people do it we use goodhosts for it.

I do not see why this problem needs to be solved, or why we need a dedicated IP for VVV, or why the hosts file needs modifying at all.

All that traefik will do is point domains to containers based on domain names, which in a monolithic container makes zero sense.

But that's exactly what we need, even if it all points to the same container. We need a way to route traffic from a list of domains to a VVV container. Who cares how many containers, it needs to work and it needs to work reliably. The port mapping system does not work.

tomjn commented 1 year ago

If we need extra scripts to wrap around vagrant to get it working that's ok, but a dedicated IP, goodhosts, etc are not requirements, and if Traefik works then Traefik works. None of this is ideologically pure docker container based right now, so the "proper" use of traefik isn't relevant, if it solves the problem it solves the problem.

This PR currently fails on machines that have Parallels installed, it's improved but not fixed. Adding additional ports is a workaround but it's not feasible, and doesn't work on my machine.

tomjn commented 1 year ago

From my own research, using traefik type proxies becomes a lot easier if you can use docker compose, which requires a separate vagrant plugin. Given we plan to implement a CLI I think it's reasonable for a vvv-docker.sh or equivalent as a stopgap to work around that and make interacting with docker easier. I can see a certain someone might have tried to dynamically write a dockerfile in the distant past, though we both know better now :)

Reliability and urgency are the sides of the triangle I'm interested in right now, making things ideal and well formed comes later. I can very much see ourselves using a generic MariaDB container in the future, or PHP containers for each version or site, but until I can load vvv.test and see the dashboard there's not much point thinking too hard about that yet

msaggiorato commented 1 year ago

Ok, now the ports should be working, but goodhosts is not working as expected.

There's an issue with Vagrant that prevents it from using the port 80 and 443 on the host if there's other containers using them, even if off. Fix is already merged (2 days ago) but not sure when a new version of docker will be released. Until then, i've implemented a patch in our vagrantfile that let the port selection to continue.

Goodhosts is not working because it's expecting the vm to have an IP set, and in docker we don't. I've forced the configuration to be set to 127.0.0.1, but it's just ignoring this unless the VM has an IP.

Will check ways around it.

msaggiorato commented 1 year ago

Update here: Vagrant 2.3.5 released with the above mentioned issue solved https://github.com/hashicorp/vagrant/blob/v2.3.5/CHANGELOG.md

GitHub
vagrant/CHANGELOG.md at v2.3.5 · hashicorp/vagrant
Vagrant is a tool for building and distributing development environments. - vagrant/CHANGELOG.md at v2.3.5 · hashicorp/vagrant

msaggiorato commented 1 year ago

Did a PR to the goodhosts extension so that the 127.0.0.1 ip is actually pointed by goodhosts.

After that is merged, vvv provisions and loads when entering http://vvv.test

msaggiorato commented 1 year ago

Ok, docker provisioner works as of today with the last goodhosts update on this PR.

tomjn commented 1 year ago

Noting there has been no change with this PR despite the new commits and new goodhosts release:

only a single port is mapped for SSH, no HTTP/HTTPS ports are mapped
Docker desktop indicates that no attempt was made to map the relevant ports
it still depends on a docker image that isn't listed on docker hub with no inline documentation for where it comes from or how to build it, or what it does
there are numerous issues in the provisioner, both default sites failed to provision due to Nginx problems, it still tries to synchronise clocks which isn't possible, errors from attempting to restart services on halt, etc

Lando/VIP's proxy container has these port mappings in Docker desktop, and Altis Local servers proxy container has something similar:

This is VVV's:

Also not seeing the docker environment variable used in provisioners to do conditionals listed on that screen

A quick check of lsof indicates nothing is listening on port 80, and output from vagrant shows no attempts to map ports, so this implies the functions in the vagrant file being used to check the ports are not working as they should be.

Given that vagrant allows us to specify a build dir with a docker file, I don't see the need for the custom container image when we can put a docker file in a subfolder, but more than that, vagrant supports docker-compose. Why not write out a docker compose file and use Traefik? Throw away all the problems with IP assignment and port mapping and do it the normal way that works for everybody else?

tomjn commented 1 year ago

Switching to this mapped the ports:

    d.ports =  [ "80:80" ]
    d.ports += [ "443:443" ]
    d.ports += [ "3306:3306" ]
    d.ports += [ "8025:8025" ]

however I had to destroy the container and recreate it for it to apply, using vagrant 2.3.7

tomjn commented 1 year ago

With that change I can load the dashboard, the 2 sites fail to provision though with Nginx issues though

tomjn commented 1 year ago

It looks like the Nginx config didn't have proper search replacements, here's the vvv-nginx.conf it tried to install:

server {
    listen       80;
    listen       443 ssl http2;
    server_name  {vvv_hosts};
    root         "{vvv_path_to_site}{vvv_public_dir}";

    # Nginx logs
    error_log    "{vvv_path_to_site}/log/nginx-error.log";
    access_log   "{vvv_path_to_site}/log/nginx-access.log";

    # This is needed to set the PHP being used
    set          $upstream {upstream};

    # Enable server push if SSL/HTTP2 is being used for link preload headers
    http2_push_preload on;

    {vvv_tls_cert}
    {vvv_tls_key}

    # Nginx rules for WordPress, rewrite rules, permalinks, etc
    include      /etc/nginx/nginx-wp-common.conf;

    {{LIVE_URL}}

    location ~* \.(css|eot|gif|ico|jpeg|jpg|js|png|svg|tiff|tiff|ttf|webp|woff|woff2)$ {
        expires 100d;
    }
}

There's lots of unreadable temporary files in the provision folder too:

the log suggests that sed couldn't open the file:

[0;38;5;2m ▷ Running the [0m[1m[0;38;5;5m'site-wordpress-one'[21m[0;38;5;2m provisioner...[0m[0m[0m
 * Setting the default PHP CLI version ( 7.4 ) for this site
Now using node v14.21.3 (npm v6.14.18)
[0m[39m[2m * Pulling down the master branch of https://github.com/Varying-Vagrant-Vagrants/custom-site-template.git[21m[0m
 * Updating wordpress-one provisioner repo in /srv/www/wordpress-one (https://github.com/Varying-Vagrant-Vagrants/custom-site-template.git, master)
 * Any local changes not present on the server will be discarded in favor of the remote branch
 * Checking that remote origin is https://github.com/Varying-Vagrant-Vagrants/custom-site-template.git
 * Fetching origin master
From https://github.com/Varying-Vagrant-Vagrants/custom-site-template
 * branch            master     -> FETCH_HEAD
 * performing a hard reset on origin/master
HEAD is now at 9f03c75 Update README.md
 * Updating provisioner repo complete
 * Adding domains to the virtual machine's /etc/hosts file...
 * Adding hosts for the site to the VM hosts file
   - Added one.wordpress.test from /vagrant/config.yml
 * Searching for a site template provisioner, vvv-init.sh
 * Found vvv-init.sh at /srv/www/wordpress-one/provision/vvv-init.sh
 * Custom site template provisioner wordpress-one - downloads and installs a copy of WP stable for testing, building client sites, etc
 * Creating database 'wordpress-one' (if it's not already there)
 * Granting the wp user priviledges to the 'wordpress-one' database
 * DB operations done.
 * Setting up the log subfolder for Nginx logs
 * Creating the public folder at 'public_html' if it doesn't exist already
 * Install type is 'single'
Notice: Undefined index: HTTP_HOST in /srv/www/wordpress-one/public_html/wp-includes/functions.php on line 6095
 * WordPress is present but isn't installed to the database, checking for SQL dumps in wp-content/database.sql or the main backup folder.
 * Installing WordPress
 * Installing using wp core install --url="one.wordpress.test" --title="one.wordpress.test" --admin_name="admin" --admin_email="admin@local.test" --admin_password="password"
Success: WordPress installed successfully.
 * WordPress was installed, with the username 'admin', and the password 'password' at 'admin@local.test'
 * Copying the sites Nginx config template
 * Using the default vvv-nginx-default.conf, to customize, create a vvv-nginx-custom.conf
 * Applying public dir setting to Nginx config
sed: couldn't open temporary file /srv/www/wordpress-one/provision/sed1xsKca: Permission denied
sed: couldn't open temporary file /srv/www/wordpress-one/provision/sedRul2Ut: Permission denied
 * Adding constant 'WP_DEBUG' with value 'True' to wp-config.php
Success: Updated the constant 'WP_DEBUG' in the 'wp-config.php' file with the raw value 'True'.
 * Adding constant 'WP_DEBUG_LOG' with value 'True' to wp-config.php
Success: Updated the constant 'WP_DEBUG_LOG' in the 'wp-config.php' file with the raw value 'True'.
 * Adding constant 'WP_DISABLE_FATAL_ERROR_HANDLER' with value 'True' to wp-config.php
Success: Updated the constant 'WP_DISABLE_FATAL_ERROR_HANDLER' in the 'wp-config.php' file with the raw value 'True'.
 * Site Template provisioner script completed for wordpress-one
[0m[39m[2m * sourcing of vvv-init.sh reported success[21m[0m
[0m[39m[2m * VVV is adding an Nginx config from /srv/www/wordpress-one/provision/vvv-nginx.conf[21m[0m
nginx: [warn] the "listen ... http2" directive is deprecated, use the "http2" directive instead in /etc/nginx/custom-sites/vvv-wordpress-one-60730e70.conf:3
nginx: [warn] the "http2_push_preload" directive is obsolete, ignored in /etc/nginx/custom-sites/vvv-wordpress-one-60730e70.conf:15
nginx: [emerg] unexpected "{" in /etc/nginx/custom-sites/vvv-wordpress-one-60730e70.conf:23
nginx: configuration file /etc/nginx/nginx.conf test failed
[0;38;5;9m ! Installing an Nginx config failed! VVV tried to install vvv-wordpress-one-60730e70.conf into sites from /tmp/vvv-site-IMkTb but a syntax test with sudo nginx -t failed![0m[0m
[0;38;5;9m ! VVV is now deleting the config to avoid further breakage[0m[0m
[0;38;5;3m ! This sites nginx config had problems, it may not load. Look at the above errors to diagnose the problem[0m[0m
[0m[39m[2m ! VVV will now continue with provisioning so that other sites have an opportunity to run[21m[0m
[0;38;5;2m * Installed vvv-wordpress-one-60730e70.conf[0m[0m
[0;38;5;2m ✔ The [0m[1m[0;38;5;5m'site-wordpress-one'[21m[0;38;5;2m provisioner completed in [0m[0m[1m[0;38;5;5m3[21m[0;38;5;2m seconds.[0m[0m[0m

tomjn commented 1 year ago

The -i parameter is the problem, the temporary files it makes aren't writable in certain filesystems, this is fixed in v4.8 of sed, but we should look into an alternative such as in memory replacement instead, or explicit temporary files

msaggiorato commented 1 year ago

The -i parameter is the problem, the temporary files it makes aren't writable in certain filesystems, this is fixed in v4.8 of sed, but we should look into an alternative such as in memory replacement instead, or explicit temporary files

Hmm, that problem is common for me in volumes with Docker on Mac. I always have to go with explicit temp files, which makes all commands more complicated to write.

Switching to this mapped the ports:
    d.ports =  [ "80:80" ]
    d.ports += [ "443:443" ]
    d.ports += [ "3306:3306" ]
    d.ports += [ "8025:8025" ]
however I had to destroy the container and recreate it for it to apply, using vagrant 2.3.7

The need to destroy and recreate the container should be on the docker provider for Vagrant.

I've implemented the "magic" (not really, as it didn't work for you) port mapping to avoid conflicts with other containers that may be using that port already. We can remove the function that is not used anymore (if you didn't remove it already) and just force it to collide with anything else you may have running.

tomjn commented 1 year ago

I've implemented the "magic" (not really, as it didn't work for you) port mapping to avoid conflicts with other containers that may be using that port already. We can remove the function that is not used anymore (if you didn't remove it already) and just force it to collide with anything else you may have running.

Honestly I'm happy to just document it as that's what all the other docker local environments do, fully stop your other environments before starting ours is very common to see.

In the long run it might be nice to explore reusing their Traefik proxies but for now this is reasonable, and more user friendly than switching ports

Varying-Vagrant-Vagrants / VVV

Monolithic Docker provider #2632

Checks

Update