Open columbian-chris opened 3 weeks ago
So this is an issue in vagrant itself rather than VVV, I encountered it and made a PR upstream so a new release of vagrant should fix this ( 2.4.2 most likely but it hasn't been released yet ).
As for how I resolved this, it involved removing docker networks, docker network ls
will list all the networks available and their network IDs. Use docker network prune
to auto-remove unused networks and try the vagrant command again. If that doesn't help, remove anything vagrant related that's listed in docker network ls
using docker network rm
and try again.
Note that this only impacts the Docker provider, other providers are unaffected. Also remember that vagrant will always stick to the provider of the current VM, even if you change your mind. So if you start with a docker provider, you'll need to destroy that VM before you can switch to Parallels/etc
And then finally attempting to provision with VMWare via vagrant up --provision --provider="vmware_desktop" resulted with this:
I'm happy VMWare got at least that far, support for VMWare is spotty at best though, nobody working on VVV uses it but if we try and deprecate or remove support people complain as they still use it. A full copy of the provisioner log for VMWare would be helpful though, happy to try and fix issues with VMWare but it might mean you need to help test since I can't verify a fix works at my end.
Trying to provision with Parallels via vagrant up --provision --provider="parallels" resulted in this:
Those look like the VM has DNS issues, configuring the VM internally to use a 3rd party DNS server such as 1.1.1.1
helps here, I've tried to automate this in https://github.com/Varying-Vagrant-Vagrants/VVV/pull/2701 but it didn't work, manually setting up the DNS only needs to happen once though, and you can use any generic Ubuntu guide for the terminal to do it
A full copy of the provisioner log for VMWare would be helpful though, happy to try and fix issues with VMWare but it might mean you need to help test since I can't verify a fix works at my end.
So this is an issue in vagrant itself rather than VVV, I encountered it and made a PR upstream so a new release of vagrant should fix this ( 2.4.2 most likely but it hasn't been released yet ).
Do you maybe have a link to your PR so I can ask the vagrant folks if that's going to be released here at some point soon? Is it this issue? I wonder if doing a bleeding edge build of vagrant from source would have this modification included?
As for how I resolved this, it involved removing docker networks,
docker network ls
will list all the networks available and their network IDs. Usedocker network prune
to auto-remove unused networks and try the vagrant command again.
Unfortunately, docker network prune
didn't change anything from what docker network ls
listed, probably because the provision didn't get far enough to fire up a VM.
NETWORK ID NAME DRIVER SCOPE
EXAMPLE1234A bridge bridge local
EXAMPLE1234B host host local
EXAMPLE1234C none null local
Trying to provision with Parallels via vagrant up --provision --provider="parallels" resulted in this:
Those look like the VM has DNS issues, configuring the VM internally to use a 3rd party DNS server such as
1.1.1.1
helps here...
Tried reprovisioning and got stuck at a weird mysql root password error.
I reprovisioned again and didn't get the error trying to connect to mirror.rackspace.com. Everything looks good on the surface but I can't access any sites in a web browser, including vvv.test
. At least I could get it to fire up without any errors.
Tried reprovisioning and got stuck at a weird mysql root password error...
Surprising but I see rackspace have more issues with their mariadb mirror again in your log output >_<
W: Failed to fetch https://mirror.rackspace.com/mariadb/repo/10.5/ubuntu/dists/focal/main/source/Sources Could not connect to mirror.rackspace.com:443 (166.78.229.131). - connect (111: Connection refused) [IP: 166.78.229.131 443]
Can you check https://mariadb.gb.ssimn.org/repo/10.5/ubuntu and let me know how quickly that loads for you? Is it super slow or is it a reasonable loading time? I'm assuming from the username you're in an American timezone?
The network check used to halt provisioning but it seems that some people could provision even with it failing, we had someone pop up with rackspace Apt issues a few days ago which resolved on their own.
Do you maybe have a link to your PR so I can ask the vagrant folks if that's going to be released here at some point soon? Is it https://github.com/hashicorp/vagrant/issues/13371?
That's the issue, my PR to fix it got merged but looking at what was already there someone else had tried fixing it for another reason and that hadn't been released either.
https://github.com/hashicorp/vagrant/issues/13339#issuecomment-1906805586 seems to diagnose the issue better and reveal the root causes, and also implies downgrading docker to <v25 might also be a solution though I'm loathe to do that. You could also apply the change directly to the vagrant gem ruby files which is how I debugged the issue though that's more of a hack ( albeit one that gets stuff working )
Given 2.4.1 was in January it's been quite a while for a 2.4.2 release
I wonder if doing a bleeding edge build of vagrant from source would have this modification included?
Their release page should have a 2.4.2 dev build so you shouldn't have to build from source, here's one from July https://github.com/hashicorp/vagrant/releases/tag/2.4.2.dev%2B000038-26a1ff10
GitHubCustom Vagrant build on main at 26a1ff10
For VMWare this was what failed the provisioning during the Nginx package install:
Setting up php8.2-bcmath (8.2.22-1+ubuntu20.04.1+deb.sury.org+1) ...
Creating config file /etc/php/8.2/mods-available/bcmath.ini with new version
Setting up nginx (1.27.1-1~focal) ...
chown: changing ownership of '/var/log/nginx/access.log': Operation not permitted
dpkg: error processing package nginx (--configure):
installed nginx package post-installation script subprocess returned error exit status 1
Setting up libdrm-radeon1:arm64 (2.4.107-8ubuntu1~20.04.2) ...
Setting up libglvnd0:arm64 (1.3.2-1~ubuntu0.20.04.2) ...
Setting up php8.2-opcache (8.2.22-1+ubuntu20.04.1+deb.sury.org+1) ...
Currently we mount that folder and others like this:
# The VMware Provider does not understand "dmode"/"fmode" in the "mount_options" as
# those are specific to Virtualbox. The folder is therefore overridden with one that
# uses corresponding VMware mount options.
config.vm.provider :vmware_desktop do |_v, override|
override.vm.synced_folder 'www/', '/srv/www', owner: 'vagrant', group: 'www-data', mount_options: ['umask=002']
override.vm.synced_folder 'log/memcached', '/var/log/memcached', owner: 'root', create: true, group: 'root', mount_options: ['umask=000']
override.vm.synced_folder 'log/nginx', '/var/log/nginx', owner: 'root', create: true, group: 'root', mount_options: ['umask=000']
override.vm.synced_folder 'log/php', '/var/log/php', create: true, owner: 'root', group: 'root', mount_options: ['umask=000']
override.vm.synced_folder 'log/provisioners', '/var/log/provisioners', create: true, owner: 'root', group: 'root', mount_options: ['umask=000']
I wonder if it's the owner/group or the umask that needs changing for that. Also curious if VMWare is running the Arm64 box or if it's using x86 + Rosetta 2 or transpiling on the fly
I wonder if it's the owner/group or the umask that needs changing for that.
I also tried changing the owner and group to vagrant
(the umasks still set to 777
) and this was the result of my provision:
Also curious if VMWare is running the Arm64 box or if it's using x86 + Rosetta 2 or transpiling on the fly.
Here's what Activity Monitor is reporting:
I believe vmrest
, VMware Fusion Applications Menu
, and VMware Fusion Applications Menu Helper
are running Rosetta 2 since the "Kind" listed is Intel
.
Can you check https://mariadb.gb.ssimn.org/repo/10.5/ubuntu and let me know how quickly that loads for you? Is it super slow or is it a reasonable loading time? I'm assuming from the username you're in an American timezone?
I'm loaded the URL at either ~650-800 ms or ~170 ms (assuming these requests are cached) in a Chrome browser. Yes, I am in the Pacific time zone on the west coast of the United States.
Their release page should have a 2.4.2 dev build so you shouldn't have to build from source, here's one from July https://github.com/hashicorp/vagrant/releases/tag/2.4.2.dev%2B000038-26a1ff10
How do I upgrade vagrant to the 2.4.2 build with this .deb
file (vagrant_2.4.2.dev-1_amd64.deb
)?
GitHubCustom Vagrant build on main at 26a1ff10
I also tried changing the owner and group to vagrant (the umasks still set to 777) and this was the result of my provision:
That change helped, have you tried doing that to the other mounts?
How do I upgrade vagrant to the 2.4.2 build with this .deb file (vagrant_2.4.2.dev-1_amd64.deb)?
The .deb
being for Ubuntu/Debian Linux you'd probably need to build from source still, but given most of it is ruby files the file that needs changing is still a plaintext file somewhere on your machine that can be edited ( /opt/vagrant/embedded/gems/gems/vagrant-2.4.1/plugins/providers/docker/driver.rb
)
I also tried changing the owner and group to vagrant (the umasks still set to 777) and this was the result of my provision:
That change helped, have you tried doing that to the other mounts?
I changed all owner/group fields in the Vagrantfile
to vagrant
instead of root
and all references to file permissions to 777
. Here's the resulting provision log:
How do I upgrade vagrant to the 2.4.2 build with this .deb file (vagrant_2.4.2.dev-1_amd64.deb)?
The
.deb
being for Ubuntu/Debian Linux you'd probably need to build from source still, but given most of it is ruby files the file that needs changing is still a plaintext file somewhere on your machine that can be edited (/opt/vagrant/embedded/gems/gems/vagrant-2.4.1/plugins/providers/docker/driver.rb
)
I edited that driver file (saved myself a lot of time going through the build process) with the edit you mentioned and reprovisioned my docker provision and it appears to have booted normally. This looks like the route to take. You have been such a giant help, @tomjn; thank you so sincerely.
One issue did come up when I fire up the box:
==> default: Running trigger: VVV Post-Reload...
default: Running: inline script
default: * Reinit /etc/hosts
default: * Restarting Nginx
default: * Restarting nginx nginx
default: start-stop-daemon: warning: failed to kill 9744: No such process
default: ...done.
default: * Restarting MariaDB
default: * Stopping MariaDB database server mariadbd
default: ...done.
default: * Starting MariaDB database server mariadbd
default: ...done.
default: * Restarting PHP-FPM
default: * Restarting PHP 8.2 FastCGI Process Manager php-fpm8.2
default: ...done.
default: * Restarting Memcache
default: start-stop-daemon: warning: failed to kill 10102: No such process
default: Restarting memcached: memcached_default.
default: * Restarting Mailhog
default:
default: ✧ ▄▀▀▀▄▄▄▄▄▄▄▀▀▀▄ ✧ Thanks for __ __ __ __
default: ✧█▒▒░░░░░░░░░▒▒█ using \ V\ V\ V /
default: ✧ █░░█░░░░░█░░█ ✧ \_/\_/\_/
default: ▄▄ █░░░▀█▀░░░█ ▄▄✧
default: █░░█ ▀▄░░░░░░░▄▀ █░░█ Vagrant Up has finished! Visit http://vvv.test
default: ──────────────────────────────────────────────────────────────────────
It has an issue restarting a couple of services. When I SSH into the box and try to reload any process, I get this notice that systemd
isn't running:
> systemctl status nginx
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
Does this box use something besides systemd
?
I provisioned a fresh VVV instance with Docker on a Mac running on an Intel processor and it provisioned fine after making the same change to the driver.rb
file, but when I SSH'ed into the box I got the same error:
vagrant@vvv:~$ systemctl reload nginx
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
The Vagrantfile
refers to a pentatonicfunk/vagrant-ubuntu-base-images
box, but I couldn't find the source box online to inspect it.
that won't be a problem with the image since the image doesn't come with Nginx, that image comes from https://github.com/pentatonicfunk/docker-vagrant-ubuntu-base-images it's goal is to get a base container image we can run provisioners in.
https://github.com/Varying-Vagrant-Vagrants/VVV/pull/2687 tries to bundle the dockerfile so we don't need the dependency but it takes longer to provision as it has to build the image each time. It's unlikely this has a docker based solution and it's more likely that Nginx inside the VM is not installed properly for unknown reasons
GitHubVagrant compatible Ubuntu docker images. Contribute to pentatonicfunk/docker-vagrant-ubuntu-base-images development by creating an account on GitHub.
You won't find it on Hashicorps box catalog because it's a container image not a box:
https://hub.docker.com/r/pentatonicfunk/vagrant-ubuntu-base-images
So what now? If nothing else, I thought I could have at least gotten a VVV box to run on Parallels on an Apple Silicon processor.
I mean, I guess I could still use the Docker box for testing/development without the ability to control system processes with systemd
, although not preferred obviously.
Most VVV users have no idea what systemd is, nor should you need to. Nginx should start and stop without any intervention. Is Nginx installed inside the VM and available in path? Have you tried instead using the service
command? Most containers don't seem to use systemd at all
As for Parallels I lost track of where you got up to in the above, I thought you ran into a DNS issue, resolved it then had root password issue that you fixed
https://github.com/Varying-Vagrant-Vagrants/VVV/pull/2728 should swap out the rackspace servers and upgrade to MariaDB 10.11
These commands output should prove interesting if not outright useful:
sudo service nginx start
sudo nginx -t
As for Parallels I lost track of where you got up to in the above, I thought you ran into a DNS issue, resolved it then had root password issue that you fixed
It'll boot fine, but nothing comes up for vvv.test
or any of the provisioned sites. When I ping vvv.test
, I get 192.168.56.4
, which I can reach the VVV dashboard by accessing in a browser. I looked at my hosts file on my computer and the entries that were added appear correct to me:
192.168.56.4 vvv.test one.wordpress.test two.wordpress.test
10.211.55.5 vvv-parallels-attempt-default-1724804344010-90665.shared vvv-parallels-attempt-default-1724804344010-90665 #prl_hostonly shared
The 10.211.55.5
address also listed in the hosts file also brings up the VVV dashboard for me (I'm not sure what this additional hosts entry is used for). I also discovered I can access the VVV dashboard from vvv.local
but still none of the links to the sites work. I set a custom IP using the private_network_ip
option of the VVV config file just to see if that would change anything, but with no avail.
What do you get instead for vvv.test? Sometimes duplicates from older provisions can cause issues, do the site provisioner logs say anything about their nginx configs?
What do you get instead for vvv.test?
Just an ERR_CONNECTION_REFUSED error:
Do the site provisioner logs say anything about their nginx configs?
Here are last Parallels provision logs (I don't see any nginx errors):
hmm usually that happens if you've still got left over entries in the hosts file from another VVV with a different IP
hmm usually that happens if you've still got left over entries in the hosts file from another VVV with a different IP
I agree, which is why it's so odd that there isn't. I'm giving up on Parallels for now and continuing my focus on Docker.
It still would be so much better of an experience with my Docker setup if I could get systemd
to work. I've got several software packages that include installation instructions that require systemd
commands so I'm unable to use them, which were packages that I used regularly before I switched to an Apple processor and couldn't utilize VirtualBox anymore. 😕
what packages are those? Systemd on a docker container is rare, not to say you couldn't install it manually but that isn't ideal
Sure, I'll give you one I failed to get running on my Docker-provisioned box: Elasticsearch. After installing, it couldn't find the service with service elasticsearch start
and I'm trying to use that command as an alternative to any systemctl
utility command.
I haven't extensively fired up a lot of Docker containers so I didn't realize this was a rarity. Why wouldn't Systemd work in that environment? Is there possibly an alternative box that runs Systemd properly I could base my provision off of? It is clearly installed in the box because it wouldn't fire back with that kind of error if it wasn't, it just happens to not be working properly. Reinstalling Systemd didn't make it work. I don't know what alternative to Systemd is running; I don't see grub or upstart installed on the box.
Let me ask you this, @tomjn - if you happen to be running VVV on an Apple M-series processor, what setup are you using?
I did use Parallels but I'm currently on a docker provisioned instance. MacOS Sonoma 14.5/M1 Pro/Docker 27.1.1
Docker has been my most successful attempt in an arm64 processor environment. I would have used Parallels if I could have gotten it to work. I never could find a way to install Elasticsearch in this environment, which something I have always been able to do in VVV running on virtualbox.
Now I'm getting a provision error via docker. It seems it can't find this package:
Err:17 http://ppa.launchpad.net/pitti/systemd/ubuntu focal Release
404 Not Found [IP: 185.125.190.80 80]
[0;38;5;2m ▷ Running the [0m[1m[0;38;5;5m'main'[21m[0;38;5;2m provisioner...[0m[0m[0m
[0m[39m[2m ▷ Running [0m[1m[0;38;5;5minit[21m[0m[39m[2m hook[21m[0m
[0m[39m[2m * Bash profile setup and directories.[21m[0m
[0m[39m[2m * Reloading SSH Daemon[21m[0m
* Reloading OpenBSD Secure Shell server's configuration sshd
...done.
[0m[39m[2m * checking Ubuntu version[21m[0m
[0;38;5;2m ✔ Finished [0m[1m[0;38;5;5minit[21m[0;38;5;2m hook in [0m[0m[1m[0;38;5;5m0s[21m[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://ppa.launchpadcontent.net[0m with wget -q --spider --timeout=5 --tries=3 https://ppa.launchpadcontent.net[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://ppa.launchpadcontent.net[0m[0;38;5;2m detected[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://wordpress.org[0m with wget -q --spider --timeout=5 --tries=3 https://wordpress.org[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://wordpress.org[0m[0;38;5;2m detected[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://github.com[0m with wget -q --spider --timeout=5 --tries=3 https://github.com[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://github.com[0m[0;38;5;2m detected[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://raw.githubusercontent.com[0m with wget -q --spider --timeout=5 --tries=3 https://raw.githubusercontent.com[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://raw.githubusercontent.com[0m[0;38;5;2m detected[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://getcomposer.org[0m with wget -q --spider --timeout=5 --tries=3 https://getcomposer.org[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://getcomposer.org[0m[0;38;5;2m detected[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://deb.nodesource.com[0m with wget -q --spider --timeout=5 --tries=3 https://deb.nodesource.com[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://deb.nodesource.com[0m[0;38;5;2m detected[0m[0m
[0m[39m[2m * Testing network connection to [0m[4;38;5;3mhttps://mirror.rackspace.com[0m with wget -q --spider --timeout=5 --tries=3 https://mirror.rackspace.com[21m[0m
[0;38;5;2m * Successful Network connection to [0m[4;38;5;3mhttps://mirror.rackspace.com[0m[0;38;5;2m detected[0m[0m
[0;38;5;2m * Network checks succeeded[0m[0m
[0m[39m[2m * Apt package install pre-checks[21m[0m
[0m[39m[2m ▷ Running [0m[1m[0;38;5;5mbefore_packages[21m[0m[39m[2m hook[21m[0m
[0m[39m[2m * Setting up MySQL configuration file links...[21m[0m
[0m[39m[2m * mysql group exists[21m[0m
[0m[39m[2m * mysql user present and has uid 9001[21m[0m
[0m[39m[2m * Copying /srv/provision/core/mariadb/config/vvv-core.cnf to /etc/mysql/conf.d/vvv-core.cnf[21m[0m
[0m[39m[2m * Copying PHP configs[21m[0m
[0m[39m[2m * Checking supplementary PHP configs[21m[0m
[0;38;5;2m ✔ Finished [0m[1m[0;38;5;5mbefore_packages[21m[0;38;5;2m hook in [0m[0m[1m[0;38;5;5m0s[21m[0m[0m
[0m[39m[2m * Registering apt keys[21m[0m
[0m[39m[2m ▷ Running [0m[1m[0;38;5;5mregister_apt_keys[21m[0m[39m[2m hook[21m[0m
[0m[39m[2m * Replacing expired Nginx signing key...[21m[0m
OK
[0;38;5;2m ✔ Finished [0m[1m[0;38;5;5mregister_apt_keys[21m[0;38;5;2m hook in [0m[0m[1m[0;38;5;5m1s[21m[0m[0m
[0m[39m[2m * Registering apt sources[21m[0m
[0m[39m[2m ▷ Running [0m[1m[0;38;5;5mregister_apt_sources[21m[0m[39m[2m hook[21m[0m
[0m[39m[2m * git-core/ppa already present, skipping[21m[0m
[0m[39m[2m * installing MariaDB apt sources[21m[0m
[0m[39m[2m * Applying the Ondřej PHP signing key...[21m[0m
OK
[0;38;5;2m ✔ Finished [0m[1m[0;38;5;5mregister_apt_sources[21m[0;38;5;2m hook in [0m[0m[1m[0;38;5;5m0s[21m[0m[0m
[0m[39m[2m * Upgrading apt packages[21m[0m
[0m[39m[2m * Updating apt keys[21m[0m
gpg: key 3B4FE6ACC0B21F32: 3 signatures not checked due to missing keys
gpg: key 3B4FE6ACC0B21F32: "Ubuntu Archive Automatic Signing Key (2012)
Is there something I can do to correct this provision issue?
that looks like it's unrelated to VVV provisioners, I'm guessing you added that source and tried to install systemd manually?
Looking at http://ppa.launchpad.net/pitti/systemd/ubuntu/dists/ there's no focal
folder which would explain the issue.
Fundamentally systemd is there to start and end the services, but it's not the only way of doing it. I don't have a nice neat answer for how you would start and stop elastic without systemd though, I can say that the automatic starting and stopping of MariaDB is something that isn't quite working, and that there are scripts that run on up
and halt
that restart those services in the config/homebin
folder
Though in theory you shouldn't need a custom apt source to install systemd, wouldn't sudo apt-get -y install systemd
work?
I had tried the logical sudo apt-get -y install systemd
method and it didn't work so I thought I would give another suggestion someone gave online, which clearly is the cause now of my issue (thanks for pointing out what I probably should have). Turns out there's a common issue with systemd in docker environments. So to get that repository out of the mix, I had to remove /etc/apt/sources.list.d/pitti-ubuntu-systemd-focal.list
(removing/purging systemd
didn't remove it and neither did sudo ppa-purge 'http://ppa.launchpad.net/pitti/systemd/ubuntu focal'
), then I could reprovision again.
I personally think this is an acceptable fix for the issue of multiple service restarts not working when config/homebin/vagrant_up
is ran:
vvv_info " * Restarting Nginx"
- sudo service nginx restart
+ sudo service nginx status > /dev/null && sudo service nginx restart || sudo service nginx start
vvv_info " * Restarting MariaDB"
- sudo service mariadb restart
+ sudo service mariadb status > /dev/null && sudo service mariadb restart || sudo service mariadb start
vvv_info " * Restarting PHP-FPM"
find /etc/init.d/ -name "php*-fpm" -exec bash -c 'sudo service "$(basename "$0")" restart' {} \;
vvv_info " * Restarting Memcache"
- sudo service memcached restart
+ sudo service memcached status > /dev/null && sudo service memcached restart || sudo service memcached start
It checks if the service is running, reloads if it is and starts it if it isn't. Solved the issue on my end.
That looks good, happy to merge that into VVV for the next release
What was The Command Used To Provision
Using only an out of the box clone from this VVV GitHub repository with no customizations and using default config.
What Kind of VVV Provision Was This
This was a fresh install
Logs/What Broke
Trying to install with Docker via
vagrant up --provision --provider="docker"
resulted in this:Trying to provision with Parallels via
vagrant up --provision --provider="parallels"
resulted in this:And then finally attempting to provision with VMWare via
vagrant up --provision --provider="vmware_desktop"
resulted with this:I thought at least one of these would work now that I'm on an Apple Silicon machine and no longer am able to use virtualbox.
Your Environment
VVV: v3.14 Ruby: 3.1.4 git develop branch, commit 8f4eaaae Platform: darwin shell:/bin/zsh shared_db_folder_disabled Vagrant: v2.4.1 Parallels: v19.4.1 VMWare: Fusion Professional Version 13.5.2 (23775688) Docker Desktop: v4.33.0 (160616), Engine: 27.1.1 MacBook with Apple M3 Max chip MacOS Sonoma 14.6.1 (23G93)