Closed nickchomey closed 1 year ago
After doing a bit more debugging/observation, I notice this pop up in the docker log just before everything fails.
devilbox-redis-1 | 1:M 25 Jan 2023 00:33:36.494 * 10000 changes in 60 seconds. Saving...
devilbox-redis-1 | 1:M 25 Jan 2023 00:33:36.495 * Background saving started by pid 15
devilbox-redis-1 | 15:C 25 Jan 2023 00:33:36.540 * DB saved on disk
devilbox-redis-1 | 15:C 25 Jan 2023 00:33:36.540 * RDB: 2 MB of memory used by copy-on-write
devilbox-redis-1 | 1:M 25 Jan 2023 00:33:36.595 * Background saving terminated with success
However, if I restart docker without redis
, the problem still persists and I dont see anything in particular in the log.
When I look at the logs in /devilbox/log
, there's nothing at all for mariadb, and nginx-mainline. nginx-stable has various files but nothing in them seems relevant. Likewise for php-fpm
Are there any particular tests/diagnostics I could do?
In fact, even if I do NOTHING - dont connect to the vhost, mysql, redis, http://127.0.0.1, etc... - it break after 5 minutes or so.
Likewise, if I only start bind php httpd
, and then do nothing, it will break.
I tried using v2.4.0 and same problem.
If I use netstat -abon
in an elevated Powershell, I can see the docker/wsl-related ports. But they all disappear when it breaks.
TCP 127.0.0.1:25 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:53 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:80 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:80 127.0.0.1:52145 ESTABLISHED 11296
[wslhost.exe]
TCP 127.0.0.1:80 127.0.0.1:52146 ESTABLISHED 11296
[wslhost.exe]
TCP 127.0.0.1:80 127.0.0.1:52147 ESTABLISHED 11296
[wslhost.exe]
TCP 127.0.0.1:80 127.0.0.1:52148 ESTABLISHED 11296
[wslhost.exe]
TCP 127.0.0.1:80 127.0.0.1:52149 ESTABLISHED 11296
[wslhost.exe]
TCP 127.0.0.1:80 127.0.0.1:52150 ESTABLISHED 11296
[wslhost.exe]
TCP 127.0.0.1:443 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:953 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:1001 0.0.0.0:0 LISTENING 4
Can not obtain ownership information
TCP 127.0.0.1:2915 0.0.0.0:0 LISTENING 16836
[qbittorrent.exe]
TCP 127.0.0.1:3306 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:5354 0.0.0.0:0 LISTENING 5604
[mDNSResponder.exe]
TCP 127.0.0.1:5432 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:6379 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:8001 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:9001 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
TCP 127.0.0.1:9200 0.0.0.0:0 LISTENING 11296
[wslhost.exe]
I just fully reinstalled WSL2, Ubuntu, Devilbox etc... and the problem seems to have gone away. I'll close this for now and re-open if it comes back/I find more useful information about it.
Nope - problem is back and I cant find any reason or fix.
Strangely, now when Devilbox crashes, sometimes I can't even access localhost
, yet I can access 127.0.0.1
, neither of which happened previously. I'm not aware of any particular changes that I might have made to affect this.
I have updated the config files, logs, etc... in the OP here with the latest details
I'm closing this again - I restored a Windows 11 backup from a week ago and it is working again with the latest version of Devilbox. I will start applying updates one by one til I find the issue
@cytopia Nevermind, it is still happening without any updates.
Interestingly, I can access mysql with HeidiSQL if I use localhost
rather than 127.0.01
.
Also, now that I'm using .dvl.to
, the site seems to continue working even when 127.0.0.1
is broken. Though, I do believe there have been instances today where localhost
and site.dvl.to
stops working as well... Also, the frontend seems to work sometimes while, at the same time, I can't access wp-admin
- even while I can still access mysql with HeidiSQL on localhost
Quite the frustrating mystery...
Any insights/suggestions for tests/commands I can run in WSL2/Ubuntu or Windows 11 PowerShell?
Very weird - right now I can access Mysql with HeidiSQL on localhost:3306
. But I can't access Redis with Redis Insight on localhost:6379
. Likewise, I can't even access the Devilbox dashboard on either localhost
or 127.0.0.1
And now, a few minutes later after a restart of devilbox, i can access some stuff and not others.
There really doesn't seem to be a pattern. Certain things just break sporadically. And it seems independent of what I'm doing on the site, which plugins are installed etc... As I mentioned above somewhere, sometimes it just breaks even if I dont actually access anything at all - just start devilbox and leave it to die on its own.
These screen recordings might be useful to see the utter randomness of it all. They were taken back to back, so just treat them as one video rather than from separate instances.
As you can see, I can browse certain pages on the front end, but not others. Likewise in teh backend.
And I can still browse Redis Insight and HeidiSQL. However you can see that once I disconnect from Redis, I can't reconnect to it. Same thing happens with HeidiSQL (I tested that just after the video stopped)
So, there's clearly some issue with opening and maintaining ports/connections between Devilbox/docker/wsl2 and Windows.
Everything is still accessible from within WSL2/Ubuntu/Docker/Devilbox - I can open ./shell.sh
and access mysql, redis and do other php things.
Here's a screen recording of that (again, treat it as the same session - devilbox/docker was not reset)
Another, perhaps related, question - is there any sort of NGINX server-side caching going on? Perhaps that might be contributing to some of the sporadic page-load stuff (though the front-end stuff is mostly dynamically generated via AJAX requests...)
Have you checked the php and webserver error logs for anything suspicious?
@cytopia
Yes, I believe I mentioned this somewhere above and included some examples/details.
I see nothing whatsoever in the Docker logs - after the Devilbox starts up, the only entries I see while navigating look like this.
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:15:46 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:15:56 +0000 "GET /wp-admin/edit.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:16:16 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:16:46 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:16:56 +0000 "GET /index.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:16:57 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:00 +0000 "GET /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:23 +0000 "GET /index.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:24 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:27 +0000 "GET /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:28 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:31 +0000 "GET /wp-admin/users.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:35 +0000 "GET /wp-admin/nav-menus.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:42 +0000 "GET /wp-admin/edit.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:42 +0000 "POST /wp-cron.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:43 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:45 +0000 "GET /wp-admin/edit.php" 200
devilbox-redis-1 | 1:M 27 Jan 2023 17:17:51.025 * 100 changes in 300 seconds. Saving...
devilbox-redis-1 | 1:M 27 Jan 2023 17:17:51.025 * Background saving started by pid 16
devilbox-redis-1 | 16:C 27 Jan 2023 17:17:51.057 * DB saved on disk
devilbox-redis-1 | 16:C 27 Jan 2023 17:17:51.057 * RDB: 2 MB of memory used by copy-on-write
devilbox-redis-1 | 1:M 27 Jan 2023 17:17:51.125 * Background saving terminated with success
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:53 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:55 +0000 "GET /index.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:56 +0000 "POST /wp-admin/admin-ajax.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:58 +0000 "GET /index.php" 200
devilbox-php-1 | 172.16.238.11 - 27/Jan/2023:17:17:59 +0000 "GET /index.php" 200
devilbox-redis-1 | 1:M 27 Jan 2023 17:22:52.061 * 100 changes in 300 seconds. Saving...
devilbox-redis-1 | 1:M 27 Jan 2023 17:22:52.062 * Background saving started by pid 17
devilbox-redis-1 | 17:C 27 Jan 2023 17:22:52.093 * DB saved on disk
devilbox-redis-1 | 17:C 27 Jan 2023 17:22:52.094 * RDB: 2 MB of memory used by copy-on-write
devilbox-redis-1 | 1:M 27 Jan 2023 17:22:52.162 * Background saving terminated with success
And the logs in /devilbox/log
show nothing either (I've tried setting DOCKER_LOGS=1 and =0).
But it is entirely possible that I don't have logging configured properly or I'm not looking in the right place. I would have expected to see entries in the Docker log for httpd, bind, and mysql - especially since I set DEBUG_ENTRYPOINT=4
(and have also tried 1-3. They all look the EXACT same - perhaps the problem is related to that?)
I also just tried Apache 2.4 and PHP 7.4. Same problems exist. And Nothing shows up in the error logs. And there simply isn't a mariadb log at all.
I just reverted to v2.4.0 and i now see devilbox-httpd-1
in the docker logs, but still nothing for mysql.
But, still, no errors show up when things break.
I think I've discovered and fixed the problem. As some of my comments above presumed, it was an issue with WSL/Windows (the docker services still worked from within wsl, but not from Windows)
This may be off-topic, but I've seen something similar to what you describe, but with networking. PHP will begin to load, and then it fails when running apt update
. docker compose down
and restarting the stack works without issue. WSL 1.1.0 has not been offered to me in Windows 11. This may be unrelated, but I thought to share my experience, just in case someone else has seen this.
I have windows 11 insider beta, so that might be why. Though, if you install from the Windows store, you can get the latest version of wsl. But, obviously, I'd wait til 1.1.1 to fix this bug - it has messed a lot of people up. I lost dozens of hours to it and surely annoyed the devilbox Dev!
Have you already looked into this bug?
(Optional) Error message
No response
What went wrong?
Suddenly Devilbox keeps sporadically breaking. I do
docker compose up
, browse the wordpress site, and eventually everything just breaks. I can't connect to MySQL or Redis remotely (using HeidiSQL and Redis Insight).http://127.0.0.1
doesn't work either, buthttp://localhost
does, and all services seem fine. If I go tolocalhost/vhosts.php
, it shows my vhost, but saysNo Host DNS record found. Add the following to /etc/hosts: 127.0.0.1 site.loc
.If I stop all docker containers and restart them, it works again immediately.
So, it seems like there's some DNS issue, probably related to docker and/or WSL2. I don't see anything in any logs that would suggest a problem, but it is entirely possible that I'm not looking in the right place.
Expected behaviour
I expect this not to happen.
How can we reproduce the bug?
No idea.
Host Operating System
Ubuntu 22.04 via WSL2
Host Platform (amd64, arm64, other)
amd64
(Linux only) Is SELinux enabled?
No
Docker version
Docker version 20.10.23, build 7155243
Docker Compose version
Docker Compose version v2.15.1
Devilbox version
Devilbox v3.0.0-beta-0.3 10b20c0
Have you removed stopped containers before starting?
Yes
Have you pulled latest Docker images before starting?
Yes
Devilbox start command
docker compose up bind php httpd mysql
Config: .env file
Config: docker-compose.override.yml
Config: ./check-config.sh
Log: docker-compose logs
(Optional) Additional information
Windows Hosts File