Mac M2 issue? Either Varnish or ElasticSearch doesn't start & Bad Gateway 502

marcoveeneman commented 2 months ago

Hi,

Since a while I noticed my dev containers using hypernode-docker containers don't work anymore on my M2 MacBook Pro. Depending on the "Use Rosetta for x86_64/amd64 emulation on Apple Silicon" setting, either varnish or elasticsearch won't start. This happens for existing containers, as well as new containers with the latest image. Something in docker must have changed somehow because think I updated Docker Desktop a while ago.

With "Use Rosetta for x86_64/amd64 emulation on Apple Silicon" enabled, this is the list of 'screen' commands that keep running:

root@285bad760d6b / # screen -ls
There are screens on:
    267.hypernode_service_varnish   (04/18/2024 10:56:53 AM)        (Detached)
    180.hypernode_service_php       (04/18/2024 10:56:52 AM)        (Detached)
    189.hypernode_service_mysql     (04/18/2024 10:56:52 AM)        (Detached)
    194.hypernode_service_mailhog   (04/18/2024 10:56:52 AM)        (Detached)
    170.hypernode_service_nginx     (04/18/2024 10:56:51 AM)        (Detached)
    173.hypernode_service_redis     (04/18/2024 10:56:51 AM)        (Detached)
6 Sockets in /run/screen/S-root.

Trying to start elastic search manually gives the following console output:

root@285bad760d6b / # mkdir -p /var/run/elasticsearch; chown -R elasticsearch /var/run/elasticsearch; su -s /bin/bash -c 'source /etc/default/elasticsearch; export ES_HOME=/usr/share/elasticsearch; export ES_PATH_CONF=/etc/elasticsearch; export PID_DIR=/var/run/elasticsearch; export ES_SD_NOTIFY=true; /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet' elasticsearch
Exception in thread "main" java.io.IOException: Cannot run program "/usr/share/elasticsearch/jdk/bin/java": error=0, Failed to exec spawn helper.
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
        at org.elasticsearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:111)
        at org.elasticsearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:88)
        at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:59)
        at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:95)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper.
        at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:319)
        at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:250)
        at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
        ... 5 more

With "Use Rosetta for x86_64/amd64 emulation on Apple Silicon" disabled, this is the list of 'screen' commands that keep running:

root@285bad760d6b / # screen -ls
There are screens on:
    696.hypernode_service_elasticsearch     (04/18/2024 10:18:44 AM)        (Detached)
    526.hypernode_service_redis     (04/18/2024 10:18:39 AM)        (Detached)
    563.hypernode_service_mysql     (04/18/2024 10:18:39 AM)        (Detached)
    545.hypernode_service_php       (04/18/2024 10:18:39 AM)        (Detached)
    518.hypernode_service_nginx     (04/18/2024 10:18:38 AM)        (Detached)

Trying to start varnish manually gives the following console output:

root@285bad760d6b / # $VARNISH_COMMAND
Error:
Message from VCC-compiler:
Assert error in vju_subproc(), mgt/mgt_jail_unix.c line 212:
  Condition((initgroups(vju_user, vju_gid)) == 0) not true.
  errno = 1 (Operation not permitted)
qemu: uncaught target signal 6 (Aborted) - core dumped
Running VCC-compiler failed, signal 6
VCL compilation failed

In this situation (varnish not running), the frontend appears to work correctly, but after logging in to the admin panel, nginx will return a 502 Bad Gateway.

I'm wondering if anyone is experiencing the same issue, and if there is a (simple) fix available.

marcoveeneman commented 2 months ago

Some more info regarding the 502 Bad Gateway. The following is in the nginx access+error log files:

access.log:

{"time":"2024-04-18T13:32:54+00:00", "remote_addr":"172.17.0.1", "remote_user":"", "host":"127.0.0.1:8088", "request":"POST /admin HTTP/1.1", "status":"502", "body_bytes_sent":"552", "referer":"http://127.0.0.1:8088/admin", "user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36", "request_time":"0.397", "handler":"phpfpm", "country":"", "server_name":"dummytag.hypernode.io", "port":"80", "ssl_cipher":"", "ssl_protocol":""}

error.log:

2024/04/18 13:32:54 [error] 11949#11949: *123 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.17.0.1, server: dummytag.hypernode.io, request: "POST /admin HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1:8088", referrer: "http://127.0.0.1:8088/admin"

marcoveeneman commented 2 months ago

Update:

I downgraded docker desktop to: 4.26.1 (131620).

Varnish still crashes. However, the Bad Gateway 502 when logging in to the backend is gone. Probably it's not varnish which is causing the 502, but something else. For now i can continue (with an older version of Docker), but it will break again after updating so i guess more people will encounter this sooner or later.

If i can help by providing some logs, i'm happy to help, just let me know.

ByteInternet / hypernode-docker

Mac M2 issue? Either Varnish or ElasticSearch doesn't start & Bad Gateway 502 #86