Websocket zmq not working with reverse proxy

[kennhardy]

When using docker, either configuring everything manually with a regular setup or when using the install script, websocket is not working. Something seems to be missing from a regular setup using php. It seems nginx are not able to understans or translate the zmq and hence not becoming a socket. Instead only reaching the endpoint over http(s).

After spending countless hours on this I decided to cash up for a VM to try it outside my current hosting setup and came to the same conclusion result. Or I might be missing something fundamental, if so I'd love some help.

"auto-install" with ssh:

# MODE: letsencrypt
version: '3'
services:
  espocrm-mysql:
    image: mysql:8
    container_name: espocrm-mysql
    command: --default-authentication-plugin=mysql_native_password
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: pass
      MYSQL_DATABASE: espocrm
      MYSQL_USER: espocrm
      MYSQL_PASSWORD: pass
    volumes:
      - ./data/mysql/data:/var/lib/mysql
    networks:
      - internal

  espocrm-nginx:
    image: nginx
    container_name: espocrm-nginx
    restart: always
    depends_on:
      - espocrm
    volumes:
      - ./data/nginx/conf.d/:/etc/nginx/conf.d/
      - ./data/espocrm:/var/www/html
      - ./data/nginx/espocrm.conf:/etc/nginx/espocrm.conf
      - ./data/nginx/logs:/var/log/nginx
      - ./data/nginx/ssl:/etc/nginx/ssl
      - ./data/nginx/certbot:/var/www/certbot
    ports:
      - "80:80"
      - "443:443"
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
    networks:
      - external

  espocrm:
    image: espocrm/espocrm:fpm
    container_name: espocrm
    environment:
      ESPOCRM_DATABASE_HOST: espocrm-mysql
      ESPOCRM_DATABASE_USER: espocrm
      ESPOCRM_DATABASE_PASSWORD: pass
      ESPOCRM_ADMIN_USERNAME: admin
      ESPOCRM_ADMIN_PASSWORD: pass
      ESPOCRM_SITE_URL: "https://espocrm.domain.tld"
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN: "tcp://espocrm-websocket:7777"
    restart: always
    depends_on:
      - espocrm-mysql
    volumes:
     - ./data/espocrm:/var/www/html
    networks:
      - external
      - internal

  espocrm-daemon:
    image: espocrm/espocrm:fpm
    container_name: espocrm-daemon
    volumes:
      - ./data/espocrm:/var/www/html
    restart: always
    depends_on:
      - espocrm
    entrypoint: docker-daemon.sh
    networks:
      - external
      - internal

  espocrm-websocket:
    container_name: espocrm-websocket
    image: espocrm/espocrm:fpm
    environment:
      ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBSCRIBER_DSN: "tcp://*:7777"
      ESPOCRM_CONFIG_USE_WEB_SOCKET: "true"
    volumes:
     - ./data/espocrm:/var/www/html
    restart: always
    depends_on:
      - espocrm
    entrypoint: docker-websocket.sh
    ports:
      - 8080:8080
    networks:
      - external
      - internal

  espocrm-certbot:
    image: certbot/certbot
    container_name: espocrm-certbot
    restart: always
    volumes:
      - ./data/nginx/ssl:/etc/letsencrypt
      - ./data/nginx/certbot:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    networks:
      - external

networks:
  external:
    external: true
  internal:
    external: false

[kennhardy]

Has anyone else gotten the websocket's working together with docker and nginx as a reverse proxy?

[frauhottelmann]

I haven't gotten it to work with HaProxy either. Not sure about php-zmq, it doesn't seem to be maintained any more. At least not compared to the other zmq variants.

[kennhardy]

Yes. Not sure what breaks it. I have two instances running in docker, and another two running in kubernetes. Same with both. No matter what I try, websockets does not work.

[digitalextremist]

@kennhardy I have a nginx reverse proxy, docker containers, and websockets working. Will come back to chime in on this.

[kennhardy]

@digitalextremist oh, that sounds promising. Would you be able to share your config? Are you running out in docker as well?

[digitalextremist]

By the way, first impression: My configuration is very different because I have nginx running on the host machine, and that provides individual reverse proxies to each instance running. So there is no nginx in my docker-compose.yml at all. Therefore also no certbot in my docker configuration.

But looking closer at what you have, I am actually missing ESPOCRM_CONFIG_WEB_SOCKET_ZERO_M_Q_SUBMISSION_DSN completely. I am going to add that back with a new instance and double check if I have ZMQ working as I thought. I am having no websocket problems with my configuration right now that I know of though!

Will check back.

[digitalextremist]

@kennhardy I have since been able to get the part you are working with running with a reverse proxy, but I am starting to wonder if the background communications are either being dropped, or not happening. But the websocket itself is connected :)

Still have this on my mind and returning with more soon hopefully. Anything change for you?

[digitalextremist]

I do see there is ICMP traffic possible between containers though...

And when I turned on webSocketDebugMode I do see these:

websocket  | [2023-09-27 01:09:58] message newNotification for user 6501a8b5bdc913270
websocket  | [2023-09-27 01:09:58] message newNotification for user 65137fa3ec7b8ac51
websocket  | [2023-09-27 01:09:58] message streamUpdate.Case.651371e675bf83170 for all

Now it seems like the issue is that they never show up on the UX end.

[digitalextremist]

Wait, there's the error:

[Shahin-rmz]

@digitalextremist my config is much like yours. do you get notifications and reminders or you also have problem?

[digitalextremist]

I was able to get this worked out @Shahin-rmz

Would need to recall, but I think I completely redid my configuration after this point.

[Shahin-rmz]

I was able to get this worked out @Shahin-rmz

Would need to recall, but I think I completely redid my configuration after this point.

It would be awesome if you share your docker compose, nginx Proxy manager config whenever you have time. Thanks