solr status not shown in mailcow UI

zeigerpuppy commented 5 years ago

minor issue, but after running the last update and enabling Solr, I don't see the Solr status as properly shown:

2019-01-21 10 17 35 pm

I don't see any errors in the logs and FTS with solr is working correctly, so I guess it's just a miscommunication between the UI and the solr instance.

andryyy commented 5 years ago

Are you sure it is working? Any firewall on your host?

zeigerpuppy commented 5 years ago

yes, indexes are properly built and the search is sped up. There's no firewall but I do use a bit of a custom setup with a manually defined network and volumes, although it should all be compliant with the mailcow design. I can ping between the nginx and solr machines using their aliases properly.

logs show solr indexing

dovecot-mailcow_1    | Jan 21 22:40:13 mail dovecot: imap-login: Login: user=<user@server.net>, method=PLAIN, rip=172.22.1.10, lip=172.22.1.6, mpid=9540, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
solr-mailcow_1       | 2019-01-21 11:40:13.648 INFO  (qtp574568002-13) [   x:dovecot] o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select params={q=box:139c313330509857ce41000018b49b4a+AND+user:user@server.net&fl=uid&sort=uid+desc&rows=1&wt=xml} hits=0 status=0 QTime=2
solr-mailcow_1       | 2019-01-21 11:40:13.669 INFO  (qtp574568002-18) [   x:dovecot] o.a.s.c.S.Request [dovecot]  webapp=/solr path=/select params={q={!lucene+q.op%3DAND}subject:test+OR+from:test&fl=uid,score&sort=uid+asc&fq=%2Bbox:139c313330509857ce41000018b49b4a+%2Buser:user@server.net&rows=568&wt=xml} hits=0 status=0 QTime=8

The UI does show solr up and I have confirmed by opening a shell inside the machine

2019-01-21 11 01 08 pm

docker-compose file:

version: '2.1'
services:

    unbound-mailcow:
      image: mailcow/unbound:1.4
      build: ./data/Dockerfiles/unbound
      command: /usr/sbin/unbound
      environment:
        - TZ=${TZ}
      volumes:
        - ./data/conf/unbound/unbound.conf:/etc/unbound/unbound.conf:ro
      restart: always
      tty: true
      networks:
        mailcow-network:
          ipv4_address: ${IPV4_NETWORK:-172.22.1}.254
          aliases:
            - unbound

    mysql-mailcow:
      image: mariadb:10.2
      volumes:
        - mysql-vol-1:/var/lib/mysql/
        - mysql-socket-vol-1:/var/run/mysqld/
        - ./data/conf/mysql/:/etc/mysql/conf.d/:ro
      environment:
        - TZ=${TZ}
        - MYSQL_ROOT_PASSWORD=${DBROOT}
        - MYSQL_DATABASE=${DBNAME}
        - MYSQL_USER=${DBUSER}
        - MYSQL_PASSWORD=${DBPASS}
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      ports:
        - "${SQL_PORT:-127.0.0.1:13306}:3306"
      networks:
        mailcow-network:
          aliases:
            - mysql

    redis-mailcow:
      image: redis:5-alpine
      volumes:
        - redis-vol-1:/data/
      restart: always
      environment:
        - TZ=${TZ}
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          ipv4_address: ${IPV4_NETWORK:-172.22.1}.249
          aliases:
            - redis

    clamd-mailcow:
      image: mailcow/clamd:1.15
      build: ./data/Dockerfiles/clamd
      restart: always
      tty: true
      environment:
        - TZ=${TZ}
        - SKIP_CLAMD=${SKIP_CLAMD:-n}
      volumes:
        - ./data/conf/clamav/:/etc/clamav/
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          aliases:
            - clamd

    rspamd-mailcow:
      image: mailcow/rspamd:1.33
      build: ./data/Dockerfiles/rspamd
      stop_grace_period: 30s
      depends_on:
        - nginx-mailcow
      environment:
        - TZ=${TZ}
      volumes:
        - ./data/conf/rspamd/custom/:/etc/rspamd/custom
        - ./data/conf/rspamd/override.d/:/etc/rspamd/override.d
        - ./data/conf/rspamd/local.d/:/etc/rspamd/local.d
        - ./data/conf/rspamd/lua/:/etc/rspamd/lua/:ro
        - rspamd-vol-1:/var/lib/rspamd
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      hostname: rspamd
      networks:
        mailcow-network:
          aliases:
            - rspamd

    php-fpm-mailcow:
      image: mailcow/phpfpm:1.29
      build: ./data/Dockerfiles/phpfpm
      command: "php-fpm -d date.timezone=${TZ} -d expose_php=0"
      depends_on:
        - redis-mailcow
      volumes:
        - ./data/web:/web:rw
        - ./data/conf/rspamd/dynmaps:/dynmaps:ro
        - rspamd-vol-1:/var/lib/rspamd
        - mysql-socket-vol-1:/var/run/mysqld/
        - ./data/conf/sogo/:/etc/sogo/
        - ./data/conf/rspamd/meta_exporter:/meta_exporter:ro
        - ./data/conf/phpfpm/php-fpm.d/pools.conf:/usr/local/etc/php-fpm.d/z-pools.conf
        - ./data/conf/phpfpm/php-conf.d/opcache-recommended.ini:/usr/local/etc/php/conf.d/opcache-recommended.ini
        - ./data/conf/phpfpm/php-conf.d/upload.ini:/usr/local/etc/php/conf.d/upload.ini
        - ./data/conf/phpfpm/php-conf.d/other.ini:/usr/local/etc/php/conf.d/zzz-other.ini
      environment:
        - LOG_LINES=${LOG_LINES:-9999}
        - TZ=${TZ}
        - DBNAME=${DBNAME}
        - DBUSER=${DBUSER}
        - DBPASS=${DBPASS}
        - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
        - IMAP_PORT=${IMAP_PORT:-143}
        - IMAPS_PORT=${IMAPS_PORT:-993}
        - POP_PORT=${POP_PORT:-110}
        - POPS_PORT=${POPS_PORT:-995}
        - SIEVE_PORT=${SIEVE_PORT:-4190}
        - SUBMISSION_PORT=${SUBMISSION_PORT:-587}
        - SMTPS_PORT=${SMTPS_PORT:-465}
        - SMTP_PORT=${SMTP_PORT:-25}
        - API_KEY=${API_KEY:-invalid}
        - API_ALLOW_FROM=${API_ALLOW_FROM:-invalid}
        - COMPOSE_PROJECT_NAME=${COMPOSE_PROJECT_NAME:-mailcow-dockerized}
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          aliases:
            - phpfpm

    sogo-mailcow:
      image: mailcow/sogo:1.49
      build: ./data/Dockerfiles/sogo
      environment:
        - DBNAME=${DBNAME}
        - DBUSER=${DBUSER}
        - DBPASS=${DBPASS}
        - TZ=${TZ}
        - LOG_LINES=${LOG_LINES:-9999}
        - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
      volumes:
        - ./data/conf/sogo/:/etc/sogo/
        - ./data/web/inc/init_db.inc.php:/init_db.inc.php
        - ./data/conf/sogo/custom-sogo.js:/usr/lib/GNUstep/SOGo/WebServerResources/js/custom-sogo.js
        - mysql-socket-vol-1:/var/run/mysqld/
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          ipv4_address: ${IPV4_NETWORK:-172.22.1}.248
          aliases:
            - sogo

    dovecot-mailcow:
      image: mailcow/dovecot:1.52
      build: ./data/Dockerfiles/dovecot
      cap_add:
        - NET_BIND_SERVICE
      volumes:
        - ./data/conf/dovecot:/usr/local/etc/dovecot
        - ./data/assets/ssl:/etc/ssl/mail/:ro
        - ./data/conf/sogo/:/etc/sogo/
        - vmail-vol-1:/var/vmail
        - vmail-attachments-vol-1:/var/attachments
        - crypt-vol-1:/mail_crypt/
        - ./data/conf/rspamd/custom/:/etc/rspamd/custom
        - rspamd-vol-1:/var/lib/rspamd
        - mysql-socket-vol-1:/var/run/mysqld/
      environment:
        - LOG_LINES=${LOG_LINES:-9999}
        - DBNAME=${DBNAME}
        - DBUSER=${DBUSER}
        - DBPASS=${DBPASS}
        - TZ=${TZ}
        - MAILDIR_GC_TIME=${MAILDIR_GC_TIME:-1440}
      ports:
        - "${DOVEADM_PORT:-127.0.0.1:19991}:12345"
        - "${IMAP_PORT:-143}:143"
        - "${IMAPS_PORT:-993}:993"
        - "${POP_PORT:-110}:110"
        - "${POPS_PORT:-995}:995"
        - "${SIEVE_PORT:-4190}:4190"
      restart: always
      tty: true
      ulimits:
        nproc: 65535
        nofile:
          soft: 20000
          hard: 40000
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      hostname: ${MAILCOW_HOSTNAME}
      networks:
        mailcow-network:
          aliases:
            - dovecot

    postfix-mailcow:
      image: mailcow/postfix:1.29
      build: ./data/Dockerfiles/postfix
      volumes:
        - ./data/conf/postfix:/opt/postfix/conf
        - ./data/assets/ssl:/etc/ssl/mail/:ro
        - postfix-vol-1:/var/spool/postfix
        - crypt-vol-1:/var/lib/zeyple
        - rspamd-vol-1:/var/lib/rspamd
        - mysql-socket-vol-1:/var/run/mysqld/
      environment:
        - LOG_LINES=${LOG_LINES:-9999}
        - TZ=${TZ}
        - DBNAME=${DBNAME}
        - DBUSER=${DBUSER}
        - DBPASS=${DBPASS}
      cap_add:
        - NET_BIND_SERVICE
      ports:
        - "${SMTP_PORT:-25}:25"
        - "${SMTPS_PORT:-465}:465"
        - "${SUBMISSION_PORT:-587}:587"
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      hostname: ${MAILCOW_HOSTNAME}
      networks:
        mailcow-network:
          aliases:
            - postfix

    memcached-mailcow:
      image: memcached:alpine
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          aliases:
            - memcached

    nginx-mailcow:
      depends_on:
        - sogo-mailcow
        - php-fpm-mailcow
        - redis-mailcow
      image: nginx:mainline-alpine
      command: /bin/sh -c "envsubst < /etc/nginx/conf.d/templates/listen_plain.template > /etc/nginx/conf.d/listen_plain.active &&
        envsubst < /etc/nginx/conf.d/templates/listen_ssl.template > /etc/nginx/conf.d/listen_ssl.active &&
        envsubst < /etc/nginx/conf.d/templates/server_name.template > /etc/nginx/conf.d/server_name.active &&
        envsubst < /etc/nginx/conf.d/templates/sogo.template > /etc/nginx/conf.d/sogo.active &&
        envsubst < /etc/nginx/conf.d/templates/sogo_eas.template > /etc/nginx/conf.d/sogo_eas.active &&
        nginx -qt &&
        until ping phpfpm -c1 > /dev/null; do sleep 1; done &&
        until ping sogo -c1 > /dev/null; do sleep 1; done &&
        until ping redis -c1 > /dev/null; do sleep 1; done &&
        until ping rspamd -c1 > /dev/null; do sleep 1; done &&
        exec nginx -g 'daemon off;'"
      environment:
        - HTTPS_PORT=${HTTPS_PORT:-443}
        - HTTP_PORT=${HTTP_PORT:-80}
        - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
        - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1}
        - TZ=${TZ}
      volumes:
        - ./data/web:/web:ro
        - ./data/conf/rspamd/dynmaps:/dynmaps:ro
        - ./data/assets/ssl/:/etc/ssl/mail/:ro
        - ./data/conf/nginx/:/etc/nginx/conf.d/:rw
        - ./data/conf/rspamd/meta_exporter:/meta_exporter:ro
      volumes_from:
        - sogo-mailcow
      ports:
        - "${HTTPS_BIND:-0.0.0.0}:${HTTPS_PORT:-443}:${HTTPS_PORT:-443}"
        - "${HTTP_BIND:-0.0.0.0}:${HTTP_PORT:-80}:${HTTP_PORT:-80}"
      restart: always
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          aliases:
            - nginx

    acme-mailcow:
      depends_on:
        - nginx-mailcow
      image: mailcow/acme:1.45
      build: ./data/Dockerfiles/acme
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      environment:
        - LOG_LINES=${LOG_LINES:-9999}
        - ADDITIONAL_SAN=${ADDITIONAL_SAN}
        - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
        - DBNAME=${DBNAME}
        - DBUSER=${DBUSER}
        - DBPASS=${DBPASS}
        - SKIP_LETS_ENCRYPT=${SKIP_LETS_ENCRYPT:-n}
        - SKIP_IP_CHECK=${SKIP_IP_CHECK:-n}
        - LE_STAGING=${LE_STAGING:-n}
        - TZ=${TZ}
      volumes:
        - ./data/web/.well-known/acme-challenge:/var/www/acme:rw
        - ./data/assets/ssl:/var/lib/acme/:rw
        - ./data/assets/ssl-example:/var/lib/ssl-example/:ro
        - mysql-socket-vol-1:/var/run/mysqld/
      restart: always
      networks:
        mailcow-network:
          aliases:
            - acme

    netfilter-mailcow:
      image: mailcow/netfilter:1.21
      build: ./data/Dockerfiles/netfilter
      stop_grace_period: 30s
      depends_on:
        - dovecot-mailcow
        - postfix-mailcow
        - sogo-mailcow
        - php-fpm-mailcow
        - redis-mailcow
      restart: always
      privileged: true
      environment:
        - TZ=${TZ}
        - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1}
        - IPV6_NETWORK=${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}
        - SNAT_TO_SOURCE=${SNAT_TO_SOURCE:-n}
        - SNAT6_TO_SOURCE=${SNAT6_TO_SOURCE:-n}
      network_mode: "host"
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      volumes:
        - /lib/modules:/lib/modules:ro

    watchdog-mailcow:
      image: mailcow/watchdog:1.32
      # Debug
      #command: /watchdog.sh
      build: ./data/Dockerfiles/watchdog
      oom_kill_disable: true
      volumes:
        - rspamd-vol-1:/var/lib/rspamd
        - mysql-socket-vol-1:/var/run/mysqld/
      restart: always
      environment:
        - LOG_LINES=${LOG_LINES:-9999}
        - TZ=${TZ}
        - DBNAME=${DBNAME}
        - DBUSER=${DBUSER}
        - DBPASS=${DBPASS}
        - USE_WATCHDOG=${USE_WATCHDOG:-n}
        - WATCHDOG_NOTIFY_EMAIL=${WATCHDOG_NOTIFY_EMAIL}
        - MAILCOW_HOSTNAME=${MAILCOW_HOSTNAME}
        - IPV4_NETWORK=${IPV4_NETWORK:-172.22.1}
        - IP_BY_DOCKER_API=${IP_BY_DOCKER_API:-0}
        - CHECK_UNBOUND=${CHECK_UNBOUND:-1}
        - SKIP_CLAMD=${SKIP_CLAMD:-n}
        - SKIP_LETS_ENCRYPT=${SKIP_LETS_ENCRYPT:-n}
        - HTTPS_PORT=${HTTPS_PORT:-443}
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      networks:
        mailcow-network:
          aliases:
            - watchdog

    dockerapi-mailcow:
      image: mailcow/dockerapi:1.24
      restart: always
      build: ./data/Dockerfiles/dockerapi
      oom_kill_disable: true
      environment:
        - DBROOT=${DBROOT}
        - TZ=${TZ}
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - ./data/conf/rspamd/override.d/worker-controller-password.inc:/access.inc:rw
        - vmail-vol-1:/var/vmail:ro
      networks:
        mailcow-network:
          aliases:
            - dockerapi

    solr-mailcow:
      image: mailcow/solr:1.0
      build: ./data/Dockerfiles/solr
      restart: always
      volumes:
        - solr-vol-1:/opt/solr/server/solr/dovecot/data
      dns:
        - ${IPV4_NETWORK:-172.22.1}.254
        - 192.168.122.1
      #dns_search: mailcow-network
      environment:
        - SOLR_HEAP=${SOLR_HEAP:-1024}
        - SKIP_SOLR=${SKIP_SOLR:-y}
      networks:
        mailcow-network:
          aliases:
            - solr

    ipv6nat:
      depends_on:
        - unbound-mailcow
        - mysql-mailcow
        - redis-mailcow
        - clamd-mailcow
        - rspamd-mailcow
        - php-fpm-mailcow
        - sogo-mailcow
        - dovecot-mailcow
        - postfix-mailcow
        - memcached-mailcow
        - nginx-mailcow
        - acme-mailcow
        - netfilter-mailcow
        - watchdog-mailcow
        - dockerapi-mailcow
        - solr-mailcow
      image: robbertkl/ipv6nat
      restart: always
      privileged: true
      network_mode: "host"
      volumes:
        - /var/run/docker.sock:/var/run/docker.sock:ro
        - /lib/modules:/lib/modules:ro

# we have defined a bridge manually (no IPV6)
networks:
  mailcow-network:
    external:
      name: mailcow-network

#networks:
#  mailcow-network:
#    driver: bridge
#    enable_ipv6: true
#    ipam:
#      driver: default
#      config:
#        - subnet: ${IPV4_NETWORK:-172.22.1}.0/24
#        - subnet: ${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}

#volumes:
#  # Storage for email files
#  vmail-vol-1:
#  # Storage for attachments (deduplicated)
#  vmail-attachments-vol-1:
#  mysql-vol-1:
#  mysql-socket-vol-1:
#  redis-vol-1:
#  rspamd-vol-1:
#  postfix-vol-1:
#  crypt-vol-1:

# these named volumes are symlinked to ZFS datasets
volumes:
  # Storage for email files
  vmail-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/vmail-vol-1'
      o: 'bind'
  # Storage for attachments (deduplicated)
  vmail-attachments-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/vmail-attachments-vol-1'
      o: 'bind'
  mysql-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/mysql-vol-1'
      o: 'bind'
  mysql-socket-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/mysql-socket-vol-1'
      o: 'bind'
  redis-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/redis-vol-1'
      o: 'bind'
  rspamd-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/rspamd-vol-1'
      o: 'bind'
  postfix-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/postfix-vol-1'
      o: 'bind'
  crypt-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/crypt-vol-1'
      o: 'bind'
  solr-vol-1:
    driver: local
    driver_opts:
      type: 'none'
      device: '${PWD}/volumes/solr-vol-1'
      o: 'bind'

p.s. changing the #dns_search: mailcow-network had no effect

andryyy commented 5 years ago

You should try to curl it from php-fpm-mailcow on port 8983.

zeigerpuppy commented 5 years ago

curl from php-fpm-mailcow seems to work fine:

# curl http://solr:8983/solr/dovecot/update?commit=true
{
  "responseHeader":{
    "status":0,
    "QTime":15}}

andryyy commented 5 years ago

Could we get in contact so I can check that on your server?

zeigerpuppy commented 5 years ago

sure, will email you.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

clemensmach commented 5 years ago

same here were you able to find a solution?

curl status:0, running/green on debug status site only INFO log entrys (default installation without any customizations)

EDIT: nevermind another docker-compose up -d after update.sh resolved it for me

mailcow / mailcow-dockerized

solr status not shown in mailcow UI #2208