Closed SandraCHC closed 1 month ago
After you have reindexed everything check the sizes of these files. Also run the following in the manticore container, and note the number of indexed messages:
mysql -h0 -P9306
show index piler1 status;
Then restart your stack, and repeat the above checks, except don't reindex yet. Also try fixing mounting manticore.conf as below:
- /Data/Persistent/manticore/etc/manticore.conf:/etc/manticoresearch/manticore.conf
Thank you @jsuto
I started the stack, ran the reindex, and after a restart without reindex checked the data.
The file size in the data directory is identical. Only the manticore.log was slightly larger.
The output from manticore container is the same
show index piler1 status;
+-----------------------------+--------------------------------------------------------------------------+
| Variable_name | Value |
+-----------------------------+--------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 0 |
| indexed_bytes | 0 |
| ram_bytes | 2408 |
| disk_bytes | 1201 |
| disk_mapped | 0 |
| disk_mapped_cached | 0 |
| disk_mapped_doclists | 0 |
| disk_mapped_cached_doclists | 0 |
| disk_mapped_hitlists | 0 |
| disk_mapped_cached_hitlists | 0 |
| killed_documents | 0 |
| killed_rate | 0.00% |
| ram_chunk | 0 |
| ram_chunk_segments_count | 0 |
| disk_chunks | 0 |
| mem_limit | 536870912 |
| mem_limit_rate | 33.33% |
| ram_bytes_retired | 0 |
| locked | 0 |
| tid | 0 |
| tid_saved | 0 |
| query_time_1min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_5min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_15min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_total | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_1min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_5min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_15min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_total | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
+-----------------------------+--------------------------------------------------------------------------+
- /Data/Persistent/manticore/etc/manticore.conf:/etc/manticoresearch/manticore.conf
Sorry for the stupid question. But what's the difference? The mount point in the container is the same?
If this output is after reindexing then you reindexed nothing, 0 emails. What does reindex print while running? Also in the piler container run pilerconf|grep sphx. Do these look correct?
# cd /tmp && /usr/bin/reindex -a
put 864 messages to sph_index table for reindexing
# pilerconf|grep sphx
sphxdb=piler1
sphxhost=127.0.0.1
sphxport=9306
I have tested, if I can access the sphinx DB.
mysql -P 9306
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is xxx
Server version: 6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206) git branch manticore-6.3.2...origin/manticore-6.3.2
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+-----------+
| Databases |
+-----------+
| Manticore |
+-----------+
1 row in set (0.000 sec)
Fix piler.conf to have rtindex=1, and try again.
I already have that. Enclosed is my piler.conf
archive_address=
archive_emails_not_having_message_id=0
archive_only_mydomains=0
backlog=20
check_for_client_timeout_interval=20
cipher_list=ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5>
clamd_socket=/tmp/clamd
default_retention_days=2557
enable_chunking=0
enable_cjk=0
enable_folders=0
encrypt_messages=1
extra_to_field=X-Envelope-To:
extract_attachments=1
helper_timeout=20
hostid=xxxxxxxxxxx
listen_addr=0.0.0.0
listen_port=25
max_connections=64
max_message_size=50000000
max_requests_per_child=10000
max_smtp_memory=500000000
memcached_servers=127.0.0.1
memcached_to_db_interval=900
memcached_ttl=86400
min_message_size=100
min_word_len=1
mmap_dedup_test=0
mysql_connect_timeout=2
mysqlcharset=utf8mb4
mysqldb=piler
mysqlpwd=xxxxxxxxxx
mysqlsocket=
mysqluser=piler
number_of_worker_processes=2
pemfile=/etc/piler/piler.pem
pidfile=/var/run/piler/piler.pid
piler_header_field=X-piler-id:
process_rcpt_to_addresses=0
rtindex=1
security_header=
server_id=0
smtp_access_list=0
smtp_timeout=60
spam_header_line=
sphxdb=piler1
sphxhost=127.0.0.1
sphxport=9306
syslog_recipients=0
tls_enable=1
tls_min_version=TLSv1.2
tweak_sent_time_offset=0
update_counters_to_memcached=0
username=piler
verbosity=5
workdir=/var/piler/tmp
mysqlhost=mysql
Set sphxhost=manticore
, and reindex again please.
Thank you
sphxdb=piler1
sphxhost=manticore
sphxport=9306
# cd /tmp && /usr/bin/reindex -a
put 864 messages to sph_index table for reindexing
But now I don't have any mails in my default auditor account
Check the piler1 index status: show index piler1 status;
MySQL [(none)]> show index piler1 status;
+-----------------------------+--------------------------------------------------------------------------+
| Variable_name | Value |
+-----------------------------+--------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 0 |
| indexed_bytes | 0 |
| ram_bytes | 2408 |
| disk_bytes | 1640 |
| disk_mapped | 0 |
| disk_mapped_cached | 0 |
| disk_mapped_doclists | 0 |
| disk_mapped_cached_doclists | 0 |
| disk_mapped_hitlists | 0 |
| disk_mapped_cached_hitlists | 0 |
| killed_documents | 0 |
| killed_rate | 0.00% |
| ram_chunk | 0 |
| ram_chunk_segments_count | 0 |
| disk_chunks | 0 |
| mem_limit | 536870912 |
| mem_limit_rate | 33.33% |
| ram_bytes_retired | 0 |
| locked | 0 |
| tid | 0 |
| tid_saved | 0 |
| query_time_1min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_5min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_15min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_total | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_1min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_5min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_15min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_total | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
+-----------------------------+--------------------------------------------------------------------------+
After Reindex
MySQL [(none)]> show index piler1 status;
+-----------------------------+--------------------------------------------------------------------------+
| Variable_name | Value |
+-----------------------------+--------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 0 |
| indexed_bytes | 0 |
| ram_bytes | 2408 |
| disk_bytes | 1640 |
| disk_mapped | 0 |
| disk_mapped_cached | 0 |
| disk_mapped_doclists | 0 |
| disk_mapped_cached_doclists | 0 |
| disk_mapped_hitlists | 0 |
| disk_mapped_cached_hitlists | 0 |
| killed_documents | 0 |
| killed_rate | 0.00% |
| ram_chunk | 0 |
| ram_chunk_segments_count | 0 |
| disk_chunks | 0 |
| mem_limit | 536870912 |
| mem_limit_rate | 33.33% |
| ram_bytes_retired | 0 |
| locked | 0 |
| tid | 0 |
| tid_saved | 0 |
| query_time_1min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_5min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_15min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| query_time_total | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_1min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_5min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_15min | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
| found_rows_total | {"queries":0, "avg":"-", "min":"-", "max":"-", "pct95":"-", "pct99":"-"} |
+-----------------------------+--------------------------------------------------------------------------+
I'm confused. How is it possible that reindex said it has indexed 864 messages, it put to piler1 table in the manticore container, and you still can see 0 indexed documents?
I don't know. If I change the sphxhost=127.0.0.1 I see all mails after reindex.
Change the sphinx host to 'manticore' in /etc/piler/config-site.php.
Thank you. I have first changed it in piler.conf.
Now my config-site.php:
<?php
define('SITE_NAME_CONST', 'SITE_NAME');
$config[SITE_NAME_CONST] = 'mypilerdomain';
$config['SITE_URL'] = 'http://' . $config[SITE_NAME_CONST] . '/';
$config['SMTP_DOMAIN'] = $config[SITE_NAME_CONST];
$config['DECRYPT_BINARY'] = '/usr/bin/pilerget';
$config['DECRYPT_ATTACHMENT_BINARY'] = '/usr/bin/pileraget';
$config['PILER_BINARY'] = '/usr/sbin/piler';
$config['DB_HOSTNAME'] = 'mysql';
$config['DB_DATABASE'] = 'piler';
$config['DB_USERNAME'] = 'piler';
$config['DB_PASSWORD'] = 'xxxxx';
$config['MEMCACHED_ENABLED'] = 1;
$memcached_server = ['memcached', 11211];
$config['RT'] = 1;
$config['SPHINX_MAIN_INDEX'] = 'piler1';
$config['SPHINX_HOSTNAME'] = 'manticor:9306';
But I think I have more issues, but I don't find it. Are there Missing config parameters?
# cd /tmp && /usr/bin/reindex -a
put 864 messages to sph_index table for reindexing
MySQL [(none)]> show index piler1 status;
+-----------------------------+--------------------------------------------------------------------------------------------------------+
| Variable_name | Value |
+-----------------------------+--------------------------------------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 864 |
| indexed_bytes | 4464248 |
| ram_bytes | 3150287 |
| disk_bytes | 1640 |
| disk_mapped | 0 |
| disk_mapped_cached | 0 |
| disk_mapped_doclists | 0 |
| disk_mapped_cached_doclists | 0 |
| disk_mapped_hitlists | 0 |
| disk_mapped_cached_hitlists | 0 |
| killed_documents | 0 |
| killed_rate | 0.00% |
| ram_chunk | 3147879 |
| ram_chunk_segments_count | 26 |
| disk_chunks | 0 |
| mem_limit | 536870912 |
| mem_limit_rate | 33.33% |
| ram_bytes_retired | 0 |
| locked | 0 |
| tid | 864 |
| tid_saved | 0 |
| query_time_1min | {"queries":2, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.000, "pct95_sec":0.000, "pct99_sec":0.000} |
| query_time_5min | {"queries":8, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.000, "pct95_sec":0.000, "pct99_sec":0.000} |
| query_time_15min | {"queries":8, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.000, "pct95_sec":0.000, "pct99_sec":0.000} |
| query_time_total | {"queries":8, "avg_sec":0.000, "min_sec":0.000, "max_sec":0.000, "pct95_sec":0.000, "pct99_sec":0.000} |
| found_rows_1min | {"queries":2, "avg":0, "min":0, "max":0, "pct95":0, "pct99":0} |
| found_rows_5min | {"queries":8, "avg":0, "min":0, "max":0, "pct95":0, "pct99":0} |
| found_rows_15min | {"queries":8, "avg":0, "min":0, "max":0, "pct95":0, "pct99":0} |
| found_rows_total | {"queries":8, "avg":0, "min":0, "max":0, "pct95":0, "pct99":0} |
+-----------------------------+--------------------------------------------------------------------------------------------------------+
30 rows in set (0.000 sec)
But after a stop and start from my container all the values are 0 again and no mails are there
You have a typo:
$config['SPHINX_HOSTNAME'] = 'manticor:9306';
Sorry I don't understand why the e is missing in my pasted code. My config is correct with $config['SPHINX_HOSTNAME'] = 'manticore:9306';
I think that's why I now also see entries in the DB
| Variable_name | Value |
+-----------------------------+--------------------------------------------------------------------------------------------------------+
| index_type | rt |
| indexed_documents | 864 |
| indexed_bytes | 4464248 |
| ram_bytes | 3150287 |
| disk_bytes | 2877713 |
| disk_mapped | 0 |
| disk_mapped_cached | 0 |
| disk_mapped_doclists | 0 |
| disk_mapped_cached_doclists | 0 |
| disk_mapped_hitlists | 0 |
| disk_mapped_cached_hitlists | 0 |
| killed_documents | 0 |
| killed_rate | 0.00% |
| ram_chunk | 3147879 |
| ram_chunk_segments_count | 26 |
| disk_chunks | 0 |
| mem_limit | 536870912 |
| mem_limit_rate | 33.33% |
| ram_bytes_retired | 0 |
| locked | 0 |
| tid | 864 |
| tid_saved | 864 |
And after start and restart the container the db is empty again
If I check
mysql -h0 -P9306
show index piler1 status;
In my manticore Container, there are still 864 mails after reboot. Does pile use more than one DB, or reference to wrong DB?
In my piler.conf I have the entry
sphxdb=piler1
#sphxhost=manticore
sphxhost=127.0.0.1
sphxport=9306
(if I use manticore here I have no more emails in my Auditor Profile), and in my config-site.php
$config['SPHINX_HOSTNAME'] = 'manticore:9306';
You consistent settings for both the piler binaries and the gui regarding the mysql database and the manticore database. When the values in piler.conf and config-site.php (Be sure to check /var/piler/www/config.php for the defaults!) are in sync, then everything shall work.
my config-site.php:
# cat /etc/piler/config-site.php
<?php
define('SITE_NAME_CONST', 'SITE_NAME');
$config[SITE_NAME_CONST] = 'xxxxx';
$config['SITE_URL'] = 'http://' . $config[SITE_NAME_CONST] . '/';
$config['SMTP_DOMAIN'] = $config[SITE_NAME_CONST];
$config['SMTP_FROMADDR'] = 'no-reply@' . $config[SITE_NAME_CONST];
$config['ADMIN_EMAIL'] = 'admin@' . $config[SITE_NAME_CONST];
$config['DB_PASSWORD'] = 'MYSQL_PASSWORD';
$config['DECRYPT_BINARY'] = '/usr/bin/pilerget';
$config['DECRYPT_ATTACHMENT_BINARY'] = '/usr/bin/pileraget';
$config['PILER_BINARY'] = '/usr/sbin/piler';
$config['DB_HOSTNAME'] = 'mysql';
$config['DB_DATABASE'] = 'piler';
$config['DB_USERNAME'] = 'piler';
$config['DB_PASSWORD'] = 'xxxxx';
$config['MEMCACHED_ENABLED'] = 1;
$memcached_server = ['memcached', 11211];
$config['RT'] = 1;
$config['SPHINX_MAIN_INDEX'] = 'piler1';
$config['SPHINX_HOSTNAME'] = 'manticore:9306';
$config['SPHINX_STRICT_SCHEMA'] = 1; // required for Sphinx see https://bitbucket.org/jsuto/piler/issues/1085/sphinx-331.
$config['ENABLE_INSTANT_SEARCH'] = 1;
$config['ENABLE_TABLE_RESIZE'] = 1;
$config['DATE_TEMPLATE'] = 'd.m.Y - H:i';
$config['TIMEZONE'] = 'Europe/Zurich';
$config['ENABLE_AUDIT'] = 0;
//$config['ENABLE_IMAP_AUTH'] = 1;
$config['RESTORE_OVER_IMAP'] = 0;
//$config['STRIP_DOMAIN_NAME_FROM_USERNAME'] = 0;
$config['IMAP_RESTORE_FOLDER_INBOX'] = 'INBOX';
$config['IMAP_RESTORE_FOLDER_SENT'] = 'Sent';
$config['IMAP_HOST'] = 'xxxxx';
$config['IMAP_PORT'] = 993;
$config['IMAP_SSL'] = true;
$config['SMTP_DOMAIN'] = 'xxxxx';
$config['SMTP_FROMADDR'] = 'xxxxx';
$config['ADMIN_EMAIL'] = 'xxxxx';
//$config['USE_EMAIL_AS_USERNAME'] = 1;
$config['SMARTHOST'] = 'xxxxx';
$config['SMARTHOST_PORT'] = 587;
$config['SMARTHOST_USER'] = 'xxxxx';
$config['SMARTHOST_PASSWORD'] = 'xxxxx';
Do you see any issues? I don't understand why i see the values in manticore container but not in my piler container (after reboot)
Add this line too:
$config['SPHINX_HOSTNAME_READONLY'] = 'manticore:9307';
To avoid transferring old errors, I recreated the Docker container and re-imported the emails.
I have entered the additional config as described by you. Now the Web Auditor portal is empty again and no emails are available until I remove the parameter. I also asked a friend if he could test the container and he had the same behavior with my configuration.
Then start the troubleshooting over. Check if piler.conf and config-site.php have consistent index settings and use manticore in the manticore container, then check the piler1 index status in the manticiore conteiner.
Thank you very much If I change STRIP_DOMAIN_NAME_FROM_USERNAME and additional in piler.conf sphxhost to manticore, it works
Please excuse me. I thought because of your comment above I was only allowed to customize the config-site.php. That was stupid of me.
You are actually right. You should put all local site specific value to config-site.php, and not modify config.php. Anyway, I'm glad that it works by now.
Thank you. This thread fixed my indexing problems!!!
Hello
I have adopted the Docker-compose (https://github.com/jsuto/piler/blob/master/docker/docker-compose.yaml) in Portainer and created the data structure to obtain persistent data.
My Portainer stack looks like this:
I have only made one change in my manticore.conf:
As soon as I stop and restart the stack, no more emails are displayed and I have to start the reindex with the following command
I found this issue (https://github.com/jsuto/piler/issues/179), but I don't see if it's really the same problem. I also checked if I have data in the folder /Data/Persistent/manticore/data, which is the case: