Open RyDoRe opened 3 weeks ago
what is your OS \ container?
The container runs in a k3s cluster with debian11/12
could you provide the container along with data that reproduces this crash locally?
Unfortunately I'm not able to provide you with that, as this crash only happens with the large dataset we have in production and I'm not allowed and able to forward this kind of data :/
I have a smaller review-setup, also as part of a kubernetes cluster, but that one works. It has but a fraction of the data we use in production.
Is there a visible misconfiguration in the variables I use in the manticore.conf file that you can make out?
We discussed this issue on our development call today, and it's not clear how to proceed without the data necessary to reproduce it. Unfortunately, the crash backtrace has not been helpful in this case.
I'm not allowed and able to forward this kind of data :/
@RyDoRe Just FYI, the core team also offers professional services, which includes signing a work agreement with an NDA, etc. If resolving this issue is mission-critical for you, please consider this option.
I'm going to talk with my team about it and will inform you on how we want to proceed. Thank you for looking into it!
For the sake of completeness, here is the explicit Dockerfile and startup-script:
Dockerfile:
FROM manticoresearch/manticore:6.3.0
ENV TZ=Europe/Berlin
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential libmariadbd-dev make autoconf git wget php-cli php-mysqli iputils-ping
RUN apt-get update && apt install python3-pip -y && pip3 install sphinxapi-py3
USER manticore
COPY scripts/createConfig.php /var/lib/scripts/createConfig.php
COPY dict /var/lib/dict
COPY scripts/delta_indexing.sh /var/lib/scripts/delta_indexing.sh
COPY scripts/full_indexing.sh /var/lib/scripts/full_indexing.sh
COPY scripts/health-check.py /tmp/health/health-check.py
ADD start.sh /
CMD ["/start.sh"]
start.sh:
#!/bin/bash
php /var/lib/scripts/createConfig.php
if [ ! -f /var/lib/manticore/main_table1.spa ]; then
/var/lib/scripts/full_indexing.sh
fi
searchd -c /etc/manticoresearch/manticore.conf --nodetach --logdebugv
I think we were able to narrow down the issue quite a bit...
The issues we are having is related to the number of sources we are trying to use for the indexes. We have 6000+ sources we are trying to address with one index and with that amount we get the issues mentioned above. Limiting the number of sources to e.g. 200 works perfectly fine and with that, rotating after indexing does not kill the searchd process anymore.
I think our way forward is to shard our indexes to have a capped amount of sources and than add more indexes.
seems strange that you indexing your data well, indextool shows no error after that but daemon crashed on loading that index. I sure if the indexer finished without error and indextool reports no issues then daemon should load index well too
do your data sources share the same schema?
do you use joins or any other kind of data depends on each other?
The sources dataschema are identical. We use joins with the sql_joined_field in some of the indexes querys, but not to tables that are used in other indexes or that depend on each other.
I'm currently trying out sharding the indexes wit max. 1000 sources and also trying out each shard with running the indexers, to see if there is one specific shard that fails.
Just a heads up and my results so far: I now created a setup in which I shard my indexers to each use a maximum number of sources, which i defined to be 1000 currently. In the end I use a distributed table to add the shards back together for the search. This setup works reliably now from what I can see... The main indexers and also the delta indexers run trough without a problem.
Interesting observations: We currently have 5 shards and a little below 5000 sources Before the sharding, so per index around that number of sources, we had a memory usage up to and around 28-32 GiB. With the sharding the memory usage of the indexing process now only goes upo to 12 GiB.
Hi there!
I already tried my luck in the forum and got redirected to create a Ticket here: https://forum.manticoresearch.com/t/malloc-issues-when-indexing-large-plain-tables/2069/7
Setup description:
I have a plain table setup using the main+delta schema. I use manticore in Docker containers, my source is an external mariadb and the manticore container(s) are running in a k8s cluster.
I have manticore setup as a statefulset in the kubernetes setup and use a local-path storage for the manticore data.
My Dockerfile for manticore uses a custom startup CMD script, that checks wether data is already there and if not executes a full indexing before starting the searchd process with the following command: 'searchd -c /etc/manticoresearch/manticore.conf --nodetach --logdebugv'
I have two indexer scripts. One that full indexes everything and one that executes only the delta indexes. In Both scripts I call a php script that dynamically rebuilds the manticore.conf file before executing the indexers, to ensure all sources are available.
My general manticore.conf has data like this. I added one Index example of multiple to give an idea of the setup:
Bug description:
This setup generally worked already with small data amounts, but with large data it crashes reliably.
The initial indexing works without issues. It takes about two hours to run trough. and afterwards the searchd process gets started. After that, triggering any delta index crashes searchd and the whole container. I have the feeling that the rotate step somehow doesn't run trough reliable.
One of the last times i tried to execute the indexing I got more or less the following output:
I also got a crash report with the following data:
I assume the issue is my setup and not a bug, but I'm on my wits end and don't know what else to check.
Manticore Search Version:
6.3.0