sanikolaev
commented
9 months ago I can't reproduce a crash in 6.2.12 with the loading script based on your schema:
```
#!/usr/bin/php
\n");
// This function waits for an idle mysql connection for the $query, runs it and exits
function process($query) {
global $all_links;
global $requests;
foreach ($all_links as $k=>$link) {
if (@$requests[$k]) continue;
mysqli_query($link, $query, MYSQLI_ASYNC);
@$requests[$k] = microtime(true);
return true;
}
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) {
$links[] = $errors[] = $reject[] = $link;
}
$count = @mysqli_poll($links, $errors, $reject, 0, 1000);
if ($count > 0) {
foreach ($links as $j=>$link) {
$res = @mysqli_reap_async_query($links[$j]);
foreach ($all_links as $i=>$link_orig) if ($all_links[$i] === $links[$j]) break;
if ($link->error) {
echo "ERROR: {$link->error}\n";
if (!mysqli_ping($link)) {
echo "ERROR: mysql connection is down, removing it from the pool\n";
unset($all_links[$i]); // remove the original link from the pool
unset($requests[$i]); // and from the $requests too
}
return false;
}
if ($res === false and !$link->error) continue;
if (is_object($res)) {
mysqli_free_result($res);
}
$requests[$i] = microtime(true);
mysqli_query($link, $query, MYSQLI_ASYNC); // making next query
return true;
}
};
} while (true);
return true;
}
$all_links = [];
$requests = [];
$c = 0;
for ($i=0;$i<$argv[2];$i++) {
$m = @mysqli_connect('127.0.0.1', '', '', '', 9306);
if (mysqli_connect_error()) die("Cannot connect to Manticore\n");
$all_links[] = $m;
}
// init
mysqli_query($all_links[0], "drop table if exists redacted_aaa");
mysqli_query($all_links[0], "CREATE TABLE redacted_aaa ( id bigint, entity_id text stored, name text indexed, description text indexed, notes text indexed, number text indexed, holder text indexed, is_deleted bool, schema string attribute, source string attribute, attributes json ) min_infix_len='2' index_exact_words='1' charset_table='non_cjk, U+47->0' min_word_len='2' blend_chars='@, /, +, -, ., _' blend_mode='trim_none, trim_both, skip_pure' morphology='stem_ru' min_stemming_len='3' expand_keywords='1'");
$batch = [];
$query_start = "insert into redacted_aaa(id, entity_id, name, description, notes, number, holder, is_deleted, schema, source, attributes) values ";
echo "preparing...\n";
$error = false;
$cache_file_name = '/tmp/'.md5($query_start).'_'.$argv[1].'_'.$argv[3];
$c = 0;
if (!file_exists($cache_file_name)) {
$batches = [];
while ($c < $argv[3]) {
$batch[] = "($c, '1234567890', 'john smith', 'Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum', 'Lorem Ipsum is simply dummy text of the printing and typesetting industry. Lorem Ipsum has been the industrys standard dummy text ever since the 1500s, when an unknown printer took a galley of type and scrambled it to make a type specimen book. It has survived not only five centuries, but also the leap into electronic typesetting, remaining essentially unchanged. It was popularised in the 1960s with the release of Letraset sheets containing Lorem Ipsum passages, and more recently with desktop publishing software like Aldus PageMaker including versions of Lorem Ipsum', '0123456789', 'holder1 holder2 holder3', 'true', '{\"a\": 123, \"b\": 345}', 'source', '{\"a\": 123, \"b\": {\"c\": 1.2, \"d\": true}}')";
$c++;
if (floor($c/1000) == $c/1000) echo "\r".($c/$argv[3]*100)."% ";
if (count($batch) == $argv[1]) {
$batches[] = $query_start.implode(',', $batch);
$batch = [];
}
}
if ($batch) $batches[] = $query_start.implode(',', $batch);
file_put_contents($cache_file_name, serialize($batches));
} else {
echo "found in cache $cache_file_name\n";
$batches = unserialize(file_get_contents($cache_file_name));
}
$batchesMulti = [];
for ($n=0;$n<$argv[4];$n++) $batchesMulti = array_merge($batchesMulti, $batches);
$batches = $batchesMulti;
echo "querying...\n";
$t = microtime(true);
foreach ($batches as $batch) {
if (!process($batch)) die("ERROR\n");
}
// wait until all the workers finish
do {
$links = $errors = $reject = array();
foreach ($all_links as $link) $links[] = $errors[] = $reject[] = $link;
$count = @mysqli_poll($links, $errors, $reject, 0, 100);
} while (count($all_links) != count($links) + count($errors) + count($reject));
echo "finished inserting\n";
echo "Total time: ".(microtime(true) - $t)."\n";
echo round($argv[3] * $argv[4] / (microtime(true) - $t))." docs per sec\n";
```
even with the higher concurrency of 8:
# php ~/load_1891.php 500 8 10000000 1
preparing...
100% querying...
finished inserting
Total time: 832.53445100784
12012 docs per secmysql> select count(*) from redacted_aaa;
+----------+
| count(*) |
+----------+
| 10000000 |
+----------+
1 row in set (0.00 sec)There was a somewhat similar issue https://github.com/manticoresoftware/manticoresearch/issues/1458#issuecomment-1790605768 which has already been fixed. I suggest you check if the crash persists in the latest dev version - https://mnt.cr/dev/nightly
You can also try modifying the script, so it reproduces the crash, so we can reproduce it on our end to fix it.
yharahuts
MirosOwners
bZichett
tomatolog
Describe the bug Manticore crashes when inserting a large amount of data into index.
Manticore is running in
rtmode with following tables:Data is inserted via (rather large?) batches of 500 records per single insert, and whole dataset contains about 100m rows splitted into 1-3 indexes. Crash happens randomly, data can be inserted without problems at all, or can crash at ~1-2% at random line.
Since it is prod instance, I'm afraid I can not give you our datasets, or test multiple (older?) manticore versions.
To Reproduce Steps to reproduce the behavior:
Expected behavior It should not crash.
Describe the environment:
searchd -v: Manticore 6.2.12 dc5144d35@230822manticoresearch/manticore:6.2.12Messages from log files:
docker logsshows following:After that it restarts with:
Additional context While writing this issue, I came up with two ideas:
decrease batch size to maybe 50 rows per single insert;didn't helpI'll try both options, but since crash is happening randomly - I couldnt guarantee it will work or not.
Any advices is greatly appreciated,
indextool --checkon both indexes returns: