Closed Lot-Art closed 6 months ago
sanikolaev
commented
6 months ago Data with Minio according to your docs
sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore /home/ME/minio-binaries/issue-2097
I can't find issue-2097 on our side. Can you pls check it has been copied?
sanikolaev
commented
6 months ago How come it only use less then 1GB RAM when everything is mlock and preopened?
I see 27.8GB:
am I missing smth?
Lot-Art
commented
6 months ago htop memory
My bad, sorry.
Minio data on your amazon storage
This is from my history
696 curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o $HOME/minio-binaries/mc
697 chmod +x $HOME/minio-binaries/mc
700 ./mc config host add manticore http://s3.manticoresearch.com:9000 manticore manticore
715 sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore /home/ME/minio-binaries/issue-2097(so the last step is with sudo)
And the contents:
$ ls -alhR /home/ME/minio-binaries/issue-2097/manticore
/home/ME/minio-binaries/issue-2097/manticore:
total 519M
drwxr-xr-x 4 root root 4.0K Apr 27 15:43 .
drwxr-xr-x 3 root root 4.0K Apr 27 15:39 ..
-rw-r--r-- 1 root root 519M Apr 27 15:39 core
drwxr-xr-x 2 root root 12K Apr 27 15:43 la8_items_all
drwxr-xr-x 2 root root 4.0K Apr 27 15:43 la8_items_upcaft
-rw-r--r-- 1 root root 184 Apr 27 15:43 manticore.json
-rw-r--r-- 1 root root 0 Apr 27 15:43 state.sql
/home/ME/minio-binaries/issue-2097/manticore/la8_items_all:
total 40G
drwxr-xr-x 2 root root 12K Apr 27 15:43 .
drwxr-xr-x 4 root root 4.0K Apr 27 15:43 ..
-rw-r--r-- 1 root root 355M Apr 27 15:39 la8_items_all.1.spa
-rw-r--r-- 1 root root 272M Apr 27 15:39 la8_items_all.1.spb
-rw-r--r-- 1 root root 561M Apr 27 15:40 la8_items_all.1.spd
-rw-r--r-- 1 root root 17M Apr 27 15:39 la8_items_all.1.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:39 la8_items_all.1.sph
-rw-r--r-- 1 root root 183K Apr 27 15:39 la8_items_all.1.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:39 la8_items_all.1.spi
-rw-r--r-- 1 root root 302M Apr 27 15:39 la8_items_all.1.spidx
-rw-r--r-- 1 root root 466K Apr 27 15:39 la8_items_all.1.spm
-rw-r--r-- 1 root root 260M Apr 27 15:39 la8_items_all.1.spp
-rw-r--r-- 1 root root 20M Apr 27 15:39 la8_items_all.1.spt
-rw-r--r-- 1 root root 390M Apr 27 15:39 la8_items_all.10.spa
-rw-r--r-- 1 root root 314M Apr 27 15:40 la8_items_all.10.spb
-rw-r--r-- 1 root root 622M Apr 27 15:40 la8_items_all.10.spd
-rw-r--r-- 1 root root 18M Apr 27 15:39 la8_items_all.10.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:39 la8_items_all.10.sph
-rw-r--r-- 1 root root 183K Apr 27 15:39 la8_items_all.10.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:39 la8_items_all.10.spi
-rw-r--r-- 1 root root 380M Apr 27 15:40 la8_items_all.10.spidx
-rw-r--r-- 1 root root 511K Apr 27 15:39 la8_items_all.10.spm
-rw-r--r-- 1 root root 229M Apr 27 15:39 la8_items_all.10.spp
-rw-r--r-- 1 root root 21M Apr 27 15:39 la8_items_all.10.spt
-rw-r--r-- 1 root root 389M Apr 27 15:40 la8_items_all.11.spa
-rw-r--r-- 1 root root 313M Apr 27 15:40 la8_items_all.11.spb
-rw-r--r-- 1 root root 621M Apr 27 15:40 la8_items_all.11.spd
-rw-r--r-- 1 root root 18M Apr 27 15:39 la8_items_all.11.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:39 la8_items_all.11.sph
-rw-r--r-- 1 root root 183K Apr 27 15:39 la8_items_all.11.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:39 la8_items_all.11.spi
-rw-r--r-- 1 root root 375M Apr 27 15:40 la8_items_all.11.spidx
-rw-r--r-- 1 root root 510K Apr 27 15:39 la8_items_all.11.spm
-rw-r--r-- 1 root root 231M Apr 27 15:40 la8_items_all.11.spp
-rw-r--r-- 1 root root 21M Apr 27 15:39 la8_items_all.11.spt
-rw-r--r-- 1 root root 397M Apr 27 15:40 la8_items_all.12.spa
-rw-r--r-- 1 root root 318M Apr 27 15:40 la8_items_all.12.spb
-rw-r--r-- 1 root root 620M Apr 27 15:40 la8_items_all.12.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.12.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.12.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.12.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.12.spi
-rw-r--r-- 1 root root 377M Apr 27 15:40 la8_items_all.12.spidx
-rw-r--r-- 1 root root 521K Apr 27 15:40 la8_items_all.12.spm
-rw-r--r-- 1 root root 237M Apr 27 15:40 la8_items_all.12.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.12.spt
-rw-r--r-- 1 root root 395M Apr 27 15:40 la8_items_all.13.spa
-rw-r--r-- 1 root root 310M Apr 27 15:40 la8_items_all.13.spb
-rw-r--r-- 1 root root 614M Apr 27 15:40 la8_items_all.13.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.13.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.13.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.13.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.13.spi
-rw-r--r-- 1 root root 372M Apr 27 15:40 la8_items_all.13.spidx
-rw-r--r-- 1 root root 518K Apr 27 15:40 la8_items_all.13.spm
-rw-r--r-- 1 root root 237M Apr 27 15:40 la8_items_all.13.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.13.spt
-rw-r--r-- 1 root root 392M Apr 27 15:40 la8_items_all.14.spa
-rw-r--r-- 1 root root 312M Apr 27 15:40 la8_items_all.14.spb
-rw-r--r-- 1 root root 612M Apr 27 15:41 la8_items_all.14.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.14.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.14.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.14.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.14.spi
-rw-r--r-- 1 root root 367M Apr 27 15:40 la8_items_all.14.spidx
-rw-r--r-- 1 root root 515K Apr 27 15:40 la8_items_all.14.spm
-rw-r--r-- 1 root root 238M Apr 27 15:40 la8_items_all.14.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.14.spt
-rw-r--r-- 1 root root 393M Apr 27 15:40 la8_items_all.15.spa
-rw-r--r-- 1 root root 312M Apr 27 15:40 la8_items_all.15.spb
-rw-r--r-- 1 root root 611M Apr 27 15:41 la8_items_all.15.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.15.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.15.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.15.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.15.spi
-rw-r--r-- 1 root root 365M Apr 27 15:41 la8_items_all.15.spidx
-rw-r--r-- 1 root root 516K Apr 27 15:40 la8_items_all.15.spm
-rw-r--r-- 1 root root 250M Apr 27 15:40 la8_items_all.15.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.15.spt
-rw-r--r-- 1 root root 389M Apr 27 15:41 la8_items_all.16.spa
-rw-r--r-- 1 root root 308M Apr 27 15:41 la8_items_all.16.spb
-rw-r--r-- 1 root root 607M Apr 27 15:41 la8_items_all.16.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.16.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.16.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.16.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.16.spi
-rw-r--r-- 1 root root 359M Apr 27 15:41 la8_items_all.16.spidx
-rw-r--r-- 1 root root 510K Apr 27 15:40 la8_items_all.16.spm
-rw-r--r-- 1 root root 254M Apr 27 15:41 la8_items_all.16.spp
-rw-r--r-- 1 root root 21M Apr 27 15:40 la8_items_all.16.spt
-rw-r--r-- 1 root root 380M Apr 27 15:41 la8_items_all.2.spa
-rw-r--r-- 1 root root 290M Apr 27 15:41 la8_items_all.2.spb
-rw-r--r-- 1 root root 608M Apr 27 15:41 la8_items_all.2.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.2.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.2.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.2.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.2.spi
-rw-r--r-- 1 root root 322M Apr 27 15:41 la8_items_all.2.spidx
-rw-r--r-- 1 root root 499K Apr 27 15:41 la8_items_all.2.spm
-rw-r--r-- 1 root root 285M Apr 27 15:41 la8_items_all.2.spp
-rw-r--r-- 1 root root 21M Apr 27 15:41 la8_items_all.2.spt
-rw-r--r-- 1 root root 325M Apr 27 15:41 la8_items_all.20.spa
-rw-r--r-- 1 root root 247M Apr 27 15:41 la8_items_all.20.spb
-rw-r--r-- 1 root root 490M Apr 27 15:41 la8_items_all.20.spd
-rw-r--r-- 1 root root 15M Apr 27 15:41 la8_items_all.20.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.20.sph
-rw-r--r-- 1 root root 184K Apr 27 15:41 la8_items_all.20.sphi
-rw-r--r-- 1 root root 17M Apr 27 15:41 la8_items_all.20.spi
-rw-r--r-- 1 root root 286M Apr 27 15:41 la8_items_all.20.spidx
-rw-r--r-- 1 root root 426K Apr 27 15:41 la8_items_all.20.spm
-rw-r--r-- 1 root root 211M Apr 27 15:41 la8_items_all.20.spp
-rw-r--r-- 1 root root 18M Apr 27 15:41 la8_items_all.20.spt
-rw-r--r-- 1 root root 679M Apr 27 15:41 la8_items_all.21.spa
-rw-r--r-- 1 root root 519M Apr 27 15:41 la8_items_all.21.spb
-rw-r--r-- 1 root root 1.1G Apr 27 15:42 la8_items_all.21.spd
-rw-r--r-- 1 root root 31M Apr 27 15:41 la8_items_all.21.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.21.sph
-rw-r--r-- 1 root root 184K Apr 27 15:41 la8_items_all.21.sphi
-rw-r--r-- 1 root root 26M Apr 27 15:41 la8_items_all.21.spi
-rw-r--r-- 1 root root 583M Apr 27 15:42 la8_items_all.21.spidx
-rw-r--r-- 1 root root 892K Apr 27 15:41 la8_items_all.21.spm
-rw-r--r-- 1 root root 471M Apr 27 15:41 la8_items_all.21.spp
-rw-r--r-- 1 root root 37M Apr 27 15:41 la8_items_all.21.spt
-rw-r--r-- 1 root root 755M Apr 27 15:42 la8_items_all.22.spa
-rw-r--r-- 1 root root 584M Apr 27 15:42 la8_items_all.22.spb
-rw-r--r-- 1 root root 1.2G Apr 27 15:42 la8_items_all.22.spd
-rw-r--r-- 1 root root 36M Apr 27 15:41 la8_items_all.22.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.22.sph
-rw-r--r-- 1 root root 183K Apr 27 15:41 la8_items_all.22.sphi
-rw-r--r-- 1 root root 24M Apr 27 15:41 la8_items_all.22.spi
-rw-r--r-- 1 root root 645M Apr 27 15:42 la8_items_all.22.spidx
-rw-r--r-- 1 root root 991K Apr 27 15:41 la8_items_all.22.spm
-rw-r--r-- 1 root root 561M Apr 27 15:42 la8_items_all.22.spp
-rw-r--r-- 1 root root 41M Apr 27 15:41 la8_items_all.22.spt
-rw-r--r-- 1 root root 380M Apr 27 15:42 la8_items_all.3.spa
-rw-r--r-- 1 root root 297M Apr 27 15:41 la8_items_all.3.spb
-rw-r--r-- 1 root root 604M Apr 27 15:42 la8_items_all.3.spd
-rw-r--r-- 1 root root 18M Apr 27 15:41 la8_items_all.3.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.3.sph
-rw-r--r-- 1 root root 183K Apr 27 15:41 la8_items_all.3.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:41 la8_items_all.3.spi
-rw-r--r-- 1 root root 323M Apr 27 15:42 la8_items_all.3.spidx
-rw-r--r-- 1 root root 499K Apr 27 15:41 la8_items_all.3.spm
-rw-r--r-- 1 root root 283M Apr 27 15:42 la8_items_all.3.spp
-rw-r--r-- 1 root root 21M Apr 27 15:41 la8_items_all.3.spt
-rw-r--r-- 1 root root 379M Apr 27 15:42 la8_items_all.4.spa
-rw-r--r-- 1 root root 301M Apr 27 15:42 la8_items_all.4.spb
-rw-r--r-- 1 root root 613M Apr 27 15:42 la8_items_all.4.spd
-rw-r--r-- 1 root root 19M Apr 27 15:42 la8_items_all.4.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.4.sph
-rw-r--r-- 1 root root 183K Apr 27 15:42 la8_items_all.4.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.4.spi
-rw-r--r-- 1 root root 326M Apr 27 15:42 la8_items_all.4.spidx
-rw-r--r-- 1 root root 497K Apr 27 15:42 la8_items_all.4.spm
-rw-r--r-- 1 root root 282M Apr 27 15:42 la8_items_all.4.spp
-rw-r--r-- 1 root root 21M Apr 27 15:42 la8_items_all.4.spt
-rw-r--r-- 1 root root 375M Apr 27 15:42 la8_items_all.5.spa
-rw-r--r-- 1 root root 295M Apr 27 15:42 la8_items_all.5.spb
-rw-r--r-- 1 root root 600M Apr 27 15:42 la8_items_all.5.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.5.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.5.sph
-rw-r--r-- 1 root root 183K Apr 27 15:42 la8_items_all.5.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.5.spi
-rw-r--r-- 1 root root 327M Apr 27 15:42 la8_items_all.5.spidx
-rw-r--r-- 1 root root 493K Apr 27 15:42 la8_items_all.5.spm
-rw-r--r-- 1 root root 277M Apr 27 15:42 la8_items_all.5.spp
-rw-r--r-- 1 root root 21M Apr 27 15:42 la8_items_all.5.spt
-rw-r--r-- 1 root root 397M Apr 27 15:42 la8_items_all.6.spa
-rw-r--r-- 1 root root 312M Apr 27 15:42 la8_items_all.6.spb
-rw-r--r-- 1 root root 596M Apr 27 15:43 la8_items_all.6.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.6.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.6.sph
-rw-r--r-- 1 root root 184K Apr 27 15:42 la8_items_all.6.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.6.spi
-rw-r--r-- 1 root root 352M Apr 27 15:42 la8_items_all.6.spidx
-rw-r--r-- 1 root root 522K Apr 27 15:42 la8_items_all.6.spm
-rw-r--r-- 1 root root 262M Apr 27 15:42 la8_items_all.6.spp
-rw-r--r-- 1 root root 22M Apr 27 15:42 la8_items_all.6.spt
-rw-r--r-- 1 root root 387M Apr 27 15:42 la8_items_all.7.spa
-rw-r--r-- 1 root root 307M Apr 27 15:42 la8_items_all.7.spb
-rw-r--r-- 1 root root 595M Apr 27 15:43 la8_items_all.7.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.7.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.7.sph
-rw-r--r-- 1 root root 184K Apr 27 15:42 la8_items_all.7.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:42 la8_items_all.7.spi
-rw-r--r-- 1 root root 351M Apr 27 15:43 la8_items_all.7.spidx
-rw-r--r-- 1 root root 509K Apr 27 15:42 la8_items_all.7.spm
-rw-r--r-- 1 root root 258M Apr 27 15:43 la8_items_all.7.spp
-rw-r--r-- 1 root root 21M Apr 27 15:42 la8_items_all.7.spt
-rw-r--r-- 1 root root 394M Apr 27 15:43 la8_items_all.8.spa
-rw-r--r-- 1 root root 317M Apr 27 15:43 la8_items_all.8.spb
-rw-r--r-- 1 root root 601M Apr 27 15:43 la8_items_all.8.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.8.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.8.sph
-rw-r--r-- 1 root root 183K Apr 27 15:42 la8_items_all.8.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.8.spi
-rw-r--r-- 1 root root 377M Apr 27 15:43 la8_items_all.8.spidx
-rw-r--r-- 1 root root 517K Apr 27 15:42 la8_items_all.8.spm
-rw-r--r-- 1 root root 245M Apr 27 15:43 la8_items_all.8.spp
-rw-r--r-- 1 root root 22M Apr 27 15:42 la8_items_all.8.spt
-rw-r--r-- 1 root root 401M Apr 27 15:43 la8_items_all.9.spa
-rw-r--r-- 1 root root 320M Apr 27 15:43 la8_items_all.9.spb
-rw-r--r-- 1 root root 614M Apr 27 15:43 la8_items_all.9.spd
-rw-r--r-- 1 root root 18M Apr 27 15:43 la8_items_all.9.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_all.9.sph
-rw-r--r-- 1 root root 183K Apr 27 15:43 la8_items_all.9.sphi
-rw-r--r-- 1 root root 17M Apr 27 15:43 la8_items_all.9.spi
-rw-r--r-- 1 root root 391M Apr 27 15:43 la8_items_all.9.spidx
-rw-r--r-- 1 root root 526K Apr 27 15:43 la8_items_all.9.spm
-rw-r--r-- 1 root root 231M Apr 27 15:43 la8_items_all.9.spp
-rw-r--r-- 1 root root 22M Apr 27 15:43 la8_items_all.9.spt
-rw-r--r-- 1 root root 0 Apr 27 15:43 la8_items_all.lock
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_all.meta
-rw-r--r-- 1 root root 28 Apr 27 15:43 la8_items_all.ram
-rw-r--r-- 1 root root 152 Apr 27 15:43 la8_items_all.settings
/home/ME/minio-binaries/issue-2097/manticore/la8_items_upcaft:
total 604M
drwxr-xr-x 2 root root 4.0K Apr 27 15:43 .
drwxr-xr-x 4 root root 4.0K Apr 27 15:43 ..
-rw-r--r-- 1 root root 122M Apr 27 15:43 la8_items_upcaft.0.spa
-rw-r--r-- 1 root root 92M Apr 27 15:43 la8_items_upcaft.0.spb
-rw-r--r-- 1 root root 189M Apr 27 15:43 la8_items_upcaft.0.spd
-rw-r--r-- 1 root root 5.4M Apr 27 15:43 la8_items_upcaft.0.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_upcaft.0.sph
-rw-r--r-- 1 root root 174K Apr 27 15:43 la8_items_upcaft.0.sphi
-rw-r--r-- 1 root root 9.0M Apr 27 15:43 la8_items_upcaft.0.spi
-rw-r--r-- 1 root root 103M Apr 27 15:43 la8_items_upcaft.0.spidx
-rw-r--r-- 1 root root 160K Apr 27 15:43 la8_items_upcaft.0.spm
-rw-r--r-- 1 root root 79M Apr 27 15:43 la8_items_upcaft.0.spp
-rw-r--r-- 1 root root 6.6M Apr 27 15:43 la8_items_upcaft.0.spt
-rw-r--r-- 1 root root 0 Apr 27 15:43 la8_items_upcaft.lock
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_upcaft.meta
-rw-r--r-- 1 root root 28 Apr 27 15:43 la8_items_upcaft.ram
-rw-r--r-- 1 root root 152 Apr 27 15:43 la8_items_upcaft.settingsSo only root can write. Does that cause a problem?
Lot-Art
commented
6 months ago Actually, diskchunk save (FLUSH RAMCHUNK) also does some damage. Here is the searchd.log
[Sun Apr 28 00:13:29.169 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=108.017 Mb, time delta=15608 sec, took=0.803 sec)
[Sun Apr 28 00:43:09.907 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=61.208 Mb, time delta=15680 sec, took=0.309 sec)
[Sun Apr 28 02:27:57.087 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=133.525 Mb, time delta=8067 sec, took=0.882 sec)
[Sun Apr 28 02:52:41.109 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=72.313 Mb, time delta=7771 sec, took=0.398 sec)
[Sun Apr 28 05:29:14.738 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=152.845 Mb, time delta=10877 sec, took=1.059 sec)
[Sun Apr 28 06:20:37.889 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=89.146 Mb, time delta=12476 sec, took=0.555 sec)
[Sun Apr 28 11:10:18.047 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=190.684 Mb, time delta=20463 sec, took=1.682 sec)
[Sun Apr 28 12:19:31.129 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=105.370 Mb, time delta=21533 sec, took=0.800 sec)
[Sun Apr 28 14:20:17.890 2024] [47321] rt: table la8_items_all: diskchunk 24(15), segments 32 saved in 66.333469 (66.991552) sec, RAM saved/new 715542869/0 ratio 0.950000 (soft limit 2040109465, conf limit 2147483648)
[Sun Apr 28 14:21:27.500 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 541020904, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:34:09.807 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 518257623, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:34:11.285 2024] [47319] WARNING: last message repeated 1 times
[Sun Apr 28 14:35:25.443 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:39:08.996 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 541020904, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:58:40.769 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 536228500, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:08:17.958 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 518254642, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:10:56.929 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=841.860 Mb, time delta=3039 sec, took=3.889 sec)
[Sun Apr 28 15:12:23.710 2024] [47321] rt: table la8_items_all: diskchunk 25(16), segments 28 forcibly saved in 81.681049 (81.720773) sec, RAM saved/new 883533090/0 ratio 0.950000 (soft limit 2040109465, conf limit 2147483648)
[Sun Apr 28 15:14:45.663 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 443784822, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:15:05.424 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 443784822, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:15:19.888 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 443784822, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:23:11.464 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 541016708, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:33:31.143 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:33:31.143 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:26.964 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:26.964 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:33.732 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:33.732 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:36:27.853 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
(.. it keeps goind like that)You can see after a diskchunk log the warnings are happening. After the 2nd diskchunk save 2 chunks report the same warning (and the log is right, the query does fail).
Should I open a separate issue for this (its the same data, same server, same version, same same same)?
Lot-Art
commented
6 months ago This is in searchd.log after a restart
[Tue Apr 30 22:42:06.976 2024] [469662] watchdog: main process 469663 forked ok
[Tue Apr 30 22:42:06.987 2024] [469663] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 10.164.0.36:9312 for sphinx and http(s)
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 127.0.0.1:9312 for sphinx and http(s)
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 10.164.0.36:9306 for mysql
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 127.0.0.1:9306 for mysql
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 127.0.0.1:9307 for VIP mysql
[Tue Apr 30 22:42:11.589 2024] [469665] WARNING: table 'la8_items_upcaft': disabled at the JSON config
[Tue Apr 30 22:42:11.590 2024] [469665] WARNING: table 'la8_items_upcaft': prealloc: failed to lock /var/lib/manticore/la8_items_upcaft/la8_items_upcaft.lock: Resource temporarily unavailable - NOT SERVING
[Tue Apr 30 22:42:11.592 2024] [469664] prereading 1 tables
[Tue Apr 30 22:42:11.637 2024] [469663] accepting connections
[Tue Apr 30 22:42:12.926 2024] [469667] WARNING: [BUDDY] invalid output, should be 'Buddy ver, started address:port', got '
Fatal error: Uncaught Manticoresearch\Buddy\Core\Error\ManticoreSearchClientError: Cannot connect to server at http://10.164.0.36:9312/sql?mode=raw in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:133
Stack trace:
#0 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(124): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+SETT...')
#1 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(95): Manticoresearch\Buddy\Base\Lib\QueryProcessor::fetchManticoreSettings()
#2 /usr/share/manticore/modules/manticore-buddy/src/main.php(30): Manticoresearch\Buddy\Base\Lib\QueryProcessor::init()
#3 {main}
thrown in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 133
'
[Tue Apr 30 22:42:12.941 2024] [469667] [BUDDY] restarting
[Tue Apr 30 22:42:14.073 2024] [469667] WARNING: [BUDDY] invalid output, should be 'Buddy ver, started address:port', got '
Fatal error: Uncaught Manticoresearch\Buddy\Core\Error\ManticoreSearchClientError: Cannot connect to server at http://10.164.0.36:9312/sql?mode=raw in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:133
Stack trace:
#0 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(139): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+VARI...')
#1 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(95): Manticoresearch\Buddy\Base\Lib\QueryProcessor::fetchManticoreSettings()
#2 /usr/share/manticore/modules/manticore-buddy/src/main.php(30): Manticoresearch\Buddy\Base\Lib\QueryProcessor::init()
#3 {main}
thrown in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 133
'
[Tue Apr 30 22:42:14.089 2024] [469667] [BUDDY] restarting
[Tue Apr 30 22:42:15.225 2024] [469667] WARNING: [BUDDY] invalid output, should be 'Buddy ver, started address:port', got '
Fatal error: Uncaught Manticoresearch\Buddy\Core\Error\ManticoreSearchClientError: Cannot connect to server at http://10.164.0.36:9312/sql?mode=raw in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:133
Stack trace:
#0 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(139): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+VARI...')
#1 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(95): Manticoresearch\Buddy\Base\Lib\QueryProcessor::fetchManticoreSettings()
#2 /usr/share/manticore/modules/manticore-buddy/src/main.php(30): Manticoresearch\Buddy\Base\Lib\QueryProcessor::init()
#3 {main}
thrown in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 133
'
[Tue Apr 30 22:42:15.226 2024] [469667] [BUDDY] restart amount of attempts (3) has been exceeded
[Tue Apr 30 22:42:39.258 2024] [469664] preread 1 tables in 27.665 sec"And cannot connect with mysql -h 0 -P 9306. It just waits forever (until i press CTRL+C to get back my terminal).
Then when I restart manticore again, this is the searchd.log
[Tue Apr 30 22:47:49.898 2024] [469779] watchdog: main process 469780 forked ok
[Tue Apr 30 22:47:49.899 2024] [469780] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
[Tue Apr 30 22:47:49.900 2024] [469780] listening on 10.164.0.36:9312 for sphinx and http(s)
[Tue Apr 30 22:47:49.900 2024] [469780] listening on 127.0.0.1:9312 for sphinx and http(s)
[Tue Apr 30 22:47:49.901 2024] [469780] listening on 10.164.0.36:9306 for mysql
[Tue Apr 30 22:47:49.901 2024] [469780] listening on 127.0.0.1:9306 for mysql
[Tue Apr 30 22:47:49.901 2024] [469780] listening on 127.0.0.1:9307 for VIP mysql
[Tue Apr 30 22:47:53.648 2024] [469782] WARNING: table 'la8_items_upcaft': disabled at the JSON config
[Tue Apr 30 22:47:53.648 2024] [469782] WARNING: table 'la8_items_upcaft': prealloc: failed to lock /var/lib/manticore/la8_items_upcaft/la8_items_upcaft.lock: Resource temporarily unavailable - NOT SERVING
[Tue Apr 30 22:47:53.650 2024] [469783] prereading 1 tables
[Tue Apr 30 22:47:53.689 2024] [469780] accepting connections
[Tue Apr 30 22:47:53.873 2024] [469784] [BUDDY] started v1.0.18 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://10.164.0.36:9312 --threads=4' at http://127.0.0.1:46475
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] Loaded plugins:
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] local:
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] extra:
[Tue Apr 30 22:47:58.081 2024] [469783] preread 1 tables in 4.432 secAt this point I can connect, but one of my indexes is "disabled at the JSON config" (I did not do that), and is not serving.
(note, i always shut it down with systemctl that has the --stopwait option in it
Here is manticore.json
{
"clusters": {
},
"indexes": {
"la8_items_all": {
"type": "rt",
"path": "la8_items_all"
},
"la8_items_upcaft": {
"type": "rt",
"path": "la8_items_upcaft"
}
}
}It seems like manticore.json is created every time searchd starts. Im not sure if this is the file the log talks about when when it says "disabled at the JSON config", but maybe i need to do something with it, like directory permissions, or set something in the main manticore.conf file.
Lot-Art
commented
6 months ago Hi, is there an update you could give me by any chance?
Did you receive the data?
Is there something I can do about table disabled at the JSON config?
BUDDY failing?
At this point manticore is very unstable: it crashes and even the restart fails (table disabled, and buddy). I don't know if its mlock, preopen_tables...
sanikolaev
commented
6 months ago Hi @Lot-Art
Sorry for the delay. Unfortunately, we still don't have issue-2097 on our side.
You did:
sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore /home/ME/minio-binaries/issue-2097which is wrong. Looks like you copied it to just /home/ME/minio-binaries/issue-2097, not to our server. The correct command according to the docs should be:
sudo /home/ME/minio-binaries/mc config host add manticore http://s3.manticoresearch.com:9000 manticore manticore
sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore/ manticore/write-only/issue-2097Is there something I can do about table disabled at the JSON config?
What's important in this case is the next line, e.g.:
[Sat Apr 27 15:49:51.252 2024] [46345] WARNING: table 'la8_items_all': prealloc: failed to lock /var/lib/manticore/la8_items_all/la8_items_all.lock: Resource temporarily unavailable - NOT SERVINGthen you need to find out why the searchd can't lock la8_items_all.lock:
BUDDY failing?
Unlikely in this case.
Lot-Art
commented
6 months ago I did the data with minio again, maybe this time i got it right. Could you please check?
sanikolaev
commented
6 months ago Could you please check?
Still the same :(
Lot-Art
commented
6 months ago Ok, now I really think i did it right. Please check again (sorry for wasting your time with my fiddles). With that it should be possible to reproduce the issue in my opening post (the OPTIMIZE fail).
sanikolaev
commented
6 months ago @Lot-Art I see the files now, thanks. We'll try to reproduce it locally.
Lot-Art
commented
6 months ago As suggested (thank you @sanikolaev ), i increased the ram to 48. Since then, I observed the following things in the log, twice:
[Sun May 12 03:42:51.000 2024] [10538] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=361.577 Mb, time delta=23784 sec, took=2.980 sec)
[Sun May 12 04:06:16.109 2024] [10538] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=143.798 Mb, time delta=23299 sec, took=1.142 sec)
[Sun May 12 09:55:22.579 2024] [10538] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=376.776 Mb, time delta=22351 sec, took=3.131 sec)
[Sun May 12 10:34:57.990 2024] [10538] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=165.109 Mb, time delta=23321 sec, took=1.374 sec)
[Sun May 12 16:09:54.323 2024] [10538] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=380.344 Mb, time delta=22471 sec, took=3.378 sec)
[Sun May 12 17:04:26.156 2024] [10538] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=193.560 Mb, time delta=23368 sec, took=1.825 sec)
[Sun May 12 20:54:16.765 2024] [10541] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=393.822 Mb, time delta=17062 sec, took=3.334 sec)
[Sun May 12 21:46:52.319 2024] [10541] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=222.336 Mb, time delta=16946 sec, took=1.790 sec)
[Mon May 13 01:43:35.532 2024] [10541] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=399.135 Mb, time delta=17358 sec, took=3.372 sec)
[Mon May 13 01:43:56.827 2024] [10541] rt: table la8_items_upcaft: diskchunk 2(3), segments 31 forcibly saved in 16.244772 (16.281248) sec, RAM saved/new 418954916/0 ratio 0.950000 (soft li
mit 2040109465, conf limit 2147483648)
[Mon May 13 01:44:00.913 2024] [10541] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=233.367 Mb, time delta=14228 sec, took=2.027 sec)
[Mon May 13 01:44:17.899 2024] [10538] rt: table la8_items_all: diskchunk 29(20), segments 27 forcibly saved in 11.942063 (11.975312) sec, RAM saved/new 245112316/0 ratio 0.950000 (soft lim
it 2040109465, conf limit 2147483648)
[Mon May 13 01:44:45.078 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 337615371, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:44:46.497 2024] [10535] WARNING: last message repeated 3 times
[Mon May 13 01:45:02.344 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413807, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:45:38.454 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 151275584, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:45:39.556 2024] [10535] WARNING: last message repeated 2 times
[Mon May 13 01:46:11.320 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 472550774, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:12.065 2024] [10538] WARNING: last message repeated 2 times
[Mon May 13 01:46:12.065 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 151133479, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:14.670 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 151162757, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:16.097 2024] [10535] WARNING: last message repeated 2 times
[Mon May 13 01:46:26.846 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413550, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:41.863 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 151136435, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:46.439 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437414452, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:49.657 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 472592565, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:54.534 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437412644, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:46:56.450 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 472552310, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:37.788 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437412907, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:37.789 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413037, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:37.798 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413807, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:37.802 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437416259, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:42.897 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 472551286, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:55.015 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 472559745, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:47:56.200 2024] [10535] WARNING: last message repeated 2 times
[Mon May 13 01:48:04.908 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 274251914, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:48:06.711 2024] [10535] WARNING: last message repeated 1 times
[Mon May 13 01:48:08.251 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 337617180, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:48:09.714 2024] [10535] WARNING: last message repeated 3 times
[Mon May 13 01:48:29.756 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 443184151, from mm
ap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:48:29.757 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 443169114, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:48:29.791 2024] [10541] WARNING: last message repeated 2 times
[Mon May 13 01:48:29.791 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 443185888, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:48:39.127 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 151141832, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:48:57.062 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413807, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:49:13.321 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 142121372, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:49:38.005 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437412907, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:49:38.007 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413037, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:49:38.017 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413807, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:49:38.020 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437416259, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:50:04.077 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 337615238, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:50:04.287 2024] [10538] WARNING: last message repeated 2 times
[Mon May 13 01:50:04.287 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413678, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:50:41.920 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437420016, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:50:43.372 2024] [10535] WARNING: last message repeated 2 times
[Mon May 13 01:52:36.385 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437412907, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:52:36.387 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413037, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:52:36.396 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437413807, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:52:36.400 2024] [10541] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 437416259, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:52:41.390 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 268704514, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:52:41.400 2024] [10538] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_upcaft/la8_items_upcaft.2.spd' at 472548594, from mmap of 105250652, query most probably would FAIL; report the fact to dev!
[Mon May 13 01:53:01.199 2024] [10535] caught SIGTERM, shutting down
[Mon May 13 01:53:04.238 2024] [10535] WARNING: still 2 alive tasks during shutdown, after 3.039 sec
[Mon May 13 01:53:04.242 2024] [10538] rt: table la8_items_all: ramchunk saved in 0.002 sec
[Mon May 13 01:57:31.209 2024] [10534] watchdog: main process 10535 killed cleanly with SIGKILL, shutting down
[Mon May 13 01:57:31.236 2024] [792221] watchdog: main process 792222 forked ok
[Mon May 13 01:57:31.247 2024] [792222] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
[Mon May 13 01:57:31.247 2024] [792222] listening on all interfaces for sphinx and http(s), port=9312
[Mon May 13 01:57:31.247 2024] [792222] listening on all interfaces for mysql, port=9306
[Mon May 13 01:57:31.247 2024] [792222] listening on 127.0.0.1:9307 for VIP mysql
[Mon May 13 01:57:31.247 2024] [792222] listening on all interfaces for sphinx and http(s), port=9308
[Mon May 13 01:57:32.993 2024] [792224] prereading 2 tables
[Mon May 13 01:57:33.010 2024] [792222] accepting connections
[Mon May 13 01:57:33.124 2024] [792228] [BUDDY] started v1.0.18 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://0.0.0.0:9312 --threads=6' at http://127.0.0.1:44895
[Mon May 13 01:57:33.167 2024] [792226] [BUDDY] Loaded plugins:
[Mon May 13 01:57:33.167 2024] [792226] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Mon May 13 01:57:33.167 2024] [792226] [BUDDY] local:
[Mon May 13 01:57:33.167 2024] [792226] [BUDDY] extra:
[Mon May 13 01:57:40.626 2024] [792224] preread 2 tables in 7.633 sec@Lot-Art can you please share your actual configuration file?
sanikolaev
commented
6 months ago @Lot-Art pls also share your full searchd log.
sanikolaev
commented
6 months ago Your query log will be also helpful.
Lot-Art
commented
6 months ago searchd.log (that is be beginning yes, I had to empty it at one point because it got big with those warnings) manticore_conf.txt (had to make it .txt to upload here) Query logging is off, sorry.
So OPTIMIZE works in your environment... Ok, so i guess not much can be done with that.
How about this...
I provide here a script that adds data to the indexes: fill_manti_sh.txt (it should be .sh but i had to make it .txt to upload here) It also does UPDATE and DELETE statements.
Maybe the DELETE should happen more often to make killed_rate above 60% (but the script only deletes from the ones it added, not from the ones there were already indexed before the script started).
Could you please run it for about about ~400.000 docs (add ~400Mb):
./fill_manti.sh 1 400000
1st arg is starting id (sorry you probably need to check existing max id on both indexes before running it)
2nd arg is how many docs to add
Then do the following on both indexes:
sanikolaev
commented
6 months ago Thanks for the script and the other details, @Lot-Art . Unfortunately, I still can't reproduce the out-of-range warning or any other issues.
[Tue May 14 14:41:13.665 2024] [1478993] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
[Tue May 14 14:41:13.665 2024] [1478993] listening on all interfaces for mysql, port=9315
[Tue May 14 14:41:13.665 2024] [1478993] listening on all interfaces for sphinx and http(s), port=9316
[Tue May 14 14:41:16.694 2024] [1479009] prereading 2 tables
[Tue May 14 14:41:16.694 2024] [1478993] accepting connections
[Tue May 14 14:41:18.572 2024] [1479009] preread 2 tables in 1.878 sec
[Wed May 15 02:17:56.646 2024] [1479031] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=508.910 Mb, time delta=41800 sec, took=3.898 sec)
[Wed May 15 02:17:58.479 2024] [1479025] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=311.292 Mb, time delta=41804 sec, took=1.833 sec)
[Wed May 15 02:18:30.944 2024] [1479021] rt: table la8_items_all: diskchunk 23(16), segments 28 forcibly saved in 8.040073 (8.110920) sec, RAM saved/new 533412243/0 ratio 0.950000 (soft limit 2040109465, conf limit 2147483648)
[Wed May 15 02:18:36.283 2024] [1479012] rt: table la8_items_upcaft: diskchunk 1(2), segments 30 forcibly saved in 5.264019 (5.293163) sec, RAM saved/new 326955527/0 ratio 0.950000 (soft limit 2040109465, conf limit 2147483648)
[Wed May 15 02:21:23.618 2024] [1479027] rt: table la8_items_upcaft: optimized progressive chunk(s) 1 ( left 1 ) in 38s 743ms
[Wed May 15 02:24:15.469 2024] [1479030] rt: table la8_items_all: optimized progressive chunk(s) 1 ( left 15 ) in 1m 33.4sBut for the warning it's important to have the queries they are linked with. A dummy select * from la8_items_all, la8_items_upcaft where match('cat dog') facet house_match_str facet struct_category facet catalog_id facet scrapers may be not enough, so here's what I recommend:
query_log_commands=1 to the config, so we have not only selects in the query logset global log_level=debug or searchd --logdebug, this may be useful to do it via systemctl)This way we can better understand the circumstances of the warning.
Lot-Art
commented
6 months ago Ok, thank you, i will get to that. This thread became long, i will close it and will make a new one when i have all that new info.
Lot-Art
commented
5 months ago Ok, I changed server completely (CentOS, different CPU, and no resource monitoring tools), and it works. @sanikolaev it works... WORKS!!!! Ramchunk creation, diskchunk creation... even optimize works!
If I have to make a guess, then Google Cloud's debian 11 with resource monitor tools creates an environment that bothers/limits the RAM or the SSD.
Oh im such a happy camper, after 2 month my life is back to normal :)
sanikolaev
commented
5 months ago @Lot-Art Great! Interesting case.
The problem: OPTIMIZEing an RT index fails, then manticore says the table doesn't exist.
Debian, 4CPU, 32RAM There is only 1 server, running 1 manticore instance with 2 RT indexes (and this issue is talking about 1 of the indexes only). Only using the mysql interface. There are no nodes, no distribution... its a simple setup.
conf
_(Note that
rt_flush_periodis unorthodox but its fine.rt_mem_limitstill applies, it just takes 3 minutes which is ok for now. SoFLUSH RTINDEXandFLUSH RAMCHUNKare fine on both indexes.)_create rt index
The table was filled with lots of bulk insert statments (sql). Then i did a clean restart. From searchd.log:
table status
indextool
Trying to OPTIMIZE... (it takes a few minutes to get the "ERROR 2013"). After the fail it says "requires existing table" as if the table didn't exist.
Then this is in searchd.log (the first line below is the same as the last line of the above searchd.log quote, just to show continuity). You can see around the 4 WARNINGs it goes a bit crazy.
After that, if I restart manticore, the tables come back (but the nr of
disk_chunksis still 15).Data with Minio according to your docs
RAM usage is not what i expected
Maybe this is not related, but
htoplooks like this:Questions about that htop: The data files are many GB in size. How come it only use less then 1GB RAM when everything is
mlockand preopened? (The high CPU is probably because i do lots of write and read operations.)Manticore Search Version:
Manticore 6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)
Operating System Version:
Debian GNU/Linux 11 (bullseye)
Have you tried the latest development version?
Internal Checklist:
To be completed by the assignee. Check off tasks that have been completed or are not applicable.