Manticore requires more RAM and disk iops for the same performance as Elasticsearch

[githubmanticore]

The following shows that in some cases Manticore requires more RAM than Elasticsearch to provide the same performance. The task is to figure out what we can do to optimize it.

The test is currently deployed in /home/snikolaev/docker_repo/hn on our dev. server.

How to reproduce anywhere:

• git clone git@gitlab.com:manticoresearch/docker.git manticore_docker
• cd manticore_docker/hn
• docker-compose build
• chmod 777 es/volume/
• comment out

#    mem_limit: 1600m   
#    memswap_limit: 1600m   
#      - "ES_JAVA_OPTS=-Xms94m -Xmx94m"   

• docker-compose up elasticsearch
• wait for Elasticsearch to start
• docker-compose up logstash
• wait for

[root@sehs01 ~]# curl -s -XGET "http://localhost:9200/full/_stats?pretty"|grep count|head -1   
        "count" : 206998,   

to stop increasing (at 11654351)

• ./prepare.sh to merge all Lucene index segments into one
• disable query cache for the elastic index:

curl -XPOST "http://localhost:9200/full/_close"   
curl -XPUT "http://localhost:9200/full/_settings" -H 'Content-Type: application/json' -d '{   
    "index" : {   
         "queries.cache.enabled" : false   
    }   
}'   
curl -XPOST "http://localhost:9200/full/_open"   

• docker-compose down
• uncomment mem_limit, memswap_limit and ES_JAVA_OPTS in docker-compose.yml
• docker-compose up elasticsearch
• docker-compose up manticore, change the port in docker-compose.yml if the default one is in use, e.g.:

-    ports:   
-      - "9315:9314"   
     ports:   
       - "9415:9314"   

In this case update the port in test_manticore.sh as well.

• Run Manticore tests by running test_manticore.sh.\ What the tests do:
  • vmtouch -e to make sure the host's OS doesn't keep any index files in cache
  • shows current memory and heap limits
  • runs 3 times 1000 top queries from hn_top.txt, each time showing:
  • current memory stats
  • current OS caching stats for the index files
  • disk IO stats (in Ubuntu, for some reason it's empty in centos 7)\ E.g.:

[root@sehs01 hn]# ./test_manticore.sh   
Tue Dec 25 00:20:29 EST 2018   
evicting cached index files   
           Files: 13   
     Directories: 1   
   Evicted Pages: 1195149 (4G)   
         Elapsed: 0.002796 seconds   
-- limits:   
            "Memory": 4194304000,   
            "KernelMemory": 0,   
            "MemoryReservation": 0,   
            "MemorySwap": 4194304000,   
            "MemorySwappiness": 0,   
-- stats before test:   
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS   
984303d33b2b        manticore           0.77%               374MiB / 3.906GiB   9.35%               2.46MB / 4.59MB     4.68GB / 8.19kB     55   
cache 4812800   
rss 383868928   
total_cache 4812800   
total_rss 383868928   
           Files: 13   
     Directories: 1   
  Resident Pages: 12926/1195149  50M/4G  1.08%   
         Elapsed: 0.032806 seconds   
-- test:   
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................   
real0m55.574s   
user0m2.262s   
sys0m1.529s   
-- stats after test:   
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS   
984303d33b2b        manticore           0.72%               379.9MiB / 3.906GiB   9.50%               3.27MB / 6.13MB     9.33GB / 8.19kB     55   
cache 2329702400   
rss 383938560   
total_cache 2329702400   
total_rss 383938560   
Total 0   
           Files: 13   
     Directories: 1   
  Resident Pages: 580631/1195149  2G/4G  48.6%   
         Elapsed: 0.11281 seconds   
-- stats before test:   
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS   
984303d33b2b        manticore           0.76%               379.8MiB / 3.906GiB   9.50%               3.27MB / 6.13MB     9.33GB / 8.19kB     55   
cache 2329702400   
rss 383938560   
total_cache 2329702400   
total_rss 383938560   
           Files: 13   
     Directories: 1   
  Resident Pages: 580631/1195149  2G/4G  48.6%   
         Elapsed: 0.11327 seconds   
-- test:   
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................   
real0m47.390s   
user0m2.273s   
sys0m1.610s   
-- stats after test:   
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS   
984303d33b2b        manticore           0.80%               379.8MiB / 3.906GiB   9.49%               4.09MB / 7.66MB     9.33GB / 8.19kB     55   
cache 2329702400   
rss 383991808   
total_cache 2329702400   
total_rss 383991808   
Total 0   
           Files: 13   
     Directories: 1   
  Resident Pages: 580631/1195149  2G/4G  48.6%   
         Elapsed: 0.09545 seconds   
-- stats before test:   
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS   
984303d33b2b        manticore           0.75%               379.8MiB / 3.906GiB   9.49%               4.09MB / 7.66MB     9.33GB / 8.19kB     55   
cache 2329702400   
rss 383991808   
total_cache 2329702400   
total_rss 383991808   
           Files: 13   
     Directories: 1   
  Resident Pages: 580631/1195149  2G/4G  48.6%   
         Elapsed: 0.095387 seconds   
-- test:   
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................   
real0m47.133s   
user0m2.264s   
sys0m1.593s   
-- stats after test:   
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS   
984303d33b2b        manticore           0.84%               379.8MiB / 3.906GiB   9.49%               4.91MB / 9.19MB     9.33GB / 8.19kB     55   
cache 2329702400   
rss 383909888   
total_cache 2329702400   
total_rss 383909888   
Total 0   
           Files: 13   
     Directories: 1   
  Resident Pages: 580631/1195149  2G/4G  48.6%   
         Elapsed: 0.097005 seconds   
-------------------------   

• Run Elasticsearch tests by running test_elastic.sh
• Modify the limits (mem limit and heap size for Elastic / Manticore) to run more tests

What I see is:

• sehs01:
  • Elastic with 1600M memory limit 94M heap size gives:
  • 1m6s, 1m1s, 59s test execution times
  • 669M RSS 930M cached by OS after all
  • 6.88GB read from disk
  • Manticore with 1600M memory limit gives:
  • 59s, 59s, 59s test execution times
  • 378M RSS 1221M cached by OS
  • 14.7GB read from disk
• our dev server:
  • Elastic with 1600M memory limit 94M heap size gives:
  • 44s, 28s, 28s test execution times
  • 467M RSS 1034M cached by OS after all
  • 1.94GB read from disk
  • 17584 iops as reported by cgroups
  • Manticore with 1600M memory limit gives:
  • 43s, 43s, 43s test execution times
  • 365M RSS 1227M cached by OS
  • 7.36GB read from disk
  • 53512 iops
  • Manticore with 3000M memory limit gives:
  • 42s, 28s, 28s test execution times
  • 365M RSS 2217M cache
  • 2.7GB read from disk
  • 19763 iops

Information about Lucene index format can be found here:

• https://lucene.apache.org/core/5_2_0/core/org/apache/lucene/codecs/lucene50/Lucene50PostingsFormat.html
• https://youtu.be/T5RmMNDR5XI. Highlights:
  • https://youtu.be/T5RmMNDR5XI?t=2454 - "what happens when I run a TermQuery"
  • https://youtu.be/T5RmMNDR5XI?t=2873 - posting list compression: delta encoding bit packing vInt
  • https://youtu.be/T5RmMNDR5XI?t=3315 - "Pulse" optimizatiom (doc id right in the terms file for one doc terms)
• https://youtu.be/yst6VQ7Lwpo?t=1457 - about Luke - a tool which provides a UI to look at Lucene index internals

[githubmanticore]

➤ Sergey Nikolaev commented:

Tested the version which does mmap for .spd/.spp:

• without limit - 38/24.8/24.8 sec
• with limit - 37.7/39/39 sec

A little bit strange that the first run is faster than 2nd and 3rd.

So in both cases better than w/o mmap, but unfortunately mmap can't solve the main problem of this test. Under the memory constraint elastic is still more optimal.