search

quickwit-oss / tantivy

Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust
MIT License
11.79k stars 651 forks source link

Potential memory leak when adding documents to Tantivy #2481

Open softhuafei opened 2 weeks ago

softhuafei commented 2 weeks ago

I have observed a potential memory leak issue when adding documents to Tantivy. After committing the first batch of documents, the resident memory (RES) is around 18MB. After committing the second batch, the RES increases to approximately 36MB. Even after the Index and IndexWriter instances are dropped and the writing thread is terminated, the allocated memory does not seem to be released.

Here's a minimal reproducible example:

use tantivy::schema::*;
use tantivy::{doc, Index, IndexWriter};
use tempfile::TempDir;
use std::{thread, time};

fn main() -> tantivy::Result<()> {
    println!("PID {}", std::process::id());
    let index_path = TempDir::new()?;
    {
        // First we need to define a schema ...
        let mut schema_builder = Schema::builder();
        schema_builder.add_text_field("title", TEXT | STORED);
        schema_builder.add_text_field("body", TEXT);
        let schema = schema_builder.build();
        let index = Index::create_in_dir(&index_path, schema.clone())?;
        let mut index_writer: IndexWriter = index.writer(15_000_000)?;
        let title = schema.get_field("title").unwrap();
        let body = schema.get_field("body").unwrap();
        for _ in 1..100000 {
            index_writer.add_document(doc!(
                    title => "Frankenstein",
                    body => "You will rejoice to hear that no disaster has accompanied the commencement of an \
                             enterprise which you have regarded with such evil forebodings.  I arrived here \
                             yesterday, and my first task is to assure my dear sister of my welfare and \
                             increasing confidence in the success of my undertaking."
                    ))?;
        }
        println!("Block 1 commiting...");
        index_writer.commit()?;
    }
    println!("leaving block 1, sleep 5s");
    let ten_millis = time::Duration::from_millis(1000 * 10);
    thread::sleep(ten_millis);
    {
        println!("Insert second batch");
        let index = Index::open_in_dir(&index_path)?;
        let mut index_writer: IndexWriter = index.writer(15_000_000)?;
        let schema = index.schema();

        let title = schema.get_field("title").unwrap();
        let body = schema.get_field("body").unwrap();

        for _ in 1..100000 {
            index_writer.add_document(doc!(
                    title => "Frankenstein",
                    body => "You will rejoice to hear that no disaster has accompanied the commencement of an \
                             enterprise which you have regarded with such evil forebodings.  I arrived here \
                             yesterday, and my first task is to assure my dear sister of my welfare and \
                             increasing confidence in the success of my undertaking."
                    ))?;
        }
        println!("Block 2 commiting...");
        index_writer.commit()?;
    }
    println!("leaving block 2, sleep 20s");
    let ten_millis = time::Duration::from_millis(1000 * 10);
    thread::sleep(ten_millis);
    Ok(())
}

After running this code, the resident memory does not seem to be released even after the Index and IndexWriter instances are dropped, and the writing thread is terminated.

Could you please investigate the root cause of this potential memory leak and provide guidance on how to properly manage memory when working with Tantivy?

Thank you for your attention to this issue.

fulmicoton commented 2 weeks ago

Your allocator could be the problem here.

@softhuafei can you try and use jemalloc and check its internal metrics? It exposes the amount of allocated memory. This should be more accurate than resident memory.

softhuafei commented 2 weeks ago

Thank you for your response. I'am try use jemalloc and check it‘s profile. This is the new code and results:

[package]
name = "tantivy_test"
version = "0.1.0"
edition = "2021"

[dependencies]
tantivy = "0.22.0"
jemallocator = "0.3.2"
jemalloc-ctl = "0.3.2"
tempfile = "3.3.0"

[dependencies.jemalloc-sys]
version = "0.3.2"
features = ["stats", "profiling", "unprefixed_malloc_on_supported_platforms"]

[profile.release]
debug = true

// code

use tantivy::schema::*;
use tantivy::{doc, Index, IndexWriter};
use tempfile::TempDir;
use jemallocator;
use jemalloc_ctl::{AsName, Access};
use std::collections::HashMap;
use std::{thread, time};

#[global_allocator]
static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;

const PROF_ACTIVE: &'static [u8] = b"prof.active\0";
const PROF_DUMP: &'static [u8] = b"prof.dump\0";
const PROFILE_OUTPUT: &'static [u8] = b"profile.out\0";

fn set_prof_active(active: bool) {
    let name = PROF_ACTIVE.name();
    name.write(active).expect("Should succeed to set prof");
}

fn dump_profile() {
    let name = PROF_DUMP.name();
    name.write(PROFILE_OUTPUT).expect("Should succeed to dump profile")
}

fn main() -> tantivy::Result<()> {
    set_prof_active(true);
    println!("PID {}", std::process::id());
    let index_path = TempDir::new()?;

    // First we need to define a schema ...
    let mut schema_builder = Schema::builder();

    schema_builder.add_text_field("title", TEXT | STORED);

    schema_builder.add_text_field("body", TEXT);

    let schema = schema_builder.build();

    let index = Index::create_in_dir(&index_path, schema.clone())?;

    let mut index_writer: IndexWriter = index.writer(50_000_000)?;

    let title = schema.get_field("title").unwrap();
    let body = schema.get_field("body").unwrap();

    for _ in 1..100000 {
        index_writer.add_document(doc!(
                title => "Frankenstein",
                body => "You will rejoice to hear that no disaster has accompanied the commencement of an \
                            enterprise which you have regarded with such evil forebodings.  I arrived here \
                            yesterday, and my first task is to assure my dear sister of my welfare and \
                            increasing confidence in the success of my undertaking."
                ))?;
    }
    println!("commiting...");
    index_writer.commit()?;

    std::mem::drop(index);
    std::mem::drop(index_writer);
    set_prof_active(false);
    dump_profile();

    println!("After drop, sleep 20s");
    let ten_millis = time::Duration::from_millis(1000 * 30);
    thread::sleep(ten_millis);

    Ok(())
}
image image image image image image

The main memory allocation comes from IndexMerger write(), which seems normal. However, why doesn't the RES decrease after dropping the Index and IndexWriter? If this is a long-running process, could the RES keep increasing as the service progresses, eventually leading to an OOM ?

PSeitz commented 2 weeks ago

The allocator will reserve some memory and not release it back to the OS. That's not a memory leak and limited. I wrote something about that here: https://quickwit.io/blog/performance-investigation

softhuafei commented 2 weeks ago

The allocator will reserve some memory and not release it back to the OS. That's not a memory leak and limited. I wrote something about that here: https://quickwit.io/blog/performance-investigation

Thank you very much! I tested the behavior with and without jemalloc, this is my test code, it creates/writes the tantivy index three times repeatedly.


use tantivy::schema::*;
use tantivy::{doc, Index, IndexWriter};
use tempfile::TempDir;
use jemallocator;
use std::{thread, time};

#[global_allocator]
static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;

fn main() -> tantivy::Result<()> {
    // set_prof_active(true);
    println!("PID {}", std::process::id());

    for i in 1..4 {
        println!("{}-th", i);
        let index_path = TempDir::new()?;

        // First we need to define a schema ...
        let mut schema_builder = Schema::builder();

        schema_builder.add_text_field("title", TEXT | STORED);
        schema_builder.add_text_field("body", TEXT);
        let schema = schema_builder.build();
        let index = Index::create_in_dir(&index_path, schema.clone())?;
        // let index = Index::open_in_dir(&index_path)?;
        let mut index_writer: IndexWriter = index.writer(1000_000_000)?;

        let title = schema.get_field("title").unwrap();
        let body = schema.get_field("body").unwrap();

        for _ in 1..200000 {
            index_writer.add_document(doc!(
                    title => "Frankenstein",
                    body => "You will rejoice to hear that no disaster has accompanied the commencement of an \
                                enterprise which you have regarded with such evil forebodings.  I arrived here \
                                yesterday, and my first task is to assure my dear sister of my welfare and \
                                increasing confidence in the success of my undertaking."
                    ))?;
        }
        println!("commiting...");
        index_writer.commit()?;
        println!("wait merging...");
        index_writer.wait_merging_threads()?;
    }

    println!("After drop, sleep 120s");
    let ten_millis = time::Duration::from_millis(1000 * 120);
    thread::sleep(ten_millis);

    Ok(())
}

When using jemalloc, the behavior is similar to what you described in your blog. Jemalloc manages memory fragmentation and avoids repeated calls to mmap. The process only calls mmap in the first iteration. The second and third iterations do not call any mmap and call munmap before the process exits. The RES is stable at 100M～ during the whole process

strace -e trace=mmap,munmap,mremap,brk ./target/debug/tantivy_test
brk(NULL)                               = 0x55faf6c8f000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcf244b1000
mmap(NULL, 50, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x7fcf244b0000
mmap(NULL, 2133888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf242a7000
mmap(0x7fcf244ac000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7fcf244ac000
mmap(0x7fcf244ad000, 12160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fcf244ad000
munmap(0x7fcf244b0000, 50)              = 0
mmap(NULL, 44795, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fcf2429c000
mmap(NULL, 135600, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf2427a000
mmap(0x7fcf24281000, 65536, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fcf24281000
mmap(0x7fcf24291000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fcf24291000
mmap(0x7fcf24296000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b000) = 0x7fcf24296000
mmap(0x7fcf24298000, 12720, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fcf24298000
mmap(NULL, 103496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf24260000
mmap(0x7fcf24263000, 69632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fcf24263000
mmap(0x7fcf24274000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7fcf24274000
mmap(0x7fcf24278000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fcf24278000
mmap(NULL, 1331224, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf2411a000
mmap(0x7fcf24129000, 638976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7fcf24129000
mmap(0x7fcf241c5000, 626688, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xab000) = 0x7fcf241c5000
mmap(0x7fcf2425e000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x143000) = 0x7fcf2425e000
mmap(NULL, 24720, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf24113000
mmap(0x7fcf24115000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7fcf24115000
mmap(0x7fcf24117000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7fcf24117000
mmap(0x7fcf24118000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7fcf24118000
mmap(NULL, 1872544, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf23f49000
mmap(0x7fcf23f6f000, 1363968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7fcf23f6f000
mmap(0x7fcf240bc000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x173000) = 0x7fcf240bc000
mmap(0x7fcf24109000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bf000) = 0x7fcf24109000
mmap(0x7fcf2410f000, 12960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fcf2410f000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcf23f47000
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fcf23f43000
munmap(0x7fcf2429c000, 44795)           = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf244b0000
munmap(0x7fcf244b0000, 4096)            = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf23d43000
munmap(0x7fcf23d43000, 2097152)         = 0
mmap(NULL, 4190208, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf23b44000
munmap(0x7fcf23b44000, 770048)          = 0
munmap(0x7fcf23e00000, 1323008)         = 0
brk(NULL)                               = 0x55faf6c8f000
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf23a00000
mmap(NULL, 4194304, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf23600000
mmap(NULL, 270715632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf133d3000
mmap(0x7fcf135f0000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d000) = 0x7fcf135f0000
mmap(0x7fcf135f5000, 268479216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fcf135f5000
mmap(NULL, 44795, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fcf2429c000
mmap(NULL, 43520, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fcf23f38000
mmap(0x7fcf23f3b000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fcf23f3b000
mmap(0x7fcf23f3f000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fcf23f3f000
mmap(0x7fcf23f41000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7fcf23f41000
munmap(0x7fcf2429c000, 44795)           = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf242a4000
PID 105727
1-th
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf131d2000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf12bff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf127fe000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf11fff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf115ff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf109ff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf107fe000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf0ffff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf0f7ff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf0efff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf0e7ff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf0dfff000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fcf0d7ff000
mmap(NULL, 3145728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf08b00000
mmap(NULL, 3670016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fcf08780000
commiting...
wait merging...
2-th
commiting...
wait merging...
3-th
commiting...
wait merging...
After drop, sleep 120s
munmap(0x7fcf242a4000, 12288)           = 0

But for original malloc, For native malloc, the first two iterations have calls to brk, mmap, and munmap, and RES gradually increases as the iterations proceed：33M->82M->100M, and finally stabilizes at 100M.

$strace -e trace=mmap,munmap,mremap,brk ./target/debug/tantivy_test
brk(NULL)                               = 0x557b9e00b000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01ec144000
mmap(NULL, 50, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x7f01ec143000
mmap(NULL, 2133888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01ebf3a000
mmap(0x7f01ec13f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5000) = 0x7f01ec13f000
mmap(0x7f01ec140000, 12160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f01ec140000
munmap(0x7f01ec143000, 50)              = 0
mmap(NULL, 44795, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f01ebf2f000
mmap(NULL, 103496, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01ebf15000
mmap(0x7f01ebf18000, 69632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f01ebf18000
mmap(0x7f01ebf29000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x7f01ebf29000
mmap(0x7f01ebf2d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f01ebf2d000
mmap(NULL, 135600, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01ebef3000
mmap(0x7f01ebefa000, 65536, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f01ebefa000
mmap(0x7f01ebf0a000, 20480, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f01ebf0a000
mmap(0x7f01ebf0f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b000) = 0x7f01ebf0f000
mmap(0x7f01ebf11000, 12720, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f01ebf11000
mmap(NULL, 1331224, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01ebdad000
mmap(0x7f01ebdbc000, 638976, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf000) = 0x7f01ebdbc000
mmap(0x7f01ebe58000, 626688, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xab000) = 0x7f01ebe58000
mmap(0x7f01ebef1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x143000) = 0x7f01ebef1000
mmap(NULL, 24720, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01ebda6000
mmap(0x7f01ebda8000, 8192, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f01ebda8000
mmap(0x7f01ebdaa000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f01ebdaa000
mmap(0x7f01ebdab000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f01ebdab000
mmap(NULL, 1872544, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01ebbdc000
mmap(0x7f01ebc02000, 1363968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x26000) = 0x7f01ebc02000
mmap(0x7f01ebd4f000, 311296, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x173000) = 0x7f01ebd4f000
mmap(0x7f01ebd9c000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bf000) = 0x7f01ebd9c000
mmap(0x7f01ebda2000, 12960, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f01ebda2000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01ebbda000
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01ebbd7000
munmap(0x7f01ebf2f000, 44795)           = 0
brk(NULL)                               = 0x557b9e00b000
brk(0x557b9e02c000)                     = 0x557b9e02c000
mmap(NULL, 270715632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01db9aa000
mmap(0x7f01dbbc7000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1d000) = 0x7f01dbbc7000
mmap(0x7f01dbbcc000, 268479216, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f01dbbcc000
mmap(NULL, 44795, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f01ebf2f000
mmap(NULL, 43520, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f01db99f000
mmap(0x7f01db9a2000, 16384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f01db9a2000
mmap(0x7f01db9a6000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f01db9a6000
mmap(0x7f01db9a8000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f01db9a8000
munmap(0x7f01ebf2f000, 44795)           = 0
mmap(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01ebf37000
PID 76409
1-th
mmap(NULL, 1523712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f01db82b000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01db62a000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01db429000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01db228000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01db024000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01dae20000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01dac1c000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01daa18000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01da814000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01da610000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01da40f000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01da208000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01da004000
mmap(NULL, 2101248, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f01d9e00000
brk(NULL)                               = 0x557b9e02c000
brk(0x557b9e04d000)                     = 0x557b9e04d000
brk(NULL)                               = 0x557b9e04d000
brk(0x557b9e06e000)                     = 0x557b9e06e000
brk(NULL)                               = 0x557b9e06e000
brk(0x557b9e08f000)                     = 0x557b9e08f000
brk(NULL)                               = 0x557b9e08f000
brk(0x557b9e0b0000)                     = 0x557b9e0b0000
brk(NULL)                               = 0x557b9e0b0000
brk(0x557b9e0d1000)                     = 0x557b9e0d1000
brk(NULL)                               = 0x557b9e0d1000
brk(0x557b9e0f2000)                     = 0x557b9e0f2000
brk(NULL)                               = 0x557b9e0f2000
brk(0x557b9e113000)                     = 0x557b9e113000
brk(NULL)                               = 0x557b9e113000
brk(0x557b9e134000)                     = 0x557b9e134000
brk(NULL)                               = 0x557b9e134000
brk(0x557b9e155000)                     = 0x557b9e155000
brk(NULL)                               = 0x557b9e155000
brk(0x557b9e176000)                     = 0x557b9e176000
brk(NULL)                               = 0x557b9e176000
brk(0x557b9e197000)                     = 0x557b9e197000
brk(NULL)                               = 0x557b9e197000
brk(0x557b9e1b8000)                     = 0x557b9e1b8000
brk(NULL)                               = 0x557b9e1b8000
brk(0x557b9e1d9000)                     = 0x557b9e1d9000
brk(NULL)                               = 0x557b9e1d9000
brk(0x557b9e1fa000)                     = 0x557b9e1fa000
brk(NULL)                               = 0x557b9e1fa000
brk(0x557b9e21b000)                     = 0x557b9e21b000
brk(NULL)                               = 0x557b9e21b000
brk(0x557b9e23c000)                     = 0x557b9e23c000
brk(NULL)                               = 0x557b9e23c000
brk(0x557b9e25d000)                     = 0x557b9e25d000
brk(NULL)                               = 0x557b9e25d000
brk(0x557b9e27e000)                     = 0x557b9e27e000
brk(NULL)                               = 0x557b9e27e000
brk(0x557b9e29f000)                     = 0x557b9e29f000
brk(NULL)                               = 0x557b9e29f000
brk(0x557b9e2c0000)                     = 0x557b9e2c0000
brk(NULL)                               = 0x557b9e2c0000
brk(0x557b9e2e1000)                     = 0x557b9e2e1000
brk(NULL)                               = 0x557b9e2e1000
brk(0x557b9e302000)                     = 0x557b9e302000
brk(NULL)                               = 0x557b9e302000
brk(0x557b9e323000)                     = 0x557b9e323000
brk(NULL)                               = 0x557b9e323000
brk(0x557b9e344000)                     = 0x557b9e344000
brk(NULL)                               = 0x557b9e344000
brk(0x557b9e365000)                     = 0x557b9e365000
brk(NULL)                               = 0x557b9e365000
brk(0x557b9e386000)                     = 0x557b9e386000
brk(NULL)                               = 0x557b9e386000
brk(0x557b9e3a7000)                     = 0x557b9e3a7000
brk(NULL)                               = 0x557b9e3a7000
brk(0x557b9e3c8000)                     = 0x557b9e3c8000
brk(NULL)                               = 0x557b9e3c8000
brk(0x557b9e3e9000)                     = 0x557b9e3e9000
brk(NULL)                               = 0x557b9e3e9000
brk(0x557b9e40a000)                     = 0x557b9e40a000
brk(NULL)                               = 0x557b9e40a000
brk(0x557b9e42b000)                     = 0x557b9e42b000
brk(NULL)                               = 0x557b9e42b000
brk(0x557b9e44c000)                     = 0x557b9e44c000
brk(NULL)                               = 0x557b9e44c000
brk(0x557b9e46d000)                     = 0x557b9e46d000
brk(NULL)                               = 0x557b9e46d000
brk(0x557b9e48e000)                     = 0x557b9e48e000
brk(NULL)                               = 0x557b9e48e000
brk(0x557b9e4af000)                     = 0x557b9e4af000
brk(NULL)                               = 0x557b9e4af000
brk(0x557b9e4d0000)                     = 0x557b9e4d0000
brk(NULL)                               = 0x557b9e4d0000
brk(0x557b9e4f1000)                     = 0x557b9e4f1000
brk(NULL)                               = 0x557b9e4f1000
brk(0x557b9e512000)                     = 0x557b9e512000
brk(NULL)                               = 0x557b9e512000
brk(0x557b9e533000)                     = 0x557b9e533000
brk(NULL)                               = 0x557b9e533000
brk(0x557b9e554000)                     = 0x557b9e554000
brk(NULL)                               = 0x557b9e554000
brk(0x557b9e575000)                     = 0x557b9e575000
brk(NULL)                               = 0x557b9e575000
brk(0x557b9e596000)                     = 0x557b9e596000
brk(NULL)                               = 0x557b9e596000
brk(0x557b9e5b7000)                     = 0x557b9e5b7000
commiting...
mmap(NULL, 1523712, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f017c1e8000
wait merging...
munmap(0x7f017c1e8000, 1523712)         = 0
2-th
brk(NULL)                               = 0x557b9e5b7000
brk(0x557b9e5d8000)                     = 0x557b9e5d8000
brk(NULL)                               = 0x557b9e5d8000
brk(0x557b9e5f9000)                     = 0x557b9e5f9000
brk(NULL)                               = 0x557b9e5f9000
brk(0x557b9e61a000)                     = 0x557b9e61a000
brk(NULL)                               = 0x557b9e61a000
brk(0x557b9e63b000)                     = 0x557b9e63b000
brk(NULL)                               = 0x557b9e63b000
brk(0x557b9e65c000)                     = 0x557b9e65c000
brk(NULL)                               = 0x557b9e65c000
brk(0x557b9e67d000)                     = 0x557b9e67d000
brk(NULL)                               = 0x557b9e67d000
brk(0x557b9e69e000)                     = 0x557b9e69e000
brk(NULL)                               = 0x557b9e69e000
brk(0x557b9e6bf000)                     = 0x557b9e6bf000
brk(NULL)                               = 0x557b9e6bf000
brk(0x557b9e6e0000)                     = 0x557b9e6e0000
brk(NULL)                               = 0x557b9e6e0000
brk(0x557b9e701000)                     = 0x557b9e701000
brk(NULL)                               = 0x557b9e701000
brk(0x557b9e722000)                     = 0x557b9e722000
commiting...
brk(NULL)                               = 0x557b9e722000
brk(0x557b9e8a6000)                     = 0x557b9e8a6000
wait merging...
3-th
commiting...
wait merging...
After drop, sleep 120s