NVP_MSG (7564): mmap failed

[kukougu]

I run SplitFS for varmail of filebench in Centos 7(kernel 4.13) and I get the error below NVP_MSG (7564): mmap failed for Cannot allocate memory, mmap count 65367, addr -1, errno is 12 NVP_MSG (7564): Open count 1, close count 0 filebench: fileops_nvp.c:2917: nvp_get_dr_mmap_address: Assertion0' failed.`

I use the DC Optane PM (256G) and don't use PM Emulation just describe in https://github.com/utsaslab/SplitFS/blob/master/experiments.md/#kernel-setup

How to fix this problem? Thanks.

[rohankadekodi]

When running with Filebench, try changing the splitfs/common.mk file, by setting LEDGER_FILEBENCH=1 and LEDGER_PJD=0. Then recompile and run. This increases the size of the staging files for the filebench workloads, and reduces the number of memory-mappings.

[kukougu]

Thanks, it can work in most of filebench workloads by your configuration. But it get some error when I run webserver workload with multi threads (it can work with single thread) and the error is shown as below NVP_MSG (6455): _nvp_extend_write: mmap_addr = 0x7FB26C44FBE8, nvf->node->length = 331614, nvf->node->true_length = 2096. extent_length = 16450584 filereader: fileops_nvp.c:3482: _nvp_extend_write: Assertion0' failed.`

Thanks.

[rohankadekodi]

Can you share your webserver.f workload file which you are using, which fails? I will try to reproduce the issue on my end.

[kukougu]

I use the webserver in your code and just change the nthread to 10 set $nthreads=10 The output is NVP_MSG (12326): Initializing hub_init NVP_MSG (12326): _hub_init2: START NVP_MSG (12326): Initializing nvp_init NVP_MSG (12326): CLFLUSHOPT is supported! NVP_MSG (12326): init_append_log: Initializing append log NVP_MSG (12326): _nvp_init2: dr fd = 4, start addr = 0x7F3900400000 NVP_MSG (12326): _nvp_init2: dr fd = 5, start addr = 0x7F38FF200000 NVP_MSG (12326): _nvp_init2: Tbl mmaps set NVP_MSG (12326): _nvp_init2: Global locks created NVP_MSG (12326): _nvp_init2: initialized environment, OPEN_MAX = 1024 NVP_MSG (12326): _hub_init2: END Filebench Version 1.5-alpha3 0.000: Allocated 617MB of shared memory 0.001: Web-server Version 3.1 personality successfully loaded 0.001: Populating and pre-allocating filesets 0.001: logfiles populated: 1 files, avg. dir. width = 20, avg. dir. depth = 1.0, 0 leafdirs, 0.002MB total size 0.001: Removing logfiles tree (if exists) 0.001: Pre-allocating directories in logfiles tree 0.001: Pre-allocating files in logfiles tree 0.002: bigfileset populated: 1000 files, avg. dir. width = 20, avg. dir. depth = 3.3, 0 leafdirs, 14.842MB total size 0.002: Removing bigfileset tree (if exists) 0.002: Pre-allocating directories in bigfileset tree 0.003: Pre-allocating files in bigfileset tree 0.032: Waiting for pre-allocation to finish (in case of a parallel pre-allocation) 0.032: Population and pre-allocation of filesets completed 0.032: Starting 1 filereader instances NVP_MSG (12330): Initializing hub_init NVP_MSG (12330): _hub_init2: START NVP_MSG (12330): Initializing nvp_init NVP_MSG (12330): CLFLUSHOPT is supported! NVP_MSG (12330): init_append_log: Initializing append log NVP_MSG (12330): _nvp_init2: dr fd = 11, start addr = 0x7F6606400000 NVP_MSG (12330): _nvp_init2: dr fd = 12, start addr = 0x7F6605200000 NVP_MSG (12330): _nvp_init2: Tbl mmaps set NVP_MSG (12330): _nvp_init2: Global locks created NVP_MSG (12330): _nvp_init2: initialized environment, OPEN_MAX = 1024 NVP_MSG (12330): _hub_init2: END 0.126: Running... NVP_MSG (12330): _nvp_extend_write: mmap_addr = 0x7F6606419367, nvf->node->length = 105719, nvf->node->true_length = 2096. extent_length = 16673945 filereader: fileops_nvp.c:3482: _nvp_extend_write: Assertion `0' failed. 17.649: Unexpected Process termination Code 3, Errno 0 around line 82 18.129: NO VALID RESULTS! Filebench run terminated prematurely around line 82 NVP_MSG (12326): exit handler NVP_MSG (12326): Exit: print stats NVP_MSG (12326): ====================== NVP IO stats: ====================== NVP_MSG (12326): open 1002, close 1001, async close 0 NVP_MSG (12326): mmap 2, unlink 2, stat 2 NVP_MSG (12326): dr mmap 2, dr mmap critical path 0 NVP_MSG (12326): fsync 0, appendfsync: count 1001 size 0 average 0 NVP_MSG (12326): READ: count 1, size 3372, average 3372 NVP_MSG (12326): WRITE: count 1001, size 15564651, average 15549 NVP_MSG (12326): memcpy READ: count 0, size 0, average 0 NVP_MSG (12326): anon READ: count 0, size 0, average 0 NVP_MSG (12326): memcpy WRITE: count 1001, size 15564651, average 15549 NVP_MSG (12326): anon WRITE: count 1001, size 15564651, average 15549 NVP_MSG (12326): posix READ: count 1, size 3372, average 3372 NVP_MSG (12326): posix READ: count 1, size 3372, average 3372 NVP_MSG (12326): posix WRITE: count 0, size 0, average 0 NVP_MSG (12326): write extends 1001, total 1001

[rohankadekodi]

I have pushed a change to master, which works on my end. Also, make sure you run the following command before running filebench:

echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

Another thing to make sure while running filebench: in splitfs/common.mk file, set LEDGER_FILEBENCH=1 and LEDGER_PJD=0.

[kukougu]

Thanks, it can work now.