search

pmem / issues

Old issues repo for PMDK.
http://pmem.io
13 stars 7 forks source link

Tests: pmempool_sync/TEST[18-21]: SETUP (all/pmem/debug) fail #1045

Closed Greg091 closed 5 years ago

Greg091 commented 5 years ago

ISSUE:

Environment Information

Please provide a reproduction of the bug:

./RUNTESTS pmempool_sync -s TEST18 ./RUNTESTS pmempool_sync -s TEST19 ./RUNTESTS pmempool_sync -s TEST20 ./RUNTESTS pmempool_sync -s TEST21

How often bug is revealed: (always, often, rare): always

Actual behavior:

./RUNTESTS pmempool_sync -s TEST18
pmempool_sync/TEST18: SETUP (check/pmem/debug)
error: pmemobj_root failed
pmempool_sync/TEST18 failed with exit code 1.
out18.log below.

pmem18.log below.
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:236 out_init] pid 62798: program: /home/jenkins/greg/pmdk/src/test/tools/pmemobjcli/pmemobjcli
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:238 out_init] libpmem version 1.1
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST18 pmem18.log <libpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [libpmem.c:56 libpmem_init]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [pmem.c:781 pmem_init]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:419 pmem_init_funcs]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:368 pmem_cpuinfo_to_funcs]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:372 pmem_cpuinfo_to_funcs] clflush supported
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:376 pmem_cpuinfo_to_funcs] clflushopt supported
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:388 pmem_cpuinfo_to_funcs] clwb supported
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:281 use_avx_memcpy_memset] avx supported
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:285 use_avx_memcpy_memset] PMEM_AVX not set or not == 1
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:325 use_avx512f_memcpy_memset] avx512f supported
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:329 use_avx512f_memcpy_memset] PMEM_AVX512F not set or not == 1
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [pmem.c:216 pmem_has_auto_flush]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [os_auto_flush_linux.c:114 check_domain_in_region] region_path: /sys/bus/nd/devices/region0
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [os_auto_flush_linux.c:59 check_cpu_cache] domain_path: /sys/bus/nd/devices/region0/persistence_domain
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:472 pmem_init_funcs] Flushing CPU cache
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:483 pmem_init_funcs] using clwb
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [init.c:501 pmem_init_funcs] using movnt SSE2
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [pmem_posix.c:107 pmem_os_init]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [libpmem.c:69 libpmem_fini]
pmempool_sync/TEST18 pmem18.log <libpmem>: <3> [mmap.c:101 util_mmap_fini]

pmemblk18.log below.
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:236 out_init] pid 62795: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:238 out_init] libpmemblk version 1.1
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <3> [libpmemblk.c:118 libpmemblk_init]
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <3> [libpmemblk.c:130 libpmemblk_fini]
pmempool_sync/TEST18 pmemblk18.log <libpmemblk>: <3> [mmap.c:101 util_mmap_fini]

pmemlog18.log below.
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:236 out_init] pid 62795: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:238 out_init] libpmemlog version 1.1
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <3> [libpmemlog.c:118 libpmemlog_init]
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <3> [libpmemlog.c:130 libpmemlog_fini]
pmempool_sync/TEST18 pmemlog18.log <libpmemlog>: <3> [mmap.c:101 util_mmap_fini]

Last 30 lines of pmemobj18.log below (whole file has 410 lines).
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7f2480000fb8
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x8f7ea0 part 0 addr 0x7f2480000fb8 len 64 flush 1
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.0"
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0x8f7ee8
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:2887 util_replica_close] set 0x8f35a0 repidx 1
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:4423 util_replica_deep_drain] addr 0x7f1580000000 len 63413682176 set 0x8f35a0 replica_id 1
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:4356 util_replica_deep_common] addr 0x7f1580000000 len 63413682176 set 0x8f35a0 replica_id 1 flush 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x8f6fc0 part 0 addr 0x7f1580000000 len 42276487168 flush 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x8f6fc0 part 1 addr 0x7f1f57e00000 len 21137195008 flush 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.2"
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [shutdown_state.c:152 shutdown_state_clear_dirty] sds 0x7f1580000fb8
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x8f6fc0 part 0 addr 0x7f1580000fb8 len 64 flush 1
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7f1580000fb8
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x8f6fc0 part 0 addr 0x7f1580000fb8 len 64 flush 1
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0x8f7008
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:521 util_poolset_free] set 0x8f35a0
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [libpmemobj.c:65 libpmemobj_fini]
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [obj.c:325 obj_fini]
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST18 pmemobj18.log <libpmemobj>: <3> [mmap.c:101 util_mmap_fini]

pmempool18.log below.
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:236 out_init] pid 62795: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:238 out_init] libpmempool version 1.3
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [libpmempool.c:69 libpmempool_init]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [set.c:124 util_remote_init]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [libpmempool.c:85 libpmempool_fini]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST18 pmempool18.log <libpmempool>: <3> [mmap.c:101 util_mmap_fini]

rpmem18.log below.
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:236 out_init] pid 62773: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool.static-debug
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:238 out_init] librpmem version 1.2
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST18 rpmem18.log <librpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST18 rpmem18.log <librpmem>: <3> [librpmem.c:61 librpmem_init]
pmempool_sync/TEST18 rpmem18.log <librpmem>: <3> [librpmem.c:67 librpmem_init] Libfabric is fork safe
pmempool_sync/TEST18 rpmem18.log <librpmem>: <3> [librpmem.c:79 librpmem_fini]

RUNTESTS: stopping: pmempool_sync/TEST18 failed, TEST=check FS=any BUILD=debug
1 tests failed:
pmempool_sync/TEST18
$ ./RUNTESTS pmempool_sync -s TEST19
pmempool_sync/TEST19: SETUP (check/pmem/debug)
error: pmemobj_root failed
pmempool_sync/TEST19 failed with exit code 1.
out19.log below.

pmem19.log below.
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:236 out_init] pid 64007: program: /home/jenkins/greg/pmdk/src/test/tools/pmemobjcli/pmemobjcli
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:238 out_init] libpmem version 1.1
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST19 pmem19.log <libpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [libpmem.c:56 libpmem_init]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [pmem.c:781 pmem_init]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:419 pmem_init_funcs]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:368 pmem_cpuinfo_to_funcs]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:372 pmem_cpuinfo_to_funcs] clflush supported
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:376 pmem_cpuinfo_to_funcs] clflushopt supported
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:388 pmem_cpuinfo_to_funcs] clwb supported
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:281 use_avx_memcpy_memset] avx supported
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:285 use_avx_memcpy_memset] PMEM_AVX not set or not == 1
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:325 use_avx512f_memcpy_memset] avx512f supported
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:329 use_avx512f_memcpy_memset] PMEM_AVX512F not set or not == 1
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [pmem.c:216 pmem_has_auto_flush]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [os_auto_flush_linux.c:114 check_domain_in_region] region_path: /sys/bus/nd/devices/region0
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [os_auto_flush_linux.c:59 check_cpu_cache] domain_path: /sys/bus/nd/devices/region0/persistence_domain
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:472 pmem_init_funcs] Flushing CPU cache
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:483 pmem_init_funcs] using clwb
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [init.c:501 pmem_init_funcs] using movnt SSE2
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [pmem_posix.c:107 pmem_os_init]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [libpmem.c:69 libpmem_fini]
pmempool_sync/TEST19 pmem19.log <libpmem>: <3> [mmap.c:101 util_mmap_fini]

pmemblk19.log below.
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:236 out_init] pid 64004: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:238 out_init] libpmemblk version 1.1
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <3> [libpmemblk.c:118 libpmemblk_init]
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <3> [libpmemblk.c:130 libpmemblk_fini]
pmempool_sync/TEST19 pmemblk19.log <libpmemblk>: <3> [mmap.c:101 util_mmap_fini]

pmemlog19.log below.
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:236 out_init] pid 64004: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:238 out_init] libpmemlog version 1.1
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <3> [libpmemlog.c:118 libpmemlog_init]
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <3> [libpmemlog.c:130 libpmemlog_fini]
pmempool_sync/TEST19 pmemlog19.log <libpmemlog>: <3> [mmap.c:101 util_mmap_fini]

Last 30 lines of pmemobj19.log below (whole file has 410 lines).
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7fd280000fb8
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x1723ea0 part 0 addr 0x7fd280000fb8 len 64 flush 1
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.2"
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0x1723ee8
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:2887 util_replica_close] set 0x171f5a0 repidx 1
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:4423 util_replica_deep_drain] addr 0x7fc380000000 len 63413682176 set 0x171f5a0 replica_id 1
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:4356 util_replica_deep_common] addr 0x7fc380000000 len 63413682176 set 0x171f5a0 replica_id 1 flush 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x1722fc0 part 0 addr 0x7fc380000000 len 21137195008 flush 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.0"
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x1722fc0 part 1 addr 0x7fc86be00000 len 42276487168 flush 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.4"
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [shutdown_state.c:152 shutdown_state_clear_dirty] sds 0x7fc380000fb8
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x1722fc0 part 0 addr 0x7fc380000fb8 len 64 flush 1
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.0"
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7fc380000fb8
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x1722fc0 part 0 addr 0x7fc380000fb8 len 64 flush 1
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.0"
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0x1723008
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:521 util_poolset_free] set 0x171f5a0
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [libpmemobj.c:65 libpmemobj_fini]
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [obj.c:325 obj_fini]
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST19 pmemobj19.log <libpmemobj>: <3> [mmap.c:101 util_mmap_fini]

pmempool19.log below.
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:236 out_init] pid 64004: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:238 out_init] libpmempool version 1.3
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [libpmempool.c:69 libpmempool_init]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [set.c:124 util_remote_init]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [libpmempool.c:85 libpmempool_fini]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST19 pmempool19.log <libpmempool>: <3> [mmap.c:101 util_mmap_fini]

rpmem19.log below.
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:236 out_init] pid 63981: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool.static-debug
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:238 out_init] librpmem version 1.2
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST19 rpmem19.log <librpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST19 rpmem19.log <librpmem>: <3> [librpmem.c:61 librpmem_init]
pmempool_sync/TEST19 rpmem19.log <librpmem>: <3> [librpmem.c:67 librpmem_init] Libfabric is fork safe
pmempool_sync/TEST19 rpmem19.log <librpmem>: <3> [librpmem.c:79 librpmem_fini]

RUNTESTS: stopping: pmempool_sync/TEST19 failed, TEST=check FS=any BUILD=debug
1 tests failed:
pmempool_sync/TEST19
$ ./RUNTESTS pmempool_sync -s TEST20
pmempool_sync/TEST20: SETUP (check/pmem/debug)
error: pmemobj_root failed
pmempool_sync/TEST20 failed with exit code 1.
out20.log below.

pmem20.log below.
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:236 out_init] pid 64435: program: /home/jenkins/greg/pmdk/src/test/tools/pmemobjcli/pmemobjcli
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:238 out_init] libpmem version 1.1
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST20 pmem20.log <libpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [libpmem.c:56 libpmem_init]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [pmem.c:781 pmem_init]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:419 pmem_init_funcs]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:368 pmem_cpuinfo_to_funcs]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:372 pmem_cpuinfo_to_funcs] clflush supported
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:376 pmem_cpuinfo_to_funcs] clflushopt supported
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:388 pmem_cpuinfo_to_funcs] clwb supported
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:281 use_avx_memcpy_memset] avx supported
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:285 use_avx_memcpy_memset] PMEM_AVX not set or not == 1
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:325 use_avx512f_memcpy_memset] avx512f supported
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:329 use_avx512f_memcpy_memset] PMEM_AVX512F not set or not == 1
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [pmem.c:216 pmem_has_auto_flush]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [os_auto_flush_linux.c:114 check_domain_in_region] region_path: /sys/bus/nd/devices/region0
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [os_auto_flush_linux.c:59 check_cpu_cache] domain_path: /sys/bus/nd/devices/region0/persistence_domain
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:472 pmem_init_funcs] Flushing CPU cache
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:483 pmem_init_funcs] using clwb
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [init.c:501 pmem_init_funcs] using movnt SSE2
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [pmem_posix.c:107 pmem_os_init]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [libpmem.c:69 libpmem_fini]
pmempool_sync/TEST20 pmem20.log <libpmem>: <3> [mmap.c:101 util_mmap_fini]

pmemblk20.log below.
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:236 out_init] pid 64432: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:238 out_init] libpmemblk version 1.1
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <3> [libpmemblk.c:118 libpmemblk_init]
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <3> [libpmemblk.c:130 libpmemblk_fini]
pmempool_sync/TEST20 pmemblk20.log <libpmemblk>: <3> [mmap.c:101 util_mmap_fini]

pmemlog20.log below.
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:236 out_init] pid 64432: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:238 out_init] libpmemlog version 1.1
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <3> [libpmemlog.c:118 libpmemlog_init]
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <3> [libpmemlog.c:130 libpmemlog_fini]
pmempool_sync/TEST20 pmemlog20.log <libpmemlog>: <3> [mmap.c:101 util_mmap_fini]

Last 30 lines of pmemobj20.log below (whole file has 410 lines).
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7f3000000fb8
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0xbdfea0 part 0 addr 0x7f3000000fb8 len 64 flush 1
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.2"
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0xbdfee8
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:2887 util_replica_close] set 0xbdb5a0 repidx 1
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:4423 util_replica_deep_drain] addr 0x7f1c40000000 len 84552974336 set 0xbdb5a0 replica_id 1
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:4356 util_replica_deep_common] addr 0x7f1c40000000 len 84552974336 set 0xbdb5a0 replica_id 1 flush 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0xbdefc0 part 0 addr 0x7f1c40000000 len 42276487168 flush 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.4"
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0xbdefc0 part 1 addr 0x7f2617e00000 len 42276487168 flush 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [shutdown_state.c:152 shutdown_state_clear_dirty] sds 0x7f1c40000fb8
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0xbdefc0 part 0 addr 0x7f1c40000fb8 len 64 flush 1
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.4"
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7f1c40000fb8
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0xbdefc0 part 0 addr 0x7f1c40000fb8 len 64 flush 1
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.4"
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0xbdf008
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:521 util_poolset_free] set 0xbdb5a0
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [libpmemobj.c:65 libpmemobj_fini]
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [obj.c:325 obj_fini]
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST20 pmemobj20.log <libpmemobj>: <3> [mmap.c:101 util_mmap_fini]

pmempool20.log below.
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:236 out_init] pid 64432: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:238 out_init] libpmempool version 1.3
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [libpmempool.c:69 libpmempool_init]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [set.c:124 util_remote_init]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [libpmempool.c:85 libpmempool_fini]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST20 pmempool20.log <libpmempool>: <3> [mmap.c:101 util_mmap_fini]

rpmem20.log below.
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:236 out_init] pid 64410: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool.static-debug
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:238 out_init] librpmem version 1.2
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST20 rpmem20.log <librpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST20 rpmem20.log <librpmem>: <3> [librpmem.c:61 librpmem_init]
pmempool_sync/TEST20 rpmem20.log <librpmem>: <3> [librpmem.c:67 librpmem_init] Libfabric is fork safe
pmempool_sync/TEST20 rpmem20.log <librpmem>: <3> [librpmem.c:79 librpmem_fini]

RUNTESTS: stopping: pmempool_sync/TEST20 failed, TEST=check FS=any BUILD=debug
1 tests failed:
pmempool_sync/TEST20
$ ./RUNTESTS pmempool_sync -s TEST21
pmempool_sync/TEST21: SETUP (check/pmem/debug)
error: pmemobj_root failed
pmempool_sync/TEST21 failed with exit code 1.
out21.log below.

pmem21.log below.
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:236 out_init] pid 64861: program: /home/jenkins/greg/pmdk/src/test/tools/pmemobjcli/pmemobjcli
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:238 out_init] libpmem version 1.1
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST21 pmem21.log <libpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [libpmem.c:56 libpmem_init]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [pmem.c:781 pmem_init]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:419 pmem_init_funcs]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:368 pmem_cpuinfo_to_funcs]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:372 pmem_cpuinfo_to_funcs] clflush supported
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:376 pmem_cpuinfo_to_funcs] clflushopt supported
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:388 pmem_cpuinfo_to_funcs] clwb supported
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:281 use_avx_memcpy_memset] avx supported
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:285 use_avx_memcpy_memset] PMEM_AVX not set or not == 1
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:325 use_avx512f_memcpy_memset] avx512f supported
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:329 use_avx512f_memcpy_memset] PMEM_AVX512F not set or not == 1
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [pmem.c:216 pmem_has_auto_flush]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [os_auto_flush_linux.c:114 check_domain_in_region] region_path: /sys/bus/nd/devices/region0
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [os_auto_flush_linux.c:59 check_cpu_cache] domain_path: /sys/bus/nd/devices/region0/persistence_domain
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:472 pmem_init_funcs] Flushing CPU cache
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:483 pmem_init_funcs] using clwb
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [init.c:501 pmem_init_funcs] using movnt SSE2
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [pmem_posix.c:107 pmem_os_init]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [libpmem.c:69 libpmem_fini]
pmempool_sync/TEST21 pmem21.log <libpmem>: <3> [mmap.c:101 util_mmap_fini]

pmemblk21.log below.
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:236 out_init] pid 64858: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:238 out_init] libpmemblk version 1.1
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <3> [libpmemblk.c:118 libpmemblk_init]
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <3> [libpmemblk.c:130 libpmemblk_fini]
pmempool_sync/TEST21 pmemblk21.log <libpmemblk>: <3> [mmap.c:101 util_mmap_fini]

pmemlog21.log below.
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:236 out_init] pid 64858: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:238 out_init] libpmemlog version 1.1
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <3> [libpmemlog.c:118 libpmemlog_init]
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <3> [libpmemlog.c:130 libpmemlog_fini]
pmempool_sync/TEST21 pmemlog21.log <libpmemlog>: <3> [mmap.c:101 util_mmap_fini]

Last 30 lines of pmemobj21.log below (whole file has 410 lines).
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7fdfc0000fb8
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x841ea0 part 0 addr 0x7fdfc0000fb8 len 64 flush 1
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.0"
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0x841ee8
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:2887 util_replica_close] set 0x83d5a0 repidx 1
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:4423 util_replica_deep_drain] addr 0x7fcc00000000 len 84552974336 set 0x83d5a0 replica_id 1
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:4356 util_replica_deep_common] addr 0x7fcc00000000 len 84552974336 set 0x83d5a0 replica_id 1 flush 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x840fc0 part 0 addr 0x7fcc00000000 len 42276487168 flush 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x840fc0 part 1 addr 0x7fd5d7e00000 len 42276487168 flush 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.4"
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [shutdown_state.c:152 shutdown_state_clear_dirty] sds 0x7fcc00000fb8
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x840fc0 part 0 addr 0x7fcc00000fb8 len 64 flush 1
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [shutdown_state.c:56 shutdown_state_checksum] sds 0x7fcc00000fb8
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:170 os_part_deep_common] part 0x840fc0 part 0 addr 0x7fcc00000fb8 len 64 flush 1
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [file_posix.c:302 util_ddax_region_find] path "/dev/dax0.1"
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [os_deep_linux.c:56 os_deep_flush_write] region_id 0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:483 util_unmap_part] part 0x841008
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:521 util_poolset_free] set 0x83d5a0
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [libpmemobj.c:65 libpmemobj_fini]
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [obj.c:325 obj_fini]
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST21 pmemobj21.log <libpmemobj>: <3> [mmap.c:101 util_mmap_fini]

pmempool21.log below.
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:236 out_init] pid 64858: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:238 out_init] libpmempool version 1.3
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [mmap.c:67 util_mmap_init]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [libpmempool.c:69 libpmempool_init]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [set.c:124 util_remote_init]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [libpmempool.c:85 libpmempool_fini]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [set.c:139 util_remote_fini]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [set.c:194 util_remote_unload]
pmempool_sync/TEST21 pmempool21.log <libpmempool>: <3> [mmap.c:101 util_mmap_fini]

rpmem21.log below.
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:236 out_init] pid 64836: program: /home/jenkins/greg/pmdk/src/tools/pmempool/pmempool.static-debug
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:238 out_init] librpmem version 1.2
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:242 out_init] src version: 1.6-rc2+git1.gfe22f9065
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:250 out_init] compiled with support for Valgrind pmemcheck
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:255 out_init] compiled with support for Valgrind helgrind
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:260 out_init] compiled with support for Valgrind memcheck
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:265 out_init] compiled with support for Valgrind drd
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:270 out_init] compiled with support for shutdown state
pmempool_sync/TEST21 rpmem21.log <librpmem>: <1> [out.c:275 out_init] compiled with libndctl 63+
pmempool_sync/TEST21 rpmem21.log <librpmem>: <3> [librpmem.c:61 librpmem_init]
pmempool_sync/TEST21 rpmem21.log <librpmem>: <3> [librpmem.c:67 librpmem_init] Libfabric is fork safe
pmempool_sync/TEST21 rpmem21.log <librpmem>: <3> [librpmem.c:79 librpmem_fini]

RUNTESTS: stopping: pmempool_sync/TEST21 failed, TEST=check FS=any BUILD=debug
1 tests failed:
pmempool_sync/TEST21

Logs:

pmem18.log pmem19.log pmem20.log pmem21.log pmemblk18.log pmemblk19.log pmemblk20.log pmemblk21.log pmemlog18.log pmemlog19.log pmemlog20.log pmemlog21.log pmemobj18.log pmemobj19.log pmemobj20.log pmemobj21.log pmempool18.log pmempool19.log pmempool20.log pmempool21.log rpmem18.log rpmem19.log rpmem20.log rpmem21.log

Expected behavior:

Tests should pass.

Details

Additional information about Priority and Help Requested:

Are you willing to submit a pull request with a proposed change? (Yes, No)

Requested priority: (Showstopper, High, Medium, Low)

Greg091 commented 5 years ago

Verified on: 1.6-rc2-17-ga828935ef