search

fengggli / comanche

comanche
Apache License 2.0
0 stars 1 forks source link

[hstore instructions] #2

Open fengggli opened 5 years ago

fengggli commented 5 years ago

This tracks how I could use hstore

fengggli commented 5 years ago

prepare ndctl

(py36) lifen@sievert(:):~/Workspace/comanche/build/src/lib/ndctl$sudo ./ndctl-prefix/src/ndctl/ndctl/.libs/ndctl list
{
  "dev":"namespace0.0",
  "mode":"fsdax",
  "map":"mem",
  "size":2147483648,
  "sector_size":512,
  "blockdev":"pmem0",
  "numa_node":0
}

switch from daxfs to daxdev:

# unmount fs
sudo umount /mnt/pmem0
# disable namespace
(py36) lifen@sievert(:):~/Workspace/comanche/build/src/lib/ndctl$sudo ./ndctl-prefix/src/ndctl/ndctl/.libs/ndctl disable-namespace namespace0.0

distroy namespace

sudo ./ndctl-prefix/src/ndctl/ndctl/.libs/ndctl destroy-namespace all -f

Create namespace

(py36) lifen@sievert(:):~/Workspace/comanche/build/src/lib/ndctl$sudo ./ndctl-prefix/src/ndctl/ndctl/.libs/ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f
{
  "dev":"namespace0.0",
  "mode":"devdax",
  "map":"mem",
  "size":"2046.00 MiB (2145.39 MB)",
  "uuid":"41d21a2a-36ef-41c0-bb71-5efc594921ee",
  "daxregion":{
    "id":0,
    "size":"2046.00 MiB (2145.39 MB)",
    "align":2097152,
    "devices":[
      {
        "chardev":"dax0.0",
        "size":"2046.00 MiB (2145.39 MB)"
      }
    ]
  },
  "numa_node":0
}
fengggli commented 5 years ago

Also the infiniband configuration in sievert

lifen@sievert(:):~/software/MLNX_OFED_LINUX-4.6-1.0.1.1-ubuntu16.04-x86_64

Some extra notes for hstore

fengggli commented 5 years ago

configuration pmem: https://docs.pmem.io/getting-started-guide/creating-development-environments/linux-environments/linux-memmap Specifically, I could reserve two regions of pmem: using memmap, seperated by comma. However this only brings me two "regions" inside each region, I could only have only one namespace, if the device doesn't support labels. So I could add a different region like this: (so I could get namespace 1.0, but not namespace 0.1)

ndctl create-namespace --mode devdax --map mem -e namespace1.0 -f
{
  "dev":"namespace1.0",
  "mode":"devdax",
  "map":"mem",
  "size":"2046.00 MiB (2145.39 MB)",
  "uuid":"1ea90bbb-0811-432d-b041-f3dbe92e3d4e",
  "daxregion":{
    "id":1,
    "size":"2046.00 MiB (2145.39 MB)",
    "align":2097152,
    "devices":[
      {
        "chardev":"dax1.0",
        "size":"2046.00 MiB (2145.39 MB)"
      }
    ]
  },
  "numa_node":1
}

this might be related: https://peeterjoot.wordpress.com/2010/05/26/a-fun-and-curious-dig-gcc-generation-of-a-ud2a-instruction-sigill/

fengggli commented 5 years ago

A copy of the email

Hi Clem,

I was trying to run hstore unit test(hstore-test1) and but I got a segmentation fault. I was wondering whether sth is configured wrong in my side. really appreciate if you could give some suggestions on this.

Thanks Feng

environment

ubuntu 18.04, comanche #598c8b6a , IBM m4-x3650 kernel bootline: BOOT_IMAGE=/boot/vmlinuz-4.15.0-50-generic root=UUID=4628adf8-2645-4b29-9cc7-7106c7e42a8b ro 3 hugepagesz=2M hugepages=1024 intel_iommu=on memmap=2G!16G

prepare

  1. First i created a namespace following the instructions in doc/HOWTO.emulate.pm (sudo ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f)

  2. My namespace looks like this: (py36) lifen@sievert(:):~/Workspace/comanche/build/src/lib/ndctl$sudo ./ndctl-prefix/src/ndctl/ndctl/.libs/ndctl list { "dev":"namespace0.0", "mode":"devdax", "map":"mem", "size":2145386496, "uuid":"454f75ec-4cc3-41c2-a332-587dbaaba3e2", "chardev":"dax0.0", "numa_node":0 }

  3. I also changed the "/dev/dax0.1" to "/dev/dax0.0" in src/components/store/hstore/unit_test/store_map.cpp L41, since I don't have the /dev/dax0.1, (btw, is there anyway I can generate the /dev/dax0.1, so that I don't need to change the hstore unit_test code?)

I have never created a /dev/dax device (Daniel does that), but from executing "ndctl list" nio aep1, I would guess that way to do that is to create two memmaps using a comma-separated list:

memmap=2G!480G,2G!482G

and convert them both to devdax

sudo ndctl create-namespace --mode devdax --map mem -e namespace0.0 -f sudo ndctl create-namespace --mode devdax --map mem -e namespace0.1 -f

I am guessing that two minor devices dill be created, named /dev/dax0.0 and /dev/dax0.1.

  1. there is an address 0x9000000000 in the store_map.cpp, what's the meaning of this configuration param and shall I change it to sth else?

That is the virtual address at which the /dev/dax0.x device is to be mapped. It affects only your address space. But since the persisted data often includes pointers to elements in the address space, everyone who uses it must map it to the same address. If you are only using one device (as the test case does) there is no need to change it. If you are using multiple devices, you should, of course, map them to disjoint regions.

output

The output seems to say think that the device was used by a process that crashed and is trying to recover. Perhaps the segfault was due to a bug, or because there was an actual crash or shutdown using an earlier and incompatible version of Devdax_manager. In any case, I can see from frame 5 that the force_reset parameter is false. You should run it with force+reset=true, which you can do buy setting env variable DAX_RESET=1:

DAX_RESET=1 gdb --args /home/lifen/Workspace/comanche/build/src/components/store/hstore/unit_test/hstore-test1


(gdb) r
Starting program: /home/lifen/Workspace/comanche/build/src/components/store/hstore/unit_test/hstore-test1 ./src/components/store/hstore/unit_test/hstore-test1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[==========] Running 32 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 32 tests from KVStore_test
[ RUN      ] KVStore_test.Instantiate
[LOG]:Loaded component: libcomanche-hstore.so
[LOG]:Devdax_manager: region (/dev/dax0.0,9000000000)
[LOG]:Registered dax mgr instance: /dev/dax0.0
[LOG]:Devdax_manager: region (/dev/dax0.0) opened ok
[LOG]:Devdax_manager: /dev/dax0.0 size=2145386496
[LOG]:devdax_manager: no rebuild.
[LOG]:Checking undo logs..
[LOG]:undo log being applied (ptr=0xdc62a96800000020, len=4742682868250225000).
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff5a5874f in memmove_movnt_sse2_clflush () from /home/lifen/Workspace/comanche/build/dist/lib64/libpmem.so.1
(gdb) bt
#0  0x00007ffff5a5874f in memmove_movnt_sse2_clflush () from /home/lifen/Workspace/comanche/build/dist/lib64/libpmem.so.1
#1  0x00007ffff5a528e1 in memmove_nodrain_sse2_clflush () from /home/lifen/Workspace/comanche/build/dist/lib64/libpmem.so.1
#2  0x00007ffff5a5224a in pmem_memcpy_persist () from /home/lifen/Workspace/comanche/build/dist/lib64/libpmem.so.1
#3  0x00007ffff61760b2 in nupm::DM_undo_log::check_and_undo (this=0x9000000040) at /home/lifen/Workspace/comanche/src/lib/nupm/src/dax_data.h:71
#4  0x00007ffff61762a2 in nupm::DM_region_header::check_undo_logs (this=0x9000000000) at /home/lifen/Workspace/comanche/src/lib/nupm/src/dax_data.h:136
#5  0x00007ffff617543a in nupm::Devdax_manager::recover_metadata (this=0x667330, device_path=0x67cc50 "/dev/dax0.0", p=0x9000000000, p_len=2145386496, force_rebuild=false)
   at /home/lifen/Workspace/comanche/src/lib/nupm/src/dax_map.cpp:211
#6  0x00007ffff6174881 in nupm::Devdax_manager::Devdax_manager (this=0x667330, dax_configs=std::vector of length 1, capacity 1 = {...}, force_reset=false) at /home/lifen/Workspace/comanche/src/lib/nupm/src/dax_map.cpp:117
#7  0x00007ffff66b341f in Devdax_manager::Devdax_manager (this=0x667330, dax_map="[ { \"region_id\": 0, \"path\" : \"/dev/dax0.0\", \"addr\" : \"0x9000000000\" } ]", force_reset=false)
   at /home/lifen/Workspace/comanche/src/components/store/hstore/src/dax_map.cpp:210
#8  0x00007ffff66fc7dd in std::make_unique<Devdax_manager, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&&) () at /usr/include/c++/5/bits/unique_ptr.h:765
#9  0x00007ffff66fc392 in hstore_factory::create (this=0x667310, owner="owner", name="numa0", dax_map="[ { \"region_id\": 0, \"path\" : \"/dev/dax0.0\", \"addr\" : \"0x9000000000\" } ]")
   at /home/lifen/Workspace/comanche/src/components/store/hstore/src/hstore_factory.cpp:70
#10 0x0000000000407e24 in (anonymous namespace)::KVStore_test_Instantiate_Test::TestBody (this=<optimized out>) at /home/lifen/Workspace/comanche/src/components/store/hstore/unit_test/test1.cpp:137
#11 0x00000000004356a3 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x439aaf "the test body", method=<optimized out>, object=<optimized out>) at /usr/src/gtest/src/gtest.cc:2078
#12 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x660e40,
   method=(void (testing::Test::*)(testing::Test * const)) 0x407c90 <(anonymous namespace)::KVStore_test_Instantiate_Test::TestBody()>, location=location@entry=0x439aaf "the test body") at /usr/src/gtest/src/gtest.cc:2114
#13 0x000000000042cfbc in testing::Test::Run (this=this@entry=0x660e40) at /usr/src/gtest/src/gtest.cc:2151
#14 0x000000000042d195 in testing::Test::Run (this=0x660e40) at /usr/src/gtest/src/gtest.cc:2142
#15 testing::TestInfo::Run (this=0x65e010) at /usr/src/gtest/src/gtest.cc:2326
#16 0x000000000042d305 in testing::TestInfo::Run (this=<optimized out>) at /usr/src/gtest/src/gtest.cc:2301
#17 testing::TestCase::Run (this=0x65e430) at /usr/src/gtest/src/gtest.cc:2444
#18 0x000000000042d78d in testing::TestCase::Run (this=<optimized out>) at /usr/src/gtest/src/gtest.cc:4353
#19 testing::internal::UnitTestImpl::RunAllTests (this=0x65e100) at /usr/src/gtest/src/gtest.cc:4315
#20 0x0000000000435b83 in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x438ee0 "auxiliary test code (environments or event listeners)", method=<optimized out>,
   object=<optimized out>) at /usr/src/gtest/src/gtest.cc:2078
#21 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x65e100,
   method=(bool (testing::internal::UnitTestImpl::*)(testing::internal::UnitTestImpl * const)) 0x42d3f0 <testing::internal::UnitTestImpl::RunAllTests()>,
   location=location@entry=0x438ee0 "auxiliary test code (environments or event listeners)") at /usr/src/gtest/src/gtest.cc:2114
#22 0x000000000042daa5 in testing::UnitTest::Run (this=0x64bdc0 <testing::UnitTest::GetInstance()::instance>) at /usr/src/gtest/src/gtest.cc:3926
#23 0x0000000000404b0f in RUN_ALL_TESTS () at /usr/include/gtest/gtest.h:2288
#24 main (argc=2, argv=<optimized out>) at /home/lifen/Workspace/comanche/src/components/store/hstore/unit_test/test1.cpp:695
fengggli commented 5 years ago

I should read this instructions!

https://github.com/fengggli/comanche/blob/598c8b6a4f1d20705ebe1c669e4cec0ed19d042e/src/components/store/hstore/src/hstore.cpp#L56-L59