Unable to read and write data

[mimingan]

Now we have a question not writting data on a ssd for the fatcache.

Steps are as follows：

1. wget -c http://fatcache.googlecode.com/files/fatcache-0.1.0.tar.gz (Missing fc_client.h and fc_server.h files.)
2. wget -c https://github.com/twitter/fatcache/archive/master.zip
3. installing:

tar zxf fatcache-0.1.0.tar.gz

unzip master.zip

cp fatcache-master/src/fc_server.* fatcache-0.1.0/src/

cp fatcache-master/src/fc_client.* fatcache-0.1.0/src/

cd fatcache-0.1.0

./configure --prefix=/usr/local/fatcache

make && make install

Machine environment：

Installed two SSD. The first, installed centos 6.4 64bit The second, data storage.

a)fdisk or not to fdisk the ssd, we have tried, both can not write on the ssd.

start: /usr/local/fatcache/bin/fatcache -d -o /usr/local/fatcache/log/fatcache.log -D /dev/sdb

fatcache log: Mon Aug 12 12:39:47 2013] fc.c:683 fatcache-0.1.0 started on pid 9446 [Mon Aug 12 12:39:47 2013] fc.c:688 configured with debug logs disabled, asserts disabled, panic disabled [Mon Aug 12 12:39:47 2013] fc_slab.c:85 slab size 1048576, slab hdr size 12, item hdr size 52, item chunk size 88 [Mon Aug 12 12:39:47 2013] fc_slab.c:88 index memory 0, slab memory 67108864, disk space 240056795136 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 0: items 11915 size 88 data 36 slack 44 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 1: items 9362 size 112 data 60 slack 20 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 2: items 7281 size 144 data 92 slack 100 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 3: items 5698 size 184 data 132 slack 132 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 4: items 4519 size 232 data 180 slack 156 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 5: items 3542 size 296 data 244 slack 132 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 6: items 2788 size 376 data 324 slack 276 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 7: items 2221 size 472 data 420 slack 252 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 8: items 1771 size 592 data 540 slack 132 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 9: items 1409 size 744 data 692 slack 268 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 10: items 1120 size 936 data 884 slack 244 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 11: items 891 size 1176 data 1124 slack 748 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 12: items 712 size 1472 data 1420 slack 500 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 13: items 569 size 1840 data 1788 slack 1604 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 14: items 455 size 2304 data 2252 slack 244 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 15: items 364 size 2880 data 2828 slack 244 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 16: items 291 size 3600 data 3548 slack 964 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 17: items 232 size 4504 data 4452 slack 3636 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 18: items 186 size 5632 data 5580 slack 1012 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 19: items 148 size 7040 data 6988 slack 6644 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 20: items 119 size 8800 data 8748 slack 1364 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 21: items 95 size 11000 data 10948 slack 3564 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 22: items 76 size 13752 data 13700 slack 3412 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 23: items 60 size 17192 data 17140 slack 17044 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 24: items 48 size 21496 data 21444 slack 16756 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 25: items 39 size 26872 data 26820 slack 556 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 26: items 31 size 33592 data 33540 slack 7212 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 27: items 24 size 41992 data 41940 slack 40756 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 28: items 19 size 52496 data 52444 slack 51140 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 29: items 15 size 65624 data 65572 slack 64204 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 30: items 12 size 82032 data 81980 slack 64180 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 31: items 10 size 102544 data 102492 slack 23124 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 32: items 8 size 128184 data 128132 slack 23092 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 33: items 6 size 160232 data 160180 slack 87172 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 34: items 5 size 200296 data 200244 slack 47084 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 35: items 4 size 250376 data 250324 slack 47060 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 36: items 3 size 312976 data 312924 slack 109636 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 37: items 2 size 391224 data 391172 slack 266116 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 38: items 2 size 489032 data 488980 slack 70500 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 39: items 1 size 611296 data 611244 slack 437268 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 40: items 1 size 764120 data 764068 slack 284444 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 41: items 1 size 955152 data 955100 slack 93412 [Mon Aug 12 12:39:47 2013] fc_slab.c:94 class 42: items 1 size 1048564 data 1048512 slack 0

[manjuraj]

@mimingan

good catch about missing fc_client.h and fc_server.h files. I uploaded a new distribution with a fix for missing files. See: https://code.google.com/p/fatcache/downloads/list

Regarding writing to ssd, it seems like your fatcache instance is working as expected. If you read through the README.md, you will notice that fatcache first writes to memory and then batch writes slabs to ssd. I don't see any error in the log file that you uploaded

Could you describe the issue you are encountering in detail?

[mimingan]

@manjuraj

Thank you for your reply to us so quickly. Sorry, I do not describe the problem in detail.

Accordance with the above method, I run the fatcache.

The "set" and "get" commands execution is a failure.

For example:

printf "set zhangyan 0 0 5\r\n12345\r\n" | nc 127.0.0.1 11211

printf "set zhang07 0 0 6\r\n123456\r\n" | nc 127.0.0.1 11211

printf "set zhang900 0 0 3\r\n456\r\n" | nc 192.168.1.3 11211

printf "get zhang07\r\n" | nc 127.0.0.1 11211

And so on. Repeatedly executed several times, the above commands are a failure .

[mimingan]

@manjuraj

Thank you for your reply.

I downloaded the fatcache-0.1.1.tar.gz.

Now, the "set" and "get" commands execution is OK, thanks.

testing...

[mimingan]

@manjuraj

I use the 'set' command of libmemcached, write a data to fatcache, spending 600 - 1000 ms.(very slowly) use mcperf the same as libmemcached. I didn't know why. can you tell me, thanks. have I wrong usage ？

[manjuraj]

Is the backing store a ssd or disk? You can also look at the perf results in notes/ folder to get an idea on how to use fatcache

[mimingan]

In practical work, we feel the fatcache very low performance . testing is as follows, //code uint32_t flags = 0; time_t dur_time = 0; struct timeval start = {0}, end = {0}; gettimeofday(&start, NULL); rc = memcachedset(memc, key.c_str(), key.length(), value.c_str(), value.length(), dur_time, flags); gettimeofday(&end, NULL); printf("cache 'set' cmd time: %ld ms.", diff_time(&start, &end) / 1000);

1. connect to a fatcache (127.0.0.1:11211) Start the fatcache. /usr/local/fatcache/bin/fatcache -d -o /usr/local/fatcache/log/fatcache.log -D /dev/sdb

log: Wed Aug 14 09:34:00 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 1000 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 1000 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 1000 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 1000 ms. Wed Aug 14 09:34:01 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 0 ms. Wed Aug 14 09:34:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms. Wed Aug 14 09:34:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 999 ms.

1. connect to a memcached (192.168.1.3:11211) log: Wed Aug 14 11:04:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 0 ms. Wed Aug 14 11:04:02 2013 [common/memcached/mycache.cpp]:[93]:[set_key_value] cache 'set' cmd time: 0 ms.
2. Large concurrent case /usr/local/mcperf/bin/mcperf --sizes=u100,100 --num-calls=10000 --num-conns=100 --call-rate=1000 --conn-rate=10000 --method=set --server=192.168.1.3 --port=11211 Total: connections 100 requests 1000000 responses 1000000 test-duration 10.010 s Request rate: 99904.8 req/s (0.0 ms/req) Request size [B]: avg 129.0 min 129.0 max 129.0 stddev 0.00 Response rate: 99904.8 rsp/s (0.0 ms/rsp)
3. only execute once 'set' command. The fatcache is very slow. spending 1000 ms， but the memcached is far less than 1 ms.
4. Have you done once 'set' command to test? Thanks for your reply.

[manjuraj]

@mimingan

please read this: https://github.com/twitter/fatcache/blob/master/notes/performance.md and this: https://github.com/twitter/fatcache/blob/master/notes/intel-ssd-320-specification.pdf and this: https://github.com/twitter/fatcache/blob/master/notes/spec.md

your backing store must be SSD. On your SSD figure out what its read and write latencies and your latency numbers would be something similar for a 4K read / write.

hth, let me know if you how the testing works out

[manjuraj]

closing this for now. feel free to reopen if you have questions