search

syndicate-storage / syndicate-core

Scalable Software-define Storage System
Apache License 2.0
4 stars 1 forks source link

Cannot perform a listdir command to a recreated volume #29

Open iychoi opened 7 years ago

iychoi commented 7 years ago

Cannot perform a listdir command to a recreated volume. I observed the issue while running syndicatefs. The volume I tried to mount was recreated one. It was created then deleted. I guess duplicated root directories caused the issue.


6017248.263945167 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from libsyndicate.cpp:2493)
6017248.263975146 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from libsyndicate.cpp:2390)
6017248.263991273 00584:00007f27cffff700: [libsyndicate.cpp:2395] ms_entry_verify: DEBUG: Check signature of B9962681FC303901 from gateway 7092660688083880344
6017248.264019963 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:822)
6017248.264039432 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:835)
6017248.264115492 00584:00007f27cffff700: [libsyndicate.cpp:2430] ms_entry_verify: DEBUG: Verify:
type: 1
file_id: 13372918483246921985
ctime_sec: 1496814636
ctime_nsec: 13782273
mtime_sec: 1496814636
mtime_nsec: 13782273
manifest_mtime_sec: 1496814636
manifest_mtime_nsec: 13782433
owner: 1215882033360548424
coordinator: 7092660688083880344
volume: 8593192222588951184
mode: 493
size: 14232283
version: 1
max_read_freshness: 300000
max_write_freshness: 300000
name: "POV_L.Spr.I.2000m_reads.fa"
write_nonce: 1
xattr_nonce: 0
generation: 1
signature: "LYXpJk3siEFBIgHwTe3SXGV6B73JPPauX57pU8AmGNiwTtut7ssV2Nv/NwmkUl/CGfJcyrtPqNSQk7bkawksYb53t1D4KGFfxxGfavvRkUUvgrHDJLZMej5LsKQpaOaWQN0ww0RwmM2NG4iATgvJi4uAnFWo9ImXgNsZM/fi0q19ByJo1c+Q/odYTPSUfKzQQmeF2Tvuien+l51oHSMhsGcTOJWTHQ1mL6ZNqS0mpOWFg0/DhiGs1lOdJimj8k1GiMg4ZCnPng3jLpXjKJC3oHxf8v80X2xkiXEwkBWqGcQ1yd9ccWVtCrG5dcDxPNsWNMowL5jo27k3WzPa9jwo+pg5PeDpGZURwrOW6GaCQoPF8ZSpPMcjMbm/vM7vLwLS7poZZ8sJWGPPMOVTG6FXfk7AUol7Qh/rmJ8s72NX549tQxObcNHsboOI9fdd+Xg9Wa1ufkRa9zDZ2s5hdNozGECPEYiDIgVkNIKy7LWgPn20wrAz1Db21BO/hpRslTyeNgkpYNYgz36A/cqMLfA1tAbhlcVSQgLRbNkq+UOZGeT6sNDurI9MZKyFfaOCNyoJ3MaeghDsLhYi6oFEWeb4EXis8swlksM/kojKvIqZ4cRq/OYemHYzvXk/MqPMm54NBiQHEMhJHSOo/daiTX5DB9Uxis8fIeIoREoe5hbxciI="
parent_id: 0
num_children: 0
capacity: 16

6017248.264968963 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from libsyndicate.cpp:2493)
6017248.265004214 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from libsyndicate.cpp:2390)
6017248.265021223 00584:00007f27cffff700: [libsyndicate.cpp:2395] ms_entry_verify: DEBUG: Check signature of 732B1409ACCC923C from gateway 7092660688083880344
6017248.265036391 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:822)
6017248.265052639 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:835)
6017248.265127246 00584:00007f27cffff700: [libsyndicate.cpp:2430] ms_entry_verify: DEBUG: Verify:
type: 1
file_id: 8298748770146619964
ctime_sec: 1496814638
ctime_nsec: 997577324
mtime_sec: 1496814638
mtime_nsec: 997577324
manifest_mtime_sec: 1496814638
manifest_mtime_nsec: 997577637
owner: 1215882033360548424
coordinator: 7092660688083880344
volume: 8593192222588951184
mode: 493
size: 21844560
version: 1
max_read_freshness: 300000
max_write_freshness: 300000
name: "POV_L.Spr.I.500m_reads.fa"
write_nonce: 1
xattr_nonce: 0
generation: 1
signature: "SyJJPReNrNJoMjAdG3L8a6PEqtXM/TzHpefm2WUXav8mB+Q5Z8aFAd7ni0KYjX1pPxZsT2m5NGOYaXfZ9jowhlpJCXwMHrvVbx7OUtTC92gBvi+kt1kgDXI1dIxlRPJ2ngIu1agfgYpxDMLoLKdbGC7Qi8Acruhp5TVIfjilMIkPNdbX1XekVYAb6WFh/6fRTmIkAz7qPQMZyEAlDDTCrrxmcvfhS38YribzLugHd+aWfglNX9+G8Tkq6ti8Jvhq4FEYqdokPMXNrifNhF7nRol26mf/q5mP38CemDVQfAP7LuBup2yuv56LIm5grOi39DxRIJXXxCT38LoaRNLhU0e1fUaV6QUGWrKlrPXGm1KdIhfasHqmClkWXRjiMGVdx0Sm8h+tXKV3hzaa87FKXiaNDV36wxkaxq1l0zE3qfi9ZMOgRo7aZKiLFsLzV/OZDYDVMd2KR3ELyWcXwS4RgJrNgqnl4o8kSHXFDWrmxRaVsc7msIYqm3wOUlzfp3qe/ulh9yUWijxJRBh2HEeIj1bt8vrYXsEu4epC5W2k2QEQECfZrVei3+a6QNXlbqmCxKu3YHr7fOEQVSHZ7G5gMrwBbrjYuxVhQuDs4SXRmnswajao1wVneBXRie33LrAnxvK9sDmAqaU61pxVmipb8g4dpQyHloD15io8PWg4tUk="
parent_id: 0
num_children: 0
capacity: 16

6017248.265955811 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from libsyndicate.cpp:2493)
6017248.266010280 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c8003290 ref 0 (from download.cpp:917)
6017248.266029766 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c8003290, from download.cpp:920 (refcount 0)
6017248.266071914 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: DEC73519649A5F27
6017248.266118960 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 679551BC1A676707
6017248.266141230 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: AD902D7A83DE15D5
6017248.266161459 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 4B39F96631B6E11B
6017248.266182488 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 97369BC237B0D261
6017248.266214852 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 9440B9648343B033
6017248.266236831 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: B9CCA73717C50FFA
6017248.266257012 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 2C2DEF886F78F2B
6017248.266277625 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: B9962681FC303901
6017248.266298548 00584:00007f27cffff700: [  ms/listdir.cpp:0230] ms_client_get_dir_metadata_end: DEBUG: 0x7f27c8003290: 732B1409ACCC923C
6017248.266320434 00584:00007f27cffff700: [  ms/listdir.cpp:0453] ms_client_get_dir_metadata: DEBUG: Fetched 10 (20 downloaded total)
6017248.266338975 00584:00007f27cffff700: [  ms/listdir.cpp:0459] ms_client_get_dir_metadata: DEBUG: Downloaded 20 children (29 given by inode); l.u.g. is now 14968146718936
6017248.266364040 00584:00007f27cffff700: [  ms/listdir.cpp:0417] ms_client_get_dir_metadata: DEBUG: Finished 1 downloads (rc = -11)
6017248.266383242 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:639)
6017248.266399300 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:643)
6017248.266414850 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:629)
6017248.266429524 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:632)
6017248.266444156 00584:00007f27cffff700: [     ms/core.cpp:0588] ms_client_config_rlock2: DEBUG: ms_client_config_rlock 0x24d65d0 (from ms/core.cpp:619)
6017248.266458641 00584:00007f27cffff700: [     ms/core.cpp:0610] ms_client_config_unlock2: DEBUG: ms_client_config_unlock 0x24d65d0 (from ms/core.cpp:622)
6017248.266528761 00584:00007f27cffff700: [    download.cpp:0753] md_download_context_init: DEBUG: Initialize download context 0x7f27c8003290
6017248.266564990 00584:00007f27cffff700: [    download.cpp:1108] md_download_set_add: DEBUG: Add download context 0x7f27c8003290 to download set 0x7f27c8002f28
6017248.266583253 00584:00007f27cffff700: [    download.cpp:0881] md_download_context_ref2: DEBUG: download 0x7f27c8003290 ref 1 (from download.cpp:1181)
6017248.266602803 00584:00007f27cffff700: [    download.cpp:0418] md_downloader_insert_pending: DEBUG: download 0x7f27c8003290 ref 2
6017248.266618832 00584:00007f27cffff700: [    download.cpp:0427] md_downloader_insert_pending: DEBUG: Start download context 0x7f27c8003290
6017248.266634928 00584:00007f27cffff700: [    download.cpp:1037] md_download_context_wait_any: DEBUG: Wait on download set 0x7f27c8002f28 (3 contexts)
* Found bundle for host syndicate-ms-datasets.appspot.com: 0x7f27d0001930
* Re-using existing connection! (#0) with host syndicate-ms-datasets.appspot.com
* Connected to syndicate-ms-datasets.appspot.com (172.217.2.244) port 80 (#0)
> GET /FILE/LISTDIR/8593192222588951184.1.1496814602/0?lug=14968146718936 HTTP/1.1

User-Agent: Syndicate-Gateway/1.0

Host: syndicate-ms-datasets.appspot.com

Accept: */*

< HTTP/1.1 500 Internal Server Error

< Cache-Control: no-cache

< Content-Type: text/plain

< X-Cloud-Trace-Context: 61888446c0f04ec45ee66a0698112996;o=1

< Date: Wed, 07 Jun 2017 07:42:29 GMT

* Server Google Frontend is not blacklisted
< Server: Google Frontend

< Content-Length: 0

< 

* Connection #1 to host syndicate-ms-datasets.appspot.com left intact
6017248.336420057 00584:00007f27d7dc9700: [    download.cpp:1407] md_downloader_finalize_download_context: DEBUG: Finalized download context 0x7f27c8003340 (http://syndicate-ms-datasets.appspot.com:80/FILE/LISTDIR/8593192222588951184.1.1496814602/0?lug=14968146718936)
6017248.336450543 00584:00007f27d7dc9700: [    download.cpp:1418] md_downloader_finalize_download_context: DEBUG: download 0x7f27c8003340 ref 1
6017248.336491408 00584:00007f27d7dc9700: [    download.cpp:1253] md_download_set_wakeup: DEBUG: Wake up download set 0x7f27c8002f28
6017248.336570261 00584:00007f27cffff700: [  ms/listdir.cpp:0185] ms_client_get_dir_metadata_end: ERROR: ms_client_download_parse_errors( 0x7f27c8003340 ) rc = -121
6017248.336630336 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c8003340 ref 0 (from download.cpp:917)
6017248.336648097 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c8003340, from download.cpp:920 (refcount 0)
6017248.336683921 00584:00007f27cffff700: [  ms/listdir.cpp:0430] ms_client_get_dir_metadata: ERROR: ms_client_get_dir_metadata_end rc = -121
6017248.336699433 00584:00007f27cffff700: [  ms/listdir.cpp:0467] ms_client_get_dir_metadata: DEBUG: Breaking loop on rc = -121
6017248.336731114 00584:00007f27cffff700: [    download.cpp:0443] md_downloader_insert_cancelling: DEBUG: Cancel download context 0x7f27c8003290
6017248.336757865 00584:00007f27cffff700: [    download.cpp:0484] md_downloader_insert_cancelling: DEBUG: download 0x7f27c8003290 ref 3
6017248.336776701 00584:00007f27cffff700: [    download.cpp:1233] md_download_context_cancel: DEBUG: Wait for cancelling download 0x7f27c8003290 (set 0x7f27c8002f28)
6017248.336795219 00584:00007f27cffff700: [    download.cpp:1011] md_download_context_wait: DEBUG: Wait on download context 0x7f27c8003290 (18446744073709551615 millis)
* Closing connection 0
6017248.341727263 00584:00007f27d7dc9700: [    download.cpp:0618] md_downloader_end_all_cancelling: DEBUG: download 0x7f27c8003290 ref 2
6017248.341799253 00584:00007f27d7dc9700: [    download.cpp:1407] md_downloader_finalize_download_context: DEBUG: Finalized download context 0x7f27c8003290 (http://syndicate-ms-datasets.appspot.com:80/FILE/LISTDIR/8593192222588951184.1.1496814602/0?lug=14968146718936)
6017248.341817500 00584:00007f27d7dc9700: [    download.cpp:1418] md_downloader_finalize_download_context: DEBUG: download 0x7f27c8003290 ref 1
6017248.341831904 00584:00007f27d7dc9700: [    download.cpp:1253] md_download_set_wakeup: DEBUG: Wake up download set 0x7f27c8002f28
6017248.341863707 00584:00007f27cffff700: [    download.cpp:1241] md_download_context_cancel: DEBUG: cancelled 0x7f27c8003290
6017248.341919451 00584:00007f27cffff700: [    download.cpp:0443] md_downloader_insert_cancelling: DEBUG: Cancel download context 0x7f27c80033f0
6017248.341944955 00584:00007f27cffff700: [    download.cpp:0484] md_downloader_insert_cancelling: DEBUG: download 0x7f27c80033f0 ref 3
6017248.341961165 00584:00007f27cffff700: [    download.cpp:1233] md_download_context_cancel: DEBUG: Wait for cancelling download 0x7f27c80033f0 (set 0x7f27c8002f28)
6017248.341976331 00584:00007f27cffff700: [    download.cpp:1011] md_download_context_wait: DEBUG: Wait on download context 0x7f27c80033f0 (18446744073709551615 millis)
* Closing connection 2
6017248.347023387 00584:00007f27d7dc9700: [    download.cpp:0618] md_downloader_end_all_cancelling: DEBUG: download 0x7f27c80033f0 ref 2
6017248.347059215 00584:00007f27d7dc9700: [    download.cpp:1407] md_downloader_finalize_download_context: DEBUG: Finalized download context 0x7f27c80033f0 (http://syndicate-ms-datasets.appspot.com:80/FILE/LISTDIR/8593192222588951184.1.1496814602/0?page_id=2)
6017248.347081636 00584:00007f27d7dc9700: [    download.cpp:1418] md_downloader_finalize_download_context: DEBUG: download 0x7f27c80033f0 ref 1
6017248.347096141 00584:00007f27d7dc9700: [    download.cpp:1253] md_download_set_wakeup: DEBUG: Wake up download set 0x7f27c8002f28
6017248.347146344 00584:00007f27cffff700: [    download.cpp:1241] md_download_context_cancel: DEBUG: cancelled 0x7f27c80033f0
6017248.347210963 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c8003290 ref 0 (from download.cpp:917)
6017248.347228096 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c8003290, from download.cpp:920 (refcount 0)
6017248.347280436 00584:00007f27cffff700: [    download.cpp:0896] md_download_context_unref2: DEBUG: download 0x7f27c80033f0 ref 0 (from download.cpp:917)
6017248.347297703 00584:00007f27cffff700: [    download.cpp:0842] md_download_context_free2: DEBUG: Free download context 0x7f27c80033f0, from download.cpp:920 (refcount 0)
6017248.347323661 00584:00007f27cffff700: [    download.cpp:2115] md_download_loop_cleanup: DEBUG: Clean up download loop 0x7f27c8002f10 (set 0x7f27c8002f28)
6017248.347339828 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c8003290 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
6017248.347365402 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c8003340 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
...
6017248.349023811 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c80075f0 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
6017248.349039730 00584:00007f27cffff700: [    download.cpp:2123] md_download_loop_cleanup: DEBUG: Clear download 0x7f27c80076a0 from set 0x7f27c8002f28 in loop 0x7f27c8002f10
6017248.349078149 00584:00007f27cffff700: [    download.cpp:1074] md_download_set_free: DEBUG: Free download set 0x7f27c8002f28
6017248.349095154 00584:00007f27cffff700: [  ms/listdir.cpp:0514] ms_client_get_dir_metadata: DEBUG: Downloaded 20 children (out of 29)
6017248.349183568 00584:00007f27cffff700: [ consistency.cpp:2125] UG_consistency_dir_ensure_fresh: ERROR: ms_client_listdir('/') rc = -121
00584:00007F27CFFFF700: [fskit          entry.c:1378] fskit_entry_wlock2 DEBUG: 0x2515ba0: 0, from path.c:825
00584:00007F27CFFFF700: [fskit          entry.c:1399] fskit_entry_unlock2 DEBUG: 0x2515ba0: 0, from path.c:855
6017248.349255956 00584:00007f27cffff700: [          fs.cpp:0296] UG_fs_open: ERROR: UG_consistency_dir_ensure_fresh('/') rc = -121
00584:00007F27CFFFF700: [fskit          route.c:0640] fskit_route_dispatch ERROR: fskit_safe_dispatch(3) rc = -121
00584:00007F27CFFFF700: [fskit          entry.c:1378] fskit_entry_wlock2 DEBUG: 0x2515ba0: 0, from opendir.c:98
00584:00007F27CFFFF700: [fskit        opendir.c:0103] fskit_opendir ERROR: fskit_run_user_open(/) rc = -121
00584:00007F27CFFFF700: [fskit          entry.c:1399] fskit_entry_unlock2 DEBUG: 0x2515ba0: 0, from opendir.c:104
00584:00007F27CFFFF700: [fskit          entry.c:1378] fskit_entry_wlock2 DEBUG: 0x2515ba0: 0, from path.c:825
00584:00007F27CFFFF700: [fskit          entry.c:1399] fskit_entry_unlock2 DEBUG: 0x2515ba0: 0, from path.c:855
00584:00007F27CFFFF700: [fskit     fskit_fuse.c:0671] fskit_fuse_opendir DEBUG: opendir(/, 0x7f27cfffec10) rc = -121