search

skoobe / riofs

Userspace S3 filesystem
GNU General Public License v3.0
393 stars 60 forks source link

riofs on rhel #128

Closed vgadhadaran closed 7 years ago

vgadhadaran commented 8 years ago

I installed the riofs on rhel server by following the below installation steps

sudo yum groupinstall "Development Tools" sudo yum install glib2-devel fuse-devel libevent-devel libxml2-devel openssl-devel libmagic-dev cp /tmp/libevent-2.0.22-stable.tar.gz /usr/local/src cd /usr/local/src tar -zxvf libevent-2.0.22-stable.tar.gz cd libevent-2.0.22-stable
./configure make sudo make install sudo echo "/usr/local/lib/" > /etc/ld.so.conf.d/riofs.conf sudo ldconfig export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig cd cd /tmp cp -r riofs-master /usr/local/src/ cd /usr/local/src cd riofs-master/ chmod +x autogen.sh ./autogen.sh ./configure make sudo make install

Now when I try to move the data, it throws below error,

[root@k8002166 who]# mv 500m1 /tmp/wow mv: writing `/tmp/wow/500m1': No such file or directory

Anythoughts?

wizzard commented 8 years ago

Hello, please provide more information, how do you start riofs, which command arguments do you use ? Try to run riofs with -l [log_file_name] command option to save logs, then paste it here.

vgadhadaran commented 8 years ago

starting riofs:

riofs ritestdemo1 /tmp/wow -c /usr/local/etc/riofs/riofs.conf.xml -o "nomultipart,sigv2,allow_other,nonempty" -v --log-file=/var/log/riofs.log

XML configuration:

False False False ``` 2 ``` 2 2 ``` 4 100 ``` http://cloud-poc.hcpdevn1.xxx.xxx ``` 1000 0 False STANDARD xxx xxx ``` -1 ``` -1 20 20 ``` -1 -1 -1 -1 ``` 0 10 False False /tmo/null 1 0 ``` False ``` 127.0.0.1 8011 /stats 1000 ```

Log File: It writes 500mb and then below is the log( only highlighting the failure portion of log file and not the entire content)

18:01:35 [dir_tree](dir_tree_file_write dir_tree.c:1499) [ino: 16, fop: 0xf4bcb0] write inode, size: 4096, off: 524279808 18:01:35 [fio](fileio_write_buffer file_io_ops.c:680) [ino: 16] Write buf size: 524283904 18:01:35 [cmng](cache_mng_store_file_buf cache_mng.c:419) [ino: 16] Written [524279808:4096] bytes, result: OK 18:01:35 [fuse](rfuse_write_cb rfuse.c:717) [req: 0xf53470] write_cb success: YES 18:01:35 [dir_tree](dir_tree_on_buffer_written_cb dir_tree.c:1445) [ino: 16, fop: 0xf4bcb0] Buffer written, count: 4096 18:01:35 [fuse](rfuse_write rfuse.c:732) [ino: 16, fi: 0x7fffca8f0ee0] write inode, size: 4096, off: 524283904 18:01:35 [dir_tree](dir_tree_file_write dir_tree.c:1499) [ino: 16, fop: 0xf4bcb0] write inode, size: 4096, off: 524283904 18:01:35 [fio](fileio_write_buffer file_io_ops.c:680) [ino: 16] Write buf size: 524288000 18:01:35 [cmng](cache_mng_store_file_buf cache_mng.c:419) [ino: 16] Written [524283904:4096] bytes, result: OK 18:01:35 [con](http_connection_acquire http_connection.c:174) [con: 0xf4c150] Connection object is acquired! 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (19msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 1 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (11msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 2 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 3 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (1msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 4 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 5 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (1msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 6 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 7 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 8 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 9 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 10 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 11 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 12 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 13 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 14 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 15 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 16 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 17 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 18 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 19 of 20 18:01:35 [con](http_connection_make_request http_connection.c:814) [con: 0xf4c150] POST /ritestdemo1/500m1?uploads bucket: ritestdemo1, host: cloud-poc.hcpdevn1.xxx.xxx, out_len: 0 18:01:35 [con](http_connection_on_responce_cb http_connection.c:464) [con: 0xf4c150] Got HTTP response from server! (2msec) 18:01:35 [con](http_connection_on_responce_cb http_connection.c:591) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented). AWS message: The request requires functionality that is not implemented in the current release 18:01:35 [con](http_connection_on_responce_cb http_connection.c:602) [con: 0xf4c150] Server returned HTTP error: 501 (Not Implemented)! Retry ID: 20 of 20 18:01:35 [con](http_connection_on_responce_cb http_connection.c:605) [con: 0xf4c150] Reached the maximum number of retries ! 18:01:35 [con](http_connection_release http_connection.c:183) [con: 0xf4c150] Connection object is released! 18:01:35 [fio](fileio_write_on_multipart_init_cb file_io_ops.c:598) [ino: 16, con: 0xf4c150] Failed to get multipart init data from the server ! 18:01:35 [fuse](rfuse_write_cb rfuse.c:717) [req: 0xf52a90] write_cb success: NO 18:01:35 [dir_tree](dir_tree_on_buffer_written_cb dir_tree.c:1445) [ino: 16, fop: 0xf4bcb0] Buffer written, count: 0 18:01:35 [fuse](rfuse_getxattr rfuse.c:927) getxattr for: 16 attr name: security.capability size: 0 18:01:35 [dir_tree](dir_tree_getxattr dir_tree.c:2261) [ino: 16] Getting Xattributes .. 18:01:35 [dir_tree](dir_tree_getxattr dir_tree.c:2285) Xattr :security.capability not supported! 18:01:35 [fuse](rfuse_getxattr_cb rfuse.c:905) [ino: 16] getattr_cb success: NO str: (null) 18:01:35 [fuse](rfuse_write rfuse.c:732) [ino: 16, fi: 0x7fffca8f0ee0] write inode, size: 4096, off: 524283904 18:01:35 [dir_tree](dir_tree_file_write dir_tree.c:1499) [ino: 16, fop: 0xf4bcb0] write inode, size: 4096, off: 524283904 18:01:35 [fio](fileio_write_buffer file_io_ops.c:671) [ino: 16] Write call with offset 524283904 is not allowed ! 18:01:35 [fuse](rfuse_write_cb rfuse.c:717) [req: 0xf4fe60] write_cb success: NO 18:01:35 [dir_tree](dir_tree_on_buffer_written_cb dir_tree.c:1445) [ino: 16, fop: 0xf4bcb0] Buffer written, count: 0 18:01:35 [fuse](rfuse_flush rfuse.c:1047) [0xf4e7d0][req: 0xf52a90] flush ino: 16 18:01:35 [fuse](rfuse_flush_cb rfuse.c:1033) [req: 0xf52a90] flush_cb success: YES 18:01:35 [fuse](rfuse_release rfuse.c:676) [ino: 16, fi: 0x7fffca8f0ee0] release inode, flags: 32769 18:01:35 [dir_tree](dir_tree_file_release dir_tree.c:1361) [ino: 16, fop: 0xf4bcb0] dir_tree_file_release 18:01:35 [fio](fileio_release_on_part_con_cb file_io_ops.c:321) [ino: 16, con: 0xf4c150] Releasing fop. Size: 524288000 18:01:38 [fio](fileio_release_on_part_con_cb file_io_ops.c:341) [ino: 16, con: 0xf4c150] UploadID is not set, aborting operation !

Iam writing data to the file on non-RioFS folder, then copy file to RioFS mounted folder.

As we have massive amount of data in terrabytes we need move to s3 storage. All our data varies from 250mb, 500mb, 1 gb/5/10/100/250 gb is sizes. We had success only moving 250 mb of data although it took 2 hours. The above failure happened when moving 500mb of data. any help appreciated.

currently i see write inode size is 4096 is there any way to increase it as it takes hugh time to copy massive data?

wizzard commented 8 years ago

Most likely you are trying to use RioFS and AWS region with only version 4 support (please check http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region), unfortunately at this point RioFS supports only version 2 regions.

vgadhadaran commented 8 years ago

I am connecting to Hitachi content platform and not to AWS.

wizzard commented 8 years ago

I'm afraid I can't help you with this issue, as I never tested RioFS against "Hitachi content platform". As you can see in the log file - server returns many "501" HTTP status codes, most likely Hitachi uses modified or custom S3 REST API.