search

pradeepgn / s3fs

Automatically exported from code.google.com/p/s3fs
GNU General Public License v2.0
0 stars 0 forks source link

FTP Client Upload Retry/Timeout #414

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Detailed description of observed behavior:
When uploading a file via FTP access to a mounted folder, the file is 
successfully uploaded to s3, but this is status is not returned to the client.  
Instead, the client seems to disconnect and reconnect, and the user is the 
asked if he/she would like to override the file that was just uploaded.
This may be a FileZilla specific problem?

What steps will reproduce the problem - please be very specific and
detailed. (if the developers cannot reproduce the issue, then it is
unlikely a fix will be found)?
Install and have vsftpd running
Mount s3 to a folder:
sudo s3fs -o allow_other -o use_cache=/home/ftp/fake_ftp_publisher_cache -o 
connect_timeout=30 -f -d my-bucket:/fake_ftp_publisher 
/home/ftp/fake_ftp_publisher
Use FileZilla to upload a file into /home/ftp/fake_ftp_publisher

The log of the upload request looks like this:
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_getattr(691): [path=/apple-touch-icon.png]
    HeadRequest(1920): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png
    RequestPerform(1483): HTTP response code 404
    RequestPerform(1507): HTTP response code 404 was returned, returning ENOENT
    HeadRequest(1920): [tpath=/apple-touch-icon.png/]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png/
    RequestPerform(1483): HTTP response code 404
    RequestPerform(1507): HTTP response code 404 was returned, returning ENOENT
    HeadRequest(1920): [tpath=/apple-touch-icon.png_$folder$]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png_%24folder%24
    RequestPerform(1483): HTTP response code 404
    RequestPerform(1507): HTTP response code 404 was returned, returning ENOENT
  list_bucket(2225): [path=/apple-touch-icon.png]
    ListBucketRequest(2270): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com?delimiter=/&prefix=fake_ftp_publisher/apple-touch-icon.png/&max-keys=1000
    RequestPerform(1483): HTTP response code 200
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
s3fs_create(798): [path=/apple-touch-icon.png][mode=100644][flags=34881]
    HeadRequest(1920): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png
    RequestPerform(1483): HTTP response code 404
    RequestPerform(1507): HTTP response code 404 was returned, returning ENOENT
    HeadRequest(1920): [tpath=/apple-touch-icon.png/]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png/
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_opendir(2050): [path=/][flags=624640]
    RequestPerform(1483): HTTP response code 404
    RequestPerform(1507): HTTP response code 404 was returned, returning ENOENT
    HeadRequest(1920): [tpath=/apple-touch-icon.png_$folder$]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png_%24folder%24
    RequestPerform(1483): HTTP response code 404
    RequestPerform(1507): HTTP response code 404 was returned, returning ENOENT
  list_bucket(2225): [path=/apple-touch-icon.png]
    ListBucketRequest(2270): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com?delimiter=/&prefix=fake_ftp_publisher/apple-touch-icon.png/&max-keys=1000
    RequestPerform(1483): HTTP response code 200
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
    create_file_object(757): [path=/apple-touch-icon.png][mode=100644]
    PutRequest(2042): [tpath=/apple-touch-icon.png]
    PutRequest(2056): create zero byte file object.
    PutRequest(2135): uploading... [path=/apple-touch-icon.png][fd=-1][size=0]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png
    RequestPerform(1483): HTTP response code 200
    DelStat(375): delete stat cache entry[path=/apple-touch-icon.png]
s3fs_getattr(691): [path=/apple-touch-icon.png]
    HeadRequest(1920): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png
    RequestPerform(1483): HTTP response code 200
    AddStat(247): add stat cache entry[path=/apple-touch-icon.png]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878662][hit count=0]
s3fs_truncate(1816): [path=/apple-touch-icon.png][size=0]
s3fs_readdir(2182): [path=/]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878662][hit count=1]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878662][hit count=2]
  list_bucket(2225): [path=/]
    ListBucketRequest(2270): [tpath=/]
    PutRequest(2042): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com?delimiter=/&prefix=fake_ftp_publisher/&max-keys=1000
    PutRequest(2135): uploading... [path=/apple-touch-icon.png][fd=5][size=0]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_access(2646): [path=/][mask=X_OK ]
s3fs_access(2646): [path=/][mask=X_OK ]
    RequestPerform(1483): HTTP response code 200
    DelStat(375): delete stat cache entry[path=/apple-touch-icon.png]
s3fs_getattr(691): [path=/apple-touch-icon.png]
    HeadRequest(1920): [tpath=/apple-touch-icon.png]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com/fake_ftp_publisher/apple-touch-icon.png
s3fs_opendir(2050): [path=/][flags=624640]
    RequestPerform(1483): HTTP response code 200
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
  readdir_multi_head(2105): [path=/][list=0]
    GetStat(170): stat cache hit [path=/autogeneratecsvfile.csv][time=1395878590][hit count=5]
    GetStat(170): stat cache hit [path=/behat-logo.png][time=1395878590][hit count=5]
    GetStat(170): stat cache hit [path=/geocoder sql.txt][time=1395878590][hit count=5]
    GetStat(170): stat cache hit [path=/spectrocat.png][time=1395878590][hit count=5]
    GetStat(170): stat cache hit [path=/test_text.txt][time=1395878590][hit count=5]
    Request(3150): [count=1]
    RequestPerform(1483): HTTP response code 200
    AddStat(247): add stat cache entry[path=/apple-touch-icon.png]
    AddStat(247): add stat cache entry[path=/apple-touch-icon.png]
    DelStat(375): delete stat cache entry[path=/apple-touch-icon.png]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878674][hit count=0]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878674][hit count=1]
    GetStat(170): stat cache hit [path=/autogeneratecsvfile.csv][time=1395878668][hit count=6]
    GetStat(170): stat cache hit [path=/behat-logo.png][time=1395878668][hit count=6]
    GetStat(170): stat cache hit [path=/geocoder sql.txt][time=1395878668][hit count=6]
    GetStat(170): stat cache hit [path=/spectrocat.png][time=1395878668][hit count=6]
    GetStat(170): stat cache hit [path=/test_text.txt][time=1395878668][hit count=6]
s3fs_getattr(691): [path=/apple-touch-icon.png]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878674][hit count=2]
s3fs_readdir(2182): [path=/]
s3fs_getattr(691): [path=/autogeneratecsvfile.csv]
  list_bucket(2225): [path=/]
    ListBucketRequest(2270): [tpath=/]
    GetStat(170): stat cache hit [path=/autogeneratecsvfile.csv][time=1395878674][hit count=7]
    RequestPerform(1467): connecting to URL http://my-bucket.s3.amazonaws.com?delimiter=/&prefix=fake_ftp_publisher/&max-keys=1000
s3fs_getattr(691): [path=/behat-logo.png]
    GetStat(170): stat cache hit [path=/behat-logo.png][time=1395878674][hit count=7]
s3fs_getattr(691): [path=/geocoder sql.txt]
    GetStat(170): stat cache hit [path=/geocoder sql.txt][time=1395878674][hit count=7]
s3fs_getattr(691): [path=/spectrocat.png]
    GetStat(170): stat cache hit [path=/spectrocat.png][time=1395878674][hit count=7]
s3fs_getattr(691): [path=/test_text.txt]
    GetStat(170): stat cache hit [path=/test_text.txt][time=1395878674][hit count=7]
    RequestPerform(1483): HTTP response code 200
    append_objects_from_xml_ex(2299): contents_xp->nodesetval is empty.
  readdir_multi_head(2105): [path=/][list=0]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878674][hit count=3]
    GetStat(170): stat cache hit [path=/autogeneratecsvfile.csv][time=1395878674][hit count=8]
    GetStat(170): stat cache hit [path=/behat-logo.png][time=1395878674][hit count=8]
    GetStat(170): stat cache hit [path=/geocoder sql.txt][time=1395878674][hit count=8]
    GetStat(170): stat cache hit [path=/spectrocat.png][time=1395878674][hit count=8]
    GetStat(170): stat cache hit [path=/test_text.txt][time=1395878674][hit count=8]
    Request(3150): [count=0]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878679][hit count=4]
    GetStat(170): stat cache hit [path=/autogeneratecsvfile.csv][time=1395878679][hit count=9]
    GetStat(170): stat cache hit [path=/behat-logo.png][time=1395878679][hit count=9]
    GetStat(170): stat cache hit [path=/geocoder sql.txt][time=1395878679][hit count=9]
    GetStat(170): stat cache hit [path=/spectrocat.png][time=1395878679][hit count=9]
    GetStat(170): stat cache hit [path=/test_text.txt][time=1395878679][hit count=9]
s3fs_getattr(691): [path=/apple-touch-icon.png]
    GetStat(170): stat cache hit [path=/apple-touch-icon.png][time=1395878679][hit count=5]
s3fs_getattr(691): [path=/autogeneratecsvfile.csv]
    GetStat(170): stat cache hit [path=/autogeneratecsvfile.csv][time=1395878679][hit count=10]
s3fs_getattr(691): [path=/behat-logo.png]
    GetStat(170): stat cache hit [path=/behat-logo.png][time=1395878679][hit count=10]
s3fs_getattr(691): [path=/geocoder sql.txt]
    GetStat(170): stat cache hit [path=/geocoder sql.txt][time=1395878679][hit count=10]
s3fs_getattr(691): [path=/spectrocat.png]
    GetStat(170): stat cache hit [path=/spectrocat.png][time=1395878679][hit count=10]
s3fs_getattr(691): [path=/test_text.txt]
    GetStat(170): stat cache hit [path=/test_text.txt][time=1395878679][hit count=10]

===================================================================
The following information is very important in order to help us to help
you.  Omission of the following details may delay your support request or
receive no attention at all.
===================================================================
Version of s3fs being used (s3fs --version):
Amazon Simple Storage Service File System 1.74
Copyright (C) 2010 Randy Rizun <rrizun@gmail.com>
License GPL2: GNU GPL version 2 <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Version of fuse being used (pkg-config --modversion fuse):
2.9.3

System information (uname -a):
Linux devbox 3.2.0-4-amd64 #1 SMP Debian 3.2.41-2+deb7u2 x86_64 GNU/Linux

Distro (cat /etc/issue):
Debian GNU/Linux 7 \n \l

s3fs command line used (if applicable):
sudo s3fs -o allow_other -o use_cache=/home/ftp/fake_ftp_publisher_cache -o 
connect_timeout=30 -f -d my-bucket:/fake_ftp_publisher 
/home/ftp/fake_ftp_publisher

/etc/fstab entry (if applicable):
N/A

s3fs syslog messages (grep s3fs /var/log/syslog):
None

Original issue reported on code.google.com by allen.t...@paperg.com on 27 Mar 2014 at 12:22

GoogleCodeExporter commented 8 years ago 
It seems that s3fs finished uploading a file without problem from your logs.
Mybe I think this timeout occurred by FileZilla.

* Please use latest version s3fs which is moved to 
Github(https://github.com/s3fs-fuse/s3fs-fuse).

Thanks,

Original comment by ggta...@gmail.com on 6 Apr 2014 at 1:52

GoogleCodeExporter commented 8 years ago 
Hi,

s3fs sends parallel requests as default, the mixmim parallel count is 20 as 
default.
if you can, please try to set "multireq_max" option to small value.
Maybe it works good without timeout.

Regards,

Original comment by ggta...@gmail.com on 1 Jun 2014 at 3:24