directory listings and libcurl 7.19.4

[GoogleCodeExporter]

Directory listings return errors under the following combination of conditions:

1. s3fs is compiled with libcurl 7.19.4 in gentoo x86 using either 'emerge
s3fs' or make.
2. The directory being listed contains a subdirectory.
3. The program doing the listing is a graphical ftp or scp client such as
Filezilla, WinSCP, or Gnome "Connect to Server" FTP browser.

Error returned by Filezilla FTP client for Windows:
{{{Command: LIST
Response:   150 Here comes the directory listing.
Response:   500 OOPS: invalid inode size in vsf_sysutil_statbuf_get_size
Error:  Failed to retrieve directory listing}}}

Error returned by WinSCP SCP client for Windows:
{{{Error listing directory '/mnt/mybucketname'.
Unexpected directory listing line 'drwxr-xr-x 1 root root
18446744073709551615 2009-03-09 20:52:46.000000000 -0700 mydirectoryname'.}}}

The correct listing and no errors are returned under the above conditions
with 'ls /mnt/mybucketname/', command line ftp, and the Gnome "Connect to
Server" SSH file browser.

I then tested the same scenario in Ubuntu 8.04 and s3f3 compiled with
libcurl 7.18.0,  keeping everything else the same.  There are no errors
when the same bucket is mounted in Ubuntu 8.04 and s3fs is compiled in
Ubuntu with libcurl 7.18.0.

Gentoo x86, s3fs r177, libcurl 7.19.4:
{{{# ls -al /mnt/mybucketname/
total 0
drwxr-xr-x 1 root root 18446744073709551615 Mar  9 20:52 mydirectoryname}}}

Ubuntu 8.04, s3fs r177, libcurl 7.18.0:
{{{# ls -al /mnt/mybucketname/
total 0
drwxr-xr-x 1 root root 0 2009-03-09 20:52 mydirectoryname}}}

Gentoo returns a strangely large value (16 exabytes) for the size of the
directory object.  Could this be related to libcurl?

Note that Gentoo just cleaned out all but the newest versions of libcurl
from the Portage tree so there is no rolling back for some users.

http://sources.gentoo.org/viewcvs.py/gentoo-x86/net-misc/curl/ChangeLog?view=mar
kup

Original issue reported on code.google.com by mie...@gmail.com on 10 Mar 2009 at 8:09

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

one more thing.. 

s3fs r177 in Ubuntu (working):

# make
g++ -ggdb -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse  -pthread -lfuse 
-lrt -ldl
   -Wl,-Bsymbolic-functions -lcurl -lgssapi_krb5   -I/usr/include/libxml2 -lxml2
-lcrypto s3fs.cpp -o s3fs
s3fs.cpp:440: warning: ‘size_t readCallback(void*, size_t, size_t, void*)’ 
defined
but not used
ok!

s3fs r177 compiling in Gentoo (not working):

# make
g++ -ggdb -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse  -pthread -lfuse 
-lrt -ldl
   -lcurl   -I/usr/include/libxml2 -lxml2 -lz -lm -lcrypto s3fs.cpp -o s3fs
s3fs.cpp:440: warning: 'size_t readCallback(void*, size_t, size_t, void*)' 
defined
but not used
ok!

same error, different results.

Original comment by mie...@gmail.com on 10 Mar 2009 at 9:52

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

It is definitely the update from curl-7.18.2 to curl-7.19.4 that introduces 
this bug 
in gentoo.  s3fs does not need to be recompiled.

As soon as curl is updated,  ls begins returning the '18446744073709551615' 
size 
attribute,  which seems to trigger other issues.

Original comment by mie...@gmail.com on 12 Mar 2009 at 7:55

[GoogleCodeExporter]

I ran into the same problem. It turns out that starting in curl 7.19.4, curl 
began
returning -1 if the Content-Length cannot be determined. from the 
curl_easy_getinfo
man page ...

       CURLINFO_CONTENT_LENGTH_DOWNLOAD
              Pass a pointer to a double to receive the content-length of the
              download. This is the value read from the  Content-Length:  field.
              Since 7.19.4, this returns -1 if the size isn't known.

for whatever reason, curl is returning -1 when Content-Length is 0. I'm not 
sure why
that is the case, I would think it would actually return 0, unless it considers 
0 to
be an unknown Content-Length.

I wrote a little patch to handle this, although I haven't full tested it to 
make sure
it doesn't break other things. Use are your own risk. Patch Attached.

Original comment by Gabriel....@gmail.com on 20 Oct 2009 at 4:57

Attachments:

• s3fs.patch

[GoogleCodeExporter]

I would like to take a look at this issue.  Unfortunately, I am unable to 
reproduce it in my current setups. Can the community confirm that it is still 
an issue with r204 and latest version of curl (I'm using curl 7.21.2 on my 
Debian machine)?

If so, please update this issue with the steps necessary to reproduce the 
issue. Thanks.

Original comment by dmoore4...@gmail.com on 19 Oct 2010 at 4:30

[GoogleCodeExporter]

No feedback on this in over a month. If this issue persists or is seen again, 
please open a new issue. Closing this old issue.

Original comment by dmoore4...@gmail.com on 3 Dec 2010 at 5:17

• Changed state: WontFix

[GoogleCodeExporter]

This bug persists into the current s3fs-1.61 release.  Running `touch 
/mnt/s3fs/foo` reliably breaks the s3fs fuse mount on a vanilla Ubuntu 11.10 
Oneiric system.  (libcurl3_7.21.6-3ubuntu3_amd64.deb)

I updated Gabriel's patch from comment #5, which is attached.

Original comment by dajhorn@vanadac.com on 13 Nov 2011 at 5:10

Attachments:

• s3fs-1.61-bug50.patch