Double Upload?

[GoogleCodeExporter]

Hi,
first of all nice work ! :)

Got a Problem in version V1.19 R271.

I startet s3fs in console mode and watched the output.

If im using this command (test doesnt exist yet):

dd if=/dev/zero bs=1 count=1 >> test

i get followring daemon output:

getattr[path=/test]
mknod[path=/test][mode=33188]
getattr[path=/test]
open[path=/test][flags=33793]
downloading[path=/test][fd=4]
flush[path=/test][fd=4]
uploading[path=/test][fd=4][size=0]
flush[path=/test][fd=4]
uploading[path=/test][fd=4][size=1]
release[path=/test][fd=4]

if i use this command again, i get this output:

getattr[path=/test]
open[path=/test][flags=33793]
flush[path=/test][fd=4]
uploading[path=/test][fd=4][size=1]
flush[path=/test][fd=4]
uploading[path=/test][fd=4][size=2]
release[path=/test][fd=4]

Is there any (amazon reason?) why its uploaded _twice_?

(i just added 1 byte so it should only upload the 2 byte file again and not the 
1 byte file and then again the 2byte file or am i wrong?)

Im using Debian squeeze for this test. ~

Original issue reported on code.google.com by sascha.a...@gmail.com on 8 Dec 2010 at 3:38

[GoogleCodeExporter]

Issue 135 has been merged into this issue.

Original comment by dmoore4...@gmail.com on 8 Dec 2010 at 11:23

[GoogleCodeExporter]

That is an interesting observation.  Admittedly, I don't know why this is 
the case. ...seems counter intuitive.

Here's the result of my similar experiment:

% echo "1234" > numbers.txt

getattr[path=/numbers.txt]
mknod[path=/numbers.txt][mode=33204]
getattr[path=/numbers.txt]
open[path=/numbers.txt][flags=32769]
downloading[path=/numbers.txt][fd=5]
flush[path=/numbers.txt][fd=5]
uploading[path=/numbers.txt][fd=5][size=0]
write[path=/numbers.txt]
flush[path=/numbers.txt][fd=5]
uploading[path=/numbers.txt][fd=5][size=5]
release[path=/numbers.txt][fd=5]

% echo "56789ABC" >> numbers.txt

getattr[path=/numbers.txt]
open[path=/numbers.txt][flags=33793]
downloading[path=/numbers.txt][fd=5]

(s3fs_flush called from FUSE)
flush[path=/numbers.txt][fd=5]
uploading[path=/numbers.txt][fd=5][size=5]

(s3fs_write called from FUSE)
write[path=/numbers.txt]

(s3fs_flush called from FUSE)
flush[path=/numbers.txt][fd=5]
uploading[path=/numbers.txt][fd=5][size=14]

(s3fs_release called from FUSE)
release[path=/numbers.txt][fd=5]

So I added some comments above on the second write.  s3fs reacts to FUSE's
requests. So here is the sequence from FUSE on the second write:

getattr
open
flush
write
flush
release

s3fs's flush handler does an upload upon every flush. Since there is
two flushes during this file system write, it gets uploaded each time.

I do not know why the code was originally written in this manner. ...is it
right, is it wrong? ...I do not know.

It does seem that the flush after open shouldn't re-upload. My guess at this
point in time is that this is a "big hammer" to account for keeping amazon's
meta data that is associated with the file from getting out of whack.

It certainly seems that this can hurt performance.

Original comment by dmoore4...@gmail.com on 17 Dec 2010 at 5:21

[GoogleCodeExporter]

I don't know how to respond to this other than "that's the way that it is"

There was recently an issue found with redirections and appends (> and >>) on 
some systems that got fixed, maybe the fix has an impact (hopefully positive) 
on this issue.

Original comment by dmoore4...@gmail.com on 5 Feb 2011 at 1:45

[GoogleCodeExporter]

see r322

Original comment by ben.lema...@gmail.com on 22 Feb 2011 at 9:32

[GoogleCodeExporter]

Can the original reporter of this issue give r322 (or later) a try and report 
back as to if this either resolves this issue or makes it better? Thanks.

Original comment by dmoore4...@gmail.com on 26 Feb 2011 at 6:25

[GoogleCodeExporter]

Sacha,  last chance to respond before this issue gets closed.

Original comment by dmoore4...@gmail.com on 7 Mar 2011 at 7:32

[GoogleCodeExporter]

Original comment by dmoore4...@gmail.com on 9 Mar 2011 at 1:56

• Changed state: Fixed