s3fs needs deep directory rename!

[GoogleCodeExporter]

createrepo doesn't work: needs deep directory rename!

createrepo uses a working folder of ".repodata" and then renames it to
"repodata"... s3fs does not support deep rename of folders therefore you
end up with an empty "repodata" folder!

Original issue reported on code.google.com by rri...@gmail.com on 29 Feb 2008 at 12:24

[GoogleCodeExporter]

Original comment by rri...@gmail.com on 4 Apr 2008 at 3:50

• Changed title: createrepo cannot work: s3fs needs deep directory rename!

[GoogleCodeExporter]

This is a problem. I would just like to have a centralized file-system for app 
files I am 
storing on EC2 across zones, but a mv  of a directory effectively deletes its 
contents 
(until you put it back).

mkdir -p test/dir/subdir
mv test test2
ls test2 (empty)
mv test2 test
ls test (dir)

I haven't looked at the source at all, but is it possible to use threading or 
something 
to rename all the files at once so that large directories don't take forever to 
move 
while each file underneath gets renamed individually?

Original comment by jacwri...@gmail.com on 25 Mar 2010 at 6:14

[GoogleCodeExporter]

I have the seam problem the command "mv" not work. I can't move files inside 
the s3fs partition any idea how to solve the problem?

Original comment by aralking on 12 Dec 2010 at 8:36

[GoogleCodeExporter]

Started looking at this one today. Looks straightforward, however, there are a 
few gotcha's lurking.  I've been successful in moving an empty directory 
around. ...and renaming plain files contained within shouldn't be too much of a 
problem either. The trick comes with re-creating all of the subdirectories 
contained within the directory hierarchy.  

Original comment by dmoore4...@gmail.com on 20 Dec 2010 at 5:33

• Changed title: s3fs needs deep directory rename!
• Changed state: Started
• Added labels: Priority-High
• Removed labels: Priority-Medium

[GoogleCodeExporter]

Here is the algorithm that I'm working on, comments are welcome.

- get the full hierarchical listing of the directory, along with whether the 
"node" is a directory itself or not and save in a linked list (also saved in 
the structure is the new full hierarchical name)

(nothing has changed yet)

- iterate over the list and create (copy) the new directory objects, preserving 
meta data

(S3 now contains a skeleton of the new directory structure, old data is still 
intact)

- iterate over the list and do a server side copy, preserving meta data, after 
successful copy of file, delete the original

(S3 now contains the moved data, but the directory skeleton of the old still 
exists)

- iterate over the list and remove the old, now empty, directories

This isn't bulletproof, but presumably will not result in loss of data in event 
of a service interruption -- you may have a mess on your hands, but the data 
will not have been destroyed

Original comment by dmoore4...@gmail.com on 21 Dec 2010 at 3:29

[GoogleCodeExporter]

Resolved in r289

Implemented as outlined above and tested on Ubuntu 10.10

A directory rename can take a long time. I renamed (moved) a directory within 
the bucket whose virtual
directory structure was two levels deep and consisted of about 1600 objects 
(file objects and directory
objects).  Overall speed is dominated by connection speed. With my typical 
residential broadband connection,
this task took fifteen minutes.  (the move does NOT download the files and 
reupload them, this is a server
side copy).

During one experiment, I did experience a connection timeout that didn't 
succeed after the default
number of retries.  The mv returned an I/O error and left the data co-located 
between the old directory
structure and the new directory.  The good news is that no data loss resulted.  
I manually cleaned this
up by moving the remaining files to the new locations and deleting the empty 
directories.  

The main motivation for fixing this was to get "git" working in a mounted 
bucket directory, issue #129 
Unfortunately, resolving this issue did not clear that one up.

Original comment by dmoore4...@gmail.com on 21 Dec 2010 at 4:58

• Changed state: Fixed