search

danilop / yas3fs

YAS3FS (Yet Another S3-backed File System) is a Filesystem in Userspace (FUSE) interface to Amazon S3. It was inspired by s3fs but rewritten from scratch to implement a distributed cache synchronized by Amazon SNS notifications. A web console is provided to easily monitor the nodes of a cluster.
http://danilop.github.io/yas3fs
MIT License
644 stars 98 forks source link

Bugfixes: python3, s3 paths with subfolders #178

Closed rebkwok closed 6 years ago

rebkwok commented 6 years ago

A few fixes for python 3 bugs and a couple of others I came across when upgrading to the latest master: 1) d98c9b0: YAS3FSPlugin. load_from_file tries to use e when raising the last exception, however it will never get to this point with e available, because it comes from a previously raised exception. 2) 95e9402: Wrap some dict keys in list() when iterating over them, otherwise it fails on python 3 because it's modifying the dict while iterating over it 3) bcbd913: Python 3 doesn't let you raise simple strings as exceptions any more 4) f4b8e39: The changes from commit 692e36c55f2ac912978d78079ed8d6c050f696e1 in the last PR merged introduce a bug if your s3 path has subfolders. (e.g. if my s3 path is s3://mybucket/foo/bar it will try to do tempfile.mkdtemp with the prefix 'mybucket-foo/bar-random' which fails because it's looking for a folder '/tmp/mybucket-foo/' which doesn't exists. This just replaces / with - in the s3_prefix so the temp folder created is 'mybucket-foo-bar-random'. Also updates the README for --cache-path to match the updated parameter help text 5) 9739764: make sure dirname in replace_empty_dirs is a str before doing replace(), otherwise raises an error on python 3 6) 40cb448: Fix inconsistent tabs and spaces introduced at some point which are making things break for me (yas3fs will not start up without this fix)

rebkwok commented 6 years ago

@danilop I'm leaving this PR open for review, but for our actual use we have reverted to commit c5cf4f8d846308f84698f29a203551c2549ee0e2 (before PR 167 was merged), with the python 3 fixes from here, because we were running into various issues with the more recent changes on master. One fairly major concern was that the randomly named temp cache folders never get removed, and a new one is created each time yas3fs is restarted. Even if the cache is cleared, these (usually) empty folders apparently hang around forever.

jazzl0ver commented 6 years ago

@rebkwok , it would be great if you could fix the conflicts!

rebkwok commented 6 years ago

Conflicts fixed, but I'm still dubious about actually using this as per previous comment

jazzl0ver commented 6 years ago

Are you talking about the temp folders which aren't got removed? That might be similar issue to this one: https://github.com/danilop/yas3fs/issues/14

jazzl0ver commented 6 years ago

@rebkwok would you please create a new PR with your changes against the latest master?

rebkwok commented 6 years ago

@jazzl0ver I rebased this PR already when I fixed the conflicts, so it is against the latest master

jazzl0ver commented 6 years ago

@rebkwok, the point is that the items 5 and 6 from your PR seem to revert the changes I made before to fix some bugs. May I ask you to remove that items from this PR, so I could merge the ones that seem OK. For the items 5 and 6 - let's discuss them separately to make sure they won't break anything.

rebkwok commented 6 years ago

I have updated the PR description. The replace() fix (item 5) is still needed, but i have revised that commit. Item 6 fixes some inconsistent tabs/spaces - not sure when those were introduced, but they stop yas3fs from even starting up for me locally.