Symbolic links stop working after unmount/mount of riofs

[darrencruse]

So this is really a defect against the symbolic link support wizzard has added (currently in the master branch only) under issue #59.

What I've noticed is that creating a link works absolutely fine at first, but if you unmount and remount rio you'll find the link no longer appears to be a link. It simply appears to be a little file whose content is what the link had referred to.

Here's a copy paste of a command line session demonstrating the problem:

$ sudo mount /mnt/production $ cd /mnt/production/objects $ ls ../ weather.txt $ sudo ln -s ../weather.txt weatherlink $ ls -al weatherlink lrwxrwxrwx 1 root root 14 Jul 25 21:32 weatherlink -> ../weather.txt $ cat weatherlink partly cloudy $ cat ../weather.txt partly cloudy $ cd $ sudo umount /mnt/production $ sudo killall riofs $ ps -ef | grep riofs ec2-user 10355 10232 0 21:33 pts/0 00:00:00 grep riofs $ ls /mnt/production $ sudo mount /mnt/production $ cd - /mnt/production/objects $ ls -al weatherlink -rw-r--r-- 1 root root 14 Jul 25 21:32 weatherlink $ cat weatherlink ../weather.txt $

I know the above is a little long but what it shows is that this file "weatherlink" truly appears and functions as a link before the remount but after it acts like just a file.

It's easily reproducible following these steps btw.

Note (for people besides wizzard fwiw): The links are implemented just as a little file in s3 but they have some metadata saying the file is a link. wizzard had been nice enough to do it this way using the same metadata as s3fs since we are transitioning from s3fs to riofs. So it's as if after the unmount/mount that riofs no longer sees it as a link for some reason. Yet - using AWS Console I can see the metadata is there correctly. In fact I can switch the mount over to using s3fs and the link created by riofs still functions correctly under s3fs.

I wondered if it had something to do with the riofs cache, but I blew away my cache under tmp and I can't seem to see any connection it still happens whether the link is cached or not.

[wizzard]

Thank you for the update! I'll investigate this issue in the next few days.

[darrencruse]

Any idea how big a problem this is?

(e.g. when a fix might make it to the master branch?)

Happy to be a play tester - if that helps at all. :)

[wizzard]

Hello, sorry for the delay, my current workload is quite high. But I will try to get back to RioFS at some point during the next week.

[darrencruse]

fwiw I was looking at this a little more just now and I don't know what to make of this (you might make of it that I'm crazy), but I would swear at least some things that had worked when I first tested a couple months ago are not working now. Which makes no sense I haven't touched the riofs install or changed anything related to riofs.

e.g. It is true that if I make a link under riofs it appears in directory listings as a link and works as a link. Until I unmount and remount riofs and then it appears as a file not a link. As I reported above.

But I just noticed if I create the link under s3fs, unmount s3fs and remount as riofs it likewise appears as a file and not a link as it should. But I would have sworn to you I did this very thing when I first tested the link handling a couple months ago and it had worked fine. I even commented as much here in the original issue for the symlink support #59

I just tried the reverse, creating the link while mounted under riofs and unmounting and remounting under s3fs. And in this case s3fs does show the link riofs had created just fine.

Not sure what this means. The metadata in S3 identifying the file as a link seems correct when viewed in AWS Console (as already suggested by the fact that a riofs created link works correctly under s3fs). And removing the riofs cache doesn't seem to change the behavior. So doesn't this suggest riofs is holding something in process (in memory?) that's allowing it to see the file as a link when I've first created it, yet that is not reinitialized from S3 properly once the riofs mount point is unmounted and remounted (i.e. leading to a new riofs process with the remount)?

[wizzard]

Please note, that we are not fully-compatible with s3fs, most likely there are some headers we don't send or values we don't update.

[darrencruse]

Hi wizzard don't know if your comment is directed specifically at what I'd said or just to others who might follow this thread...

Fwiw we started out with s3fs and were disappointed with some performance issues with it - that's mainly what led us to riofs. Unfortunately the issue in this thread had forced us back to using s3fs even though we hadn't wanted to. Because we do have a lot of symlinks being used in our content which is being migrated from a traditional file system over to S3.

I guess that's part of the reason I was jumping between s3fs and riofs (because we're already setup for that).

Not sure if your comment is warning me of something I'm not understanding regarding that.

Our hope is to switch to riofs permanently if this symlink issue is able to be resolved.

But basically I was just using s3fs as a debugging strategy trying to understand how riofs could be losing track of symlinks - even symlinks created purely thru riofs nothing whatsoever to do with s3fs - after an unmount and remount of the bucket.

As I mentioned I was also using AWS Console to look at the metadata values and they looked fine too.

[journeytosilius]

I just compiled the last master version and when I try to mount with --force-head-requests it says it is an unrecognized option, is there something I am missing ? Failed to parse command line options: Unknown option --force-head-requests Maybe a lib ? When I run ./configure everything seems fine.