search

jetwhiz / encfs4win

Windows port of EncFS
https://encfs.win
Other
400 stars 41 forks source link

General stability observations #62

Closed akasubi closed 5 years ago

akasubi commented 7 years ago

Environment

These are some general stability observations.

I don't know whether the issues above are related to encfs, Dokany or both, but I'm posting here to start with. Dokany seems to be part of them though, since the sudden unmounts were more frequent with the previous Dokany version.

jetwhiz commented 7 years ago

Hi @wezeku -- thank you for the detailed report!

Regarding the issue deleting trees in Explorer, I believe this is mostly an issue regarding locked/open files. It might have to do with the desktop.ini hidden system file that Windows places in folders when you open them with Explorer. This is why the problem does not appear when you delete via the command line (and why you can delete them after waiting a bit). I haven't figured out how to solve this problem, or if it is even possible to solve on our end.

The unmounting issue is a long-standing problem, which has definitely gotten better with newer releases of Dokan. Unfortunately there are no solid FUSE options on Windows (as there are on Linux), and this problem is amplified when doing large numbers of heavy reading/writing operations.

For the issues of starting executables from within encfs -- does a similar problem exist when you use the Dokany fuse_mirror sample?

Liryna commented 7 years ago

Sudden unmounts are made when read/write request take too much time and Dokany think your application crashed/deadlock. There is a way to request dokan for more time (currently only proposed on dokan API).

@jetwhiz do you think something should be added on dokan fuse ? (not compatible with fuse so it is going to be #if WINDOWS).

Otherwise there is still the possibility to split the read/write and only return what has been read/write in a correct time.

jetwhiz commented 7 years ago

Hi @Liryna ! Sorry for the delay -- I'm just now finally getting some time to get back on this project!

I think the timeout issue is particularly bad with encfs when working with large files. Even when doing something simple (like viewing Properties in Explorer) can take many seconds for files that are 50 MB+ for 'paranoid' mode.

For files that are many hundreds of MB in size, I think Dokany will definitely think that the program has crashed, so it might be useful to have some way to increase the timeout limit.

Liryna commented 7 years ago

There is the param daemon_timeout that currently fuse has at the mount to personalize the timeout. https://github.com/dokan-dev/dokany/blob/master/dokan_fuse/src/dokanfuse.cpp#L639

Otherwise later with https://github.com/dokan-dev/dokany/pull/307 the speed will be up and we probably will be able to change the logic to make the dokan.dll handle himself the timeout.

jetwhiz commented 7 years ago

That's good to know, I wasn't aware of that option! It might be a good choice to help users debug unmounting issues they experience with encfs4win in the future.

I will keep my eye on https://github.com/dokan-dev/dokany/pull/307 -- is there any idea on when this might be rolled out as a stable release?

Corillian commented 7 years ago

For the record if you care about performance you shouldn't be using the FUSE API wrapper. It not only creates additional overhead but forces blocking IO.

akasubi commented 7 years ago

A follow-up on my initial post: I haven't had the time to compile and try the fuse_mirror sample. However, a problem I had long ago has started to reoccur, which is issue #28, where mounting fails with the error "Please insert a disk into Removable Disk (X:)". I tried upgrading to encfs4win RC10 and Dokany 1.0.1, but the error persists.

jetwhiz commented 7 years ago

Hi @wezeku -- is this error happening only when you mount early in the boot process, or at any time? Also, does it work after unmounting and re-mounting?

ephestione commented 6 years ago

Figured that I might add to here instead of creating a new Issue report: been having sudden unmounts on one of my two volumes. Of the two of them, one is a backup of my mobile phone's personal data, the other is the ensemble of files for my website projects, among which is my management web app, running via XAMPP; both are synced to dropbox, hence encfs is necessary.

The mobile data one stays strong, apparently, but the "htdocs" one fails constantly, I'd say maybe stays put for half an hour. The management web app is a php/mysql one (mysql hosted elsewhere, backups inside the encfs folder each hour), there are constant multiple reads every 3 seconds and occasional writes. Being that this is used in a production environment, I had to switch back to EncFSMP for now.

jetwhiz commented 6 years ago

Hi @ephestione -- how large are the files that are read every 3 seconds? Does the mobile phone volume have a lighter workload, or smaller files?

ephestione commented 6 years ago

Hi and thank you for the superfast support!

The web app files are php scripts, maybe in the tens of kilobytes ballpark, they might occasionally pull other files 4 to 20mb from a remote source into a subfolder of the volume, but the volume got dismounted several times before those files were created on the source, so nothing was pulled. Then each hour a dump is created, less than 1mb.

The other volume has less than 10mb files, usually 3-6, and now that I think about it, yesterday I didn't do any sync with the phone, so they were probably never accessed

ephestione commented 6 years ago

Oh by the way, the files are utf8 encoded on the xampp volume, and have pretty much "normal" filenames, nothing you wouldn't find on old 8.3 filesystems except the total filename length

jetwhiz commented 6 years ago

@ephestione could you check to make sure your Dokany version that is installed is 1.0.3 or higher? Our installer doesn't force the included version of Dokany to be installed, so if you had an older version already there then it's possible Dokany got left behind.

Also, do you know if this problem also exists in the encfs4win-legacy install as well?

ephestione commented 6 years ago

Hi @jetwhiz and thank you Yes, Dokan 1.0.3.1 is the one I have, I just re-checked for the record, no other versions installed either. Regarding old encfs4win, I have long uninstalled it when I switched to EncFSMP, and that happened because I had other stability problems with that encfs build, not unmounts, but rather access problems of some sort, one I clearly remember (maybe the only one, surely THE main reason I was looking for an alternative and finally switched) it was being unable to delete files, and back then I had already tried installing DokanY, which was at the 0.8 version, but that didn't really solve the problem

jetwhiz commented 6 years ago

Hi @ephestione -- can you try mounting encfs using the dokany "-o daemon_timeout=M" parameter (where M is a timeout in milliseconds). Play around with increasing the timeout and see if this alleviates the issue at all for you.

Also, I believe the delete file issue you mentioned should be fixed in the latest legacy encfs4win version, so in the worst case it might be worth trying that out (you'll need to get rid of Dokany v1.0 and let legacy encfs4win install Dokany v0.7.4).

ephestione commented 6 years ago

Hello and thank you for the reply, by increasing the timeout you expect that the sudden unmounts will be reduced? As I use my volumes in a productivity environment, where I am in front of my clients while I access the data inside, I wouldn't be able to keep it under test environment because that would impair my ability to do my job if the unmounts still happened, albeit less frequently. Alas, I am very much less a tinkering person than what I was back then... life :'(

Deleting files wasn't a problem with your version of encfs4win from what I noticed, even using latest DokanY, unmount are the bane of my work tho :D

jetwhiz commented 5 years ago

Hi @ephestione

Have you been able to try this with the latest encfs4win and Dokany? Lots of upstream Dokany changes seem to have made this much more stable than before.

ephestione commented 5 years ago

Hello @jetwhiz and really thank you for this update to the thread! Right about now I was updating some of the PHP code of said management app that runs off an encfs mount, and maybe I just could test again encfs4win+dokany instead of encfsmp, I will be switching now and will be back in here for feedback, hopefully soon

ephestione commented 5 years ago

Sooooo... no go. Installed the latest stable release, opened a page in chrome and left it running before going to dinner, lo and behold it had dismounted before I came back to the pc, still back to encfsmp for me

jetwhiz commented 5 years ago

It was worth a shot! The latest releases seemed much more stable than in the past, especially when adding and removing large amounts of files. Does it unmount only while idling, or also with usage?

Hopefully after the Dokany asynchronous IO and FindFiles improvements land in Dokany then we will be able to leverage this to improve the speed and reliability.

ephestione commented 5 years ago

In this occasion, and from what I remember since last time, it isn't really either of them, I mean I just left the browser with the webpage open, but even if I wasn't interacting with it there's a javascript refreshing a housekeeping PHP script that does several albeit small read/write operations so it never really goes idle.

ephestione commented 5 years ago

Anyway, I left the encfs4win mount commands commented out in the startup script, so if you ever want to keep this thread updated when these new milestones are reached I will know it's time to test again ;)

jetwhiz commented 5 years ago

I'll keep looking into this, as well as additional options for FUSE layers. Thanks @ephestione !

ephestione commented 5 years ago

So to add to the feedback: the partition that gets dismounted at random (experience says after about 30 minutes) is way more populated, we are talking 2.67GB, 28089 files in 3599 folders, where, every 3 seconds, a PHP script is loaded and does something, might be nothing in the end or maybe moving/copying/creating/deleting/executing files; while the other partition is practically sitting there idle, 158MB 349files in 37 folders, only being tapped in when I do a sync with the data on my smartphone, otherwise totally unused

ephestione commented 5 years ago

And rather thank YOU for creating this port!

jetwhiz commented 5 years ago

@ephestione thank you for the feedback! So both of the partitions are dismounting randomly, or only the larger 2.67GB one?

Also, can you confirm that Dokany upgraded successfully? If you look in Program Files / Dokany, you should see only version 1.2.1.2000 installed. If not, it might be good to completely uninstall Dokany, restart, and then install the latest Dokany (with encfs4win), since I believe Dokany cannot be upgraded well.

ephestione commented 5 years ago

@jetwhiz only larger partition gets dismounted, and DokanY 1.2.1 is the only folder in that path :) This is a new install of Windows 10 and I didn't have Dokan installed prior to my last attempt

Rondom commented 5 years ago

@ephestione With recent contributions from Google to improve the keepalive mechanism in Dokany there should not be any unmounts due to high load as far as I understand it. Can you share logs that could point us to why it gets unmounted?

ephestione commented 5 years ago

@Rondom could you please direct me on how to produce such logs? Please explain as if I'm clueless (rather than clueless I'm actually extremely lazy about making changes to things that are already working otherwise -in this case with EncFSMP- especially when I need them in a production process, as it's the case for the larger partition which XAMPP runs the server off)

ephestione commented 5 years ago

BTW I wouldn't consider it a "high load", it's just "not idle" but pretty much always doing something albeit small

jetwhiz commented 5 years ago

Hi @ephestione -- are you using the encfsw GUI or are you running this via the "encfs.exe" command line program? If using the command line, you can run: 

encfs.exe C:\path\to\folder D: -v -o -d

This will output logs for encfs as well as Dokany's logs (but there will be a lot of log data with the "-o -d" options for Dokany added). Logs are typically stored in %TEMP%.

Also, are all your files pretty much the same size, or are some many GB in size? Are there many files per folder (e.g., hundreds in a single folder)? How deeply nested are the folders generally (how many sub-folder in sub-folders in sub-folders, or \sub\sub\sub...\sub\ etc.). I assume you're using the default settings, or is this "paranoid" mode, or custom?

ephestione commented 5 years ago

Hey, so: 1) No huge files, at all, biggest ones are pictures RAWs up to 20MBs, shouldn't be anything bigger around 2) No folders with many many files, but there are a couple folders that have hundreds (should be >1000) of subfolders, each with several files inside (hardly up to 100, most are like 10-20) 3) no deep nesting, can't say for sure but less than 7 levels and those are mainly PHP libraries, those many subfolders reach up to 4 levels 4) Default settings

Will try now with encfs+logs, I have some time to spare

ephestione commented 5 years ago

encBB31.zip

Ok, this is the log that was accessible after the incriminated volume dismounted itself (I'd say about FIVE MINUTES after I mounted it, and it had barely had the time to read/write 100MB of data, and I'm being very generous)

ephestione commented 5 years ago

BTW couldn't find any other files pertinent to logs, so either everything is in that encBB31 file,or DokanY doesn't output logs to %TEMP% as no other files were created/modified at the same time

jetwhiz commented 5 years ago

That's strange, encfs and Dokany's FUSE should be able to handle a workload that size. It also looks like the logs start becoming corrupted right before the dismount occurs, but I don't see any relevant errors being reported -- I'll have to look over the log more to see if there's anything useful to go off of.

In the meantime, do you mind trying out the next release that's in the pipeline (which will be v1.11.0) and see if this changes anything? This is based off of the latest upstream software (with the same Dokany version as the v1.10.1 release candidates). You can grab the executables off of our build system: https://ci.appveyor.com/project/jetwhiz/encfs4win/builds/21916612/artifacts

And thank you for all of your help @ephestione!

ephestione commented 5 years ago

You're very welcome, thank you for supporting so much this software :) So I replaced the executables with the ones you linked, and rebooted disabling encfsmp lines while enabling encfs ones... nothing gets mounted at all, from the line where I activated the verbose options I get a wrong password error during the mounting phase, there must be something off with the -extpass option which I need to use from the batch script

jetwhiz commented 5 years ago

Hmm that's strange ... this feature should still be working in general unless you're using --extpass in a non-standard way. Our testing framework mounts the filesystem as:

encfs.exe --extpass="tests\retpass.bat" [options]

I don't see anything in the upstream changelog between 1.9.1 and 1.9.5 that would change this behavior, but maybe you've set up the command differently?

ephestione commented 5 years ago

As I don't like creating even more files around, I just use it as: --extpass="cmd /c echo password"

which obviously worked with the latest official stable

jetwhiz commented 5 years ago

You're right, it looks like there was a regression in the code that reads in the password (upstream changes overwrote our changes)!

This should be fixed in the latest version of the dev-195 branch. Could you try with the executables off of the build system with these updates when you get a chance? https://ci.appveyor.com/project/jetwhiz/encfs4win/builds/22215230/artifacts

ephestione commented 5 years ago

This time it did mount succesfully, and stayd mounted!

For less than a minute. Without any activity being done on the files except listing the root in File Explorer . 😝

Attached is the log encBBBE.zip

jetwhiz commented 5 years ago

@ephestione that's unfortunate ... I was hoping the latest encfs upstream would fix whatever issue was causing this. I don't see anything in either logs that seem odd (except the log looked like it got corrupted due to the program crash in the -RC14 log).

I'll see if I can build encfs4win using a different FUSE backend and see if this helps anything.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.