search

cryptomator / dokany-nio-adapter

Dokany-based adapter to provide directory contents specified by a java.nio.file.Path (via dokan-java)
GNU Affero General Public License v3.0
14 stars 4 forks source link

Stability: Crash of dokany mounted volume #21

Closed infeo closed 5 years ago

infeo commented 5 years ago

dokany-nio-adapter: 8a977f4de9e9b44acee01712c01ca1b057d27ae4 OS: Windows 10 Enterprise N Ver. 1809 Dokany: 1.2.1.2000 Related issue: cryptomator/cryptomator#761, cryptomator/cryptomator#791

Description

There are several reports about a crashes of the provided volume (not responding, no access, etc) when there is high load put onto the volume it.

Steps To Reproduce

TODO

Logs

TODO

whitephoenix117 commented 5 years ago

I have been able to reliability reproduce this error using windirstat on a cryptomator mounted volume. See details in cryptomator/cryptomator#791.

Is there a way I could help replicate this just with Dokany to help with troubleshooting?

infeo commented 5 years ago

Thanks for the offering help! Actually you can do a little testing yourself with the pure dokany-nio-adapter.

There is a directory mirror example, which you can start and then try to push it to its limit. To run it i created a manual in a fork of this repository. (Didn't know where else i should put this currently)

What i tried so far over the mounted directory at the same time is the following:

The mirrored directory contained of a big git repository (nextcloud server), a lot of image files (including some high-res nasa shots) and of course the 4K movie (round about 6GB). The hardware setup was an intel nuc (TODO: add specs).

Oh, and of course the dokany volume didn't crash.

whitephoenix117 commented 5 years ago

Having never compiled or run anything from Java before that took a bit of trial and error. Especially getting from JRE 8 --> JDK 11.

I got it mounted and I set windirstat scanning a 17TB directory... I will post an update with the results when it finishes; but at the time of posting its already doing much better that the cryptomator FS. So far it completed 8% (8k files) Typically when scanning encrypted FS is crashes ~1k

The hardware I' testing this on is OLD.

Intel i7-2820QM 8GB RAM

The 17TB source FS is a Ubuntu server mounted in windows.

whitephoenix117 commented 5 years ago

Having never compiled or run anything from Java before that took a bit of trial and error. Especially getting from JRE 8 --> JDK 11.

I got it mounted and I set windirstat scanning a 17TB directory... I will post an update with the results when it finishes; but at the time of posting its already doing much better that the cryptomator FS. So far it completed 8% (8k files) Typically when scanning encrypted FS is crashes ~1k

The hardware I' testing this on is OLD.

Intel i7-2820QM 8GB RAM

The 17TB source FS is a Ubuntu server mounted in windows.

It took 5 hours but it got everything and didn't crash.

image

infeo commented 5 years ago

Dang it. Or, I mean, great!^^ Maybe the fault is not at the dokany-nio-adapter but elsewhere or in combination.

Nonetheless, i keep this issue open until the bug is found.

whitephoenix117 commented 5 years ago

Dang it. Or, I mean, great!^^ Maybe the fault is not at the dokany-nio-adapter but elsewhere or in combination.

Nonetheless, i keep this issue open until the bug is found.

I used the same mirror app on the still encrypted google drive stream FS. Was ~3TB and also ran fine.

whitephoenix117 commented 5 years ago

I broke the golden rule of troubleshooting! Only change 1 thing at a time...

The instability I was experiencing with the vault was on my older windows PC which I prefer not to use too much (its hot/ heavy). When I sat down to do this test I used my more ergonomic ultrabook; both run windows 10 however I failed to realize they have different versions of Dokan installed. I expect this is why the mount did not crash.

The (Passing) test above was done using Dokan 1..0.2 Instability of the vault was observed on a 2nd PC which was using Dokan 1.2.2.1000

After more testing I was able to make the 2nd PC stable by downgrading from 1.2.2.1000 --> 1.0.2. using the same PC I was able to upgrade to 1.2.2.1000 which resulted in the vault crashing

TLDR

Dokan 1.0.2 makes the vault stable Dokan 1.2.2.1000 makes the vault unstable

infeo commented 5 years ago

Gotcha! Found a severe memory leak of not removing already closed files. Fixed with 4d4fa3ceaddbb2a09aa438c226c4a17d8146a68f