Dokan.Net performance

[mlacamera]

Hi, I am using the dokan.net version 2.0.4.1 wrapper with its dokan library. To test the performance I use a software (already used in production) that converts pdf with MagickNet (ImageMagick .NET wrapper) using Ghostscript. The software takes a set of pdf, reads them from the filesystem and transforms them into png (multiimage if the pdf has multiple images). To improve performance the software runs in production with multiple threads working on different pdf files. For example if there are 5 threads the software reads 5 files and processes them for each thread with different temporary directories (used by MagickNet). Now if I run the software on normal file system everything works correctly with 5 threads, if instead I use it on Dokan mirror file system the software works fine with only one thread but on multiple threads (on desired file / directory) some times it fails some reading operations (MagickNet gives me an error on read but Dokan.NET doesn't). Could it be a Dokan.NET performance problem?

[Liryna]

Hi @mlacamera , You could try with the native C samples to see whether it is a .NET issue or not. You could also use Procmon to see which MagickNet call is failing. It is probably not a performance issue but some race condition in the .NET code or a call incorrectly handled.

Are you running against your own filesystem that use Dokan.Net ?

[mlacamera]

Thanks @Liryna, I try with C samples and the result it is same, read error from MagicNet with 5 thread but no error with 1 thread. With ProcMon i can to see this when the software fails with 5 thread:

I running Dokan.Net e mirror.exe (c samples) in my filesystem

[Liryna]

Could you export the procmon logs and share them here ? How can we reproduce the issue ?

[mlacamera]

There he is Logfile.zip

[Liryna]

Looks like this is the read failing

There is so much request failing that I cannot even say what is working and expected. Without more context or how to cleanly reproduce it with the native sample, it is going to be difficulty to make a correct analysis.

[mlacamera]

Can you send a sample code?

[Liryna]

See https://github.com/dokan-dev/dokan-dotnet/issues/313#issuecomment-1158777018. The samples are package with the installer. You can find how to use them in the dokany repository wiki.

[mlacamera]

The log that I sended you is run with the native C sample of the #313 (comment). I have develop a dedicate console application that use only MagickNet with 5 thread on differnte files and I run the console application without mirror of the #313 (comment) and with mirror of the #313 (comment) (C Native Sample). In first case (without mirror) the console end correctly, in second case the console crash. I am attaching you teh procmon log of the second case I hoping it will be useful to you. Logfile_withDOKANFileSystemMirrorSample.zip

[Liryna]

Are you able to find out what function is failing in you app and maybe debug Magicknet? The second log stop on a read that succeeded.

[mlacamera]

This is proc log without mirror (first case) that end correctly dirctly on windows file system Logfile_withoutDOKANFileSystem.zip. Later I will send you stack trace and info on which MagickNet function fails

[mlacamera]

This is stack trace when I run with five thread

Unhandled Exception: System.Runtime.InteropServices.SEHException: External component has thrown an exception. at ImageMagick.MagickImageCollection.NativeMethods.X64.MagickImageCollection_ReadFile(IntPtr settings, IntPtr& exception) at ImageMagick.MagickImageCollection.NativeMagickImageCollection.ReadFile(IMagickSettings1 settings) at ImageMagick.MagickImageCollection.AddImages(String fileName, IMagickReadSettings1 readSettings, Boolean ping) at TestDokan.Program.InfoFile.ConvertPDF(String pdfFile, Double density, String tmpPath) in D:\Tecnoter\TFS_Cloud\OLSA\OLSAOVDfs\TestDokan\TestDokan\TestDokan\Program.cs:line 72 at TestDokan.Program.InfoFile.ConvertPDFAndSave(String pdfFile, Double density, String tmpPath) in D:\Tecnoter\TFS_Cloud\OLSA\OLSAOVDfs\TestDokan\TestDokan\TestDokan\Program.cs:line 40 at TestDokan.Program.<>c__DisplayClass4_0.

b__5(Object o) in D:\Tecnoter\TFS_Cloud\OLSA\OLSAOVDfs\TestDokan\TestDokan\TestDokan\Program.cs:line 136 at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart(Object obj)

If I run with one thread all works fine but if I run with two thread or more crash with the above error.

I remember you that software that use MagickNet component run in production environment with more five thread on windows file system without error

[LTRData]

It is a bit strange that a call into native ImageMagick library throws a native exception like that. I would debug this by either calling the same functions from a native application and catch the exception with a filter or try/except, or run it within a debugger with native debugging turned on. It seems to me that either that library is getting unexpected data back from Dokan, or it does something else internally that causes for example a null pointer dereference. It is difficult to tell without a native debugger and native stack trace.

[mlacamera]

@LTRData I can send you the solution Net Framework that use MagickNet and a example of files but, the strange thing, is that with one thread work fine, with 2 or more thread (they use different file e temporany directory) do not work, instead on Windows file system work fine with multi thread

[LTRData]

That happens sometimes. Threads introduce a complexity that can cause issues to only show up in multi thread scenarios.

Yes I could maybe take a look at what happens. Since the exception happens in native code, I need to run the managed solution with native libraries in debugger with native debugging turned on and preferably native symbol files available.

[mlacamera]

In this zip TestDokan.zip you found the code test on Visual Studio Solution, for MagickNet yuo remember to install GostScript (I use this https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs9561/gs9561w64.exe). In this zip RootExample.zip you found three directory that you must copy on Dokan File System Root.

[LTRData]

Just tried it and I do not see the exact same behavior but a bit similar. The crash you had within the ImageMagick libraries happened here too, it happened no matter if I used a Dokan file system or NTFS and no matter if I used one or several threads. But it only happened when I ran the process as 32 bit. If I unchecked "prefer 32 bit" in project settings and ran as a 64 bit process it started working without issues, for with Dokan, NTFS and with one or more threads.

I suggest that you also post this issue in Magick.NET repository. Maybe someone knows what could cause this and how to fix or workaround it!

[mlacamera]

@LTRData in test I ran the process as a 64bit also because the ImagickNet library must run at 64bit. The example that I send you is derived from a software that run in envirnoment production where works fine on NTFS. Now I don't know why with yuo works fine and in my machines no. I use mirror sample within the dokan solution.

[LTRData]

Okay, I just assumed that you had tried to run it as 32 bit because the project that you sent was set as "prefer 32 bit" in the project settings. I thought ImageMagick libraries worked in 32 bit too? At least they seem to have support for it. But in any case, when this problem happened it was a null pointer dereference in native parts of ImageMagick libraries and it happened in a read function before that read request was sent to kernel so it was difficult to know what needs to be done about it from kernel/file system perspective. It seems a bit complicated to build a debug build of ImageMagick libraries to debug it too, so I would think that it would be easier for someone familiar with that library to build a debug build and see why this crash happens.

A couple of things I notice though. Since I run the Dokan mirror sample in debugger too, I see some details about what happens there as well. After a while I get some exceptions at memory-mapped reads, for example "The process cannot access the file '...\RootExample\Tmp\tmp_2\magick-Mv_gi9paU2TVUGILnZLTggI3towiZkX01". This might or might not have something to do with the root problem. Maybe there could be something with exclusive opened files and memory-mapped I/O, I am not sure yet. Need to follow this closer. I can come back to it when I have more time!

[mlacamera]

Okay thanks for you support

[mlacamera]

@LTRData news?

[LTRData]

No sorry I have very limited time for the moment. But it is an interesting issue to investigate. I'll come back to it later!