search

dokan-dev / dokany

User mode file system library for windows with FUSE Wrapper
http://dokan-dev.github.io
5.21k stars 661 forks source link

Dokan Mirror remote location issues #975

Closed workermaster closed 3 years ago

workermaster commented 3 years ago

Feature request can skip this form. Bug report must complete it. Check List must be 100% match or it will be automatically closed without further discussion. Please remove this line.

Environment

Check List

Description

Hi,

My apologies if this is not the right place for my issue. I downloaded the latest version of Dokan in order to map a network drive as local. The latest version is giving me errors while running Mirror.exe. It says: "Cant install drivers". Every version above 1.2.2.2000 does this. Version 1.2.2.2000 and under do not. However they cannot seem to mount the drive properly since I never see any confirmation of the drive being mounted. The drive is mounted and I can acces it but the CMD that I launch it from does not finish the mounting process.

Also when accessing the mounted drive I get an error that tells me there is a buffer issue. There is a buffer too small.

I am not a developer and do not know how to troubleshoot this problem. Any help would me much appreciated. I do not know how to attach logs.

Logs

Please attach in separate files: mirror output, library logs and kernel logs. In case of BSOD, please attach minidump or dump analyze output.

Liryna commented 3 years ago

Hi,

What is the package you are using to install the driver?

  1. Can you remove all Dokan install you have.
  2. Be sure to have no dokan1.sys in the windows driver folder.
  3. Install the latest versions with the .exe
  4. Try mirror.exe or memfs.exe
  5. If nothing is working, run dokanctl /I d and share the log
workermaster commented 3 years ago

Hi,

I have uninstalled everything and rebooted. Then I downloaded the latest version: 1.4.1.1000. I downloaded the DokanSetup.exe. Is that the correct one? Or do I need the DokanSetupDbg.exe?

Installed and rebooted. Running mirror.exe with the command "/r \nasname\sharename /l i" mounts the drive. Windows does see a drive that is 1GB large and 50% filled. The CMD that I use to start the mirror.exe: mirror

Normally I get the error "Cant install driver". No idea why that suddenly works. But as you can see the mounting does not finish. I get no confirmation of it being finished.

Where would dokan.sys be located? I can't find it in the Dokan folder.

Here is the output of the Cli tool. I do not see where it produces a log file. Could you tell me where te logfile is located? It does seem to say something about the dokan1.sys that you mentioned. Is that the one that should not be present? clitool

Hi,

What is the package you are using to install the driver?

  1. Can you remove all Dokan install you have.
  2. Be sure to have no dokan1.sys in the windows driver folder.
  3. Install the latest versions with the .exe
  4. Try mirror.exe or memfs.exe
  5. If nothing is working, run dokanctl /I d and share the log
Liryna commented 3 years ago

@workermaster Oh I see. Mirror goal is to mirror a local folder into a new location which the new location can be a network drive. So /r with a network path will not work.

There is another ticket open for adding this feature.

workermaster commented 3 years ago

@workermaster Oh I see. Mirror goal is to mirror a local folder into a new location which the new location can be a network drive. So /r with a network path will not work.

There is another ticket open for adding this feature.

Hi,

Thank you for your quick reply. I am trying to follow this thread on Reddit: https://www.reddit.com/r/DataHoarder/comments/bwm1j6/backblaze_with_nas_step_by_step_guide/ It seems that even the old Dokan (1.2xxxxx) can do that. The 1.4.1.1000 that I have installed now does that as well. It mounts a drive and Windows now sees a empty 4TB local drive. Wich is still a way off since I am trying to mount a 22TB share.

My main issue now is that the mounted drive keeps going away when my backup provider starts stressing the drive. The moment the provider starts creating a list of all the files (about 40.000 small files of >100kb) the CMD that I used to start the mirror.exe simply shows me a new line to enter a new command and the mounted drive is gone.

Could it be that the issue is that I am trying to mount a network share? I could map that share to Windows and then use mirror.exe to make it a local drive.

Could you please provide me the link of the ticket that has this as a feature request?

Liryna commented 3 years ago

@workermaster Oh my bad, this is working >< Wasn't able to see the screenshot until now.

So yes you can mirror the network drive and the size of the drive should even be the correct one. Mounting as local and network drive work.

You can get the mirror logs by adding /d /s to the command line. What I suspect is that the drive unmount because one of the access is taking too much time to complete on the NAS. There is a way to avoid this by changing the mirror to request mort time for long request but that's not implemented...

gittyupcowboy commented 3 years ago

This seems similar to what I went through when I found that Reddit thread a while ago. It might be related to #937 #511 #464 with the Win10 long file path issue.

I think I've solved my particular problem by following the steps in #464.

  1. Enabled Win 10 long file paths.
  2. Removed the comment on line 39 of mirror.c.
  3. Recompiled in VS2019.
  4. I had trouble with Backblaze not having write permissions (your mileage may vary; not responsible for any data corruption) so I've omitted that switch when I run it.
  5. Occasionally I'll have to press the resume backup on the Backblaze Control Panel to get it going again (minor inconvenience usually restarts itself eventually).

Interestingly enough I still get the occasional buffer error when I run the debug version so... I re-built the release version with the adjustments above, and wrote myself a quick and dirty batch file I run from an elevated command prompt. 

@REM Anytime mirror.exe crashes this will restart your file mount. 
@REM IF mirror location = c:\mirror\mirror.exe
@REM AND network PC = remotepc
@REM AND remote folder = remotefolder
@REM AND Drive letter = M
@REM THEN your batch file would be:

:start
C:\mirror\mirror.exe -r \\remotepc\remotefolder -l M
GOTO start

Buffer error in debug:

buffer error 

File: 
minkernel\crts\ucrt\inc\corecrt_internal_string_templates.h 
Line: 155 
Expression: (L Buffer is too small && 0)

Mirror.exe (release compile) will still crash from time to time however the batch file has by and large solved my problem, so I haven't dug into it much further... it should be noted my coding ability outside of Python is laughable (perhaps even in Python) so I have no idea what I'm doing. :)

I could also be wrong and yours could be an entirely different issue.

Perhaps if @Liryna or someone can can confirm that one line is the only change necessary, I'd be happy to drop the mirror.exe versions I compiled into a separate repo and provide a link (it seems many stumble upon this issue when finding that Reddit thread).

Liryna commented 3 years ago

-r and -l does not exist. Need to use /r or /l to pass the information.

Are you able to attach debugger to see what is the error ?

workermaster commented 3 years ago

@Liryna and @gittyupcowboy

I am currently backing up a lot of data to Backblaze. I will try your suggestions this evening. The buffer error picture that @gittyupcowboy shows is the same error that I used to get but that suddenly disappeard with the version that I am currently running.

Okay so I couldn't wait till the evening. Here is all the text that I copied from the CMD: Dokan_logs_1.txt

It crashed after Backblaze started to run through the small files.

Below is the output when I started the command. I just noticed that it was not present in the first file that I uploaded. Dokan_logs_2.txt

There is currently no program accessing the A drive. Yet Dokan is still showing a lot of activity and I get the feeling it is a loop. I stopped the mount and started the dokanctl.exe: image But that does not seem to work. Starting a mount and then running the command: image

Gittyupcowbo could you share your edited mirror.exe? I do not know how to compile it myself. I would also appreciate your batchfile that restarts Dokan after it crashes.

gittyupcowboy commented 3 years ago

Looking at your logs the longest file path I see doesn't appear long enough to trigger the issue I had, but give it a go.

I created this repo for you, follow the readme and you should be on your way.

workermaster commented 3 years ago

Looking at your logs the longest file path I see doesn't appear long enough to trigger the issue I had, but give it a go.

I created this repo for you, follow the readme and you should be on your way.

Did you create that repo just for me? Thanks!

A few hours ago I created a batchfile with the information that you posted yesterday. That file worked wonders. I was messing with the "goto" command but it would not loop. Yours does. The mirror.exe gets restarted after it crashes. Finally I can backup my files.

I just tried the mirror.exe that you put in the repo. Unfortunately that does seem to cause issues. When the process crashes it shows me a buffer error. The same one that you posted before.

This causes everything to wait for about 15 seconds and then it continues after restarting the .exe. Then it crashes again. It is faster for me to use the original .exe with the batchfile.

So everything seems to be working now but it is a bit strange that Dokan keeps crashings.

gittyupcowboy commented 3 years ago

Did you create that repo just for me? Thanks!

You're welcome, I figure it might help the next person who struggles after finding that Reddit post.

I just tried the mirror.exe that you put in the repo. Unfortunately that does seem to cause issues. When the process crashes it shows me a buffer error. The same one that you posted before.

I'm not sure what causes the troubles ... my few attempts to solve the riddle via event viewer/logs/debug/procmon haven't really provided any useful information. The batch file (at least in my experience) has been a functional work around.

Liryna commented 3 years ago

@gittyupcowboy Could it be a request that takes too much time and hit the limite of the timeout ? This triggers an unmount to protect the system.

gittyupcowboy commented 3 years ago

@gittyupcowboy Could it be a request that takes too much time and hit the limite of the timeout ? This triggers an unmount to protect the system.

My most recent trouble correlated with bzserv.exe (the application responsible for indexing the uploads to Backblaze I believe) hit this folder (which was mapped to S:\ via mirror):

\\network\folder\Music\disc.rips\The Decca Sound (2011)\CD 2 - Ataúlfo Argenta ~ España & Tchaikovsky\

mirror.exe would crash with a generic appcrash message. I could access the network share via UNC (\network\folder\etc.\etc.) in Explorer without issue and also inside of the mirror.exe mount I could access them and play the mp3s back without issue. The remote drive is in good health and shows no errors. However, each time I'd restart the backup mirror.exe would crash.

I could not determine anything different about the files/folder in question other than the tilde (~) where a hyphen was in the rest of my Decca Sound CD rips... so I replaced the tilde and renamed the folder:

CD 2 - Ataúlfo Argenta - España & Tchaikovsky\

and it stopped having issues... almost feels coincidental.

I vaguely remember using the /i (Timeout in Milliseconds ex. /i 30000) switch and still having some trouble back during my initial troubleshooting phase (early 2020) without much success (would that be the right switch for the timeout you're referencing?)

When I get some more time I'll run a ping script to see if it's a network issue, and monitor the system/drive on the remote host to see if it's an I/O issue there.

workermaster commented 3 years ago

@gittyupcowboy Could it be a request that takes too much time and hit the limite of the timeout ? This triggers an unmount to protect the system.

My most recent trouble correlated with bzserv.exe (the application responsible for indexing the uploads to Backblaze I believe) hit this folder (which was mapped to S:\ via mirror):

\\network\folder\Music\disc.rips\The Decca Sound (2011)\CD 2 - Ataúlfo Argenta ~ España & Tchaikovsky\

mirror.exe would crash with a generic appcrash message. I could access the network share via UNC (\network\folder\etc.\etc.) in Explorer without issue and also inside of the mirror.exe mount I could access them and play the mp3s back without issue. The remote drive is in good health and shows no errors. However, each time I'd restart the backup mirror.exe would crash.

I could not determine anything different about the files/folder in question other than the tilde (~) where a hyphen was in the rest of my Decca Sound CD rips... so I replaced the tilde and renamed the folder:

CD 2 - Ataúlfo Argenta - España & Tchaikovsky\

and it stopped having issues... almost feels coincidental.

I vaguely remember using the /i (Timeout in Milliseconds ex. /i 30000) switch and still having some trouble back during my initial troubleshooting phase (early 2020) without much success (would that be the right switch for the timeout you're referencing?)

When I get some more time I'll run a ping script to see if it's a network issue, and monitor the system/drive on the remote host to see if it's an I/O issue there.

I dont know if it helps but I am running Unraid. The Windows machine that runs the backup is a VM that runs on Unraid. In my case there is no "network" activity since Unraid handles all the IO. My guess is that there are some files that Backblaze can't backup. Everytime I restart the backup process it suddenly finds a couple extra files and tries to upload them. They are the same files every time.

Anyway my backup is running and will take over 3 months.

shackrock commented 3 years ago

Just noting I have been having issues w/ random crashes as well, going to be using this batch file to see if it solves my Dokan issues!

gittyupcowboy commented 3 years ago

Currently, mirror.exe is crashing when accessing this file:

High Resolution Date & Time: 3/20/2021 2:00:05.4735271 PM Event Class: File System Operation: CreateFile Result: CANCELLED Path: M:\Music\cd.rips\The Decca Sound (2011)\CD 12 - Kyung Wha Chung - Mendelssohn & Bruch\Mendelssohn, Bruch - Violin Concertos (Kyung Wha Chung, Charles Dutoit, Orchestre symphonique de Montréal, Rudolf Kempe, Royal Philharmonic Orchestra).cue TID: 19792 Duration: 0.8899404 Desired Access: Read Attributes Disposition: Open Options: Open Reparse Point Attributes: n/a ShareMode: Read, Write, Delete AllocationSize: n/a

The next line that usually appears in procmon is werfault.exe spawning. I can get to the file via UNC (\remote\folder) without issues... I went ahead and tried renaming it, this has been an adequate work around in the past... it just takes a while for Backblaze to stop trying to call on that file.

path length via mirror = 240 characters path length via UNC = 253 characters remote Unraid box responds to ping, there's no IO issues on the box at the time... and I can navigate the folder structure via the UNC mount on the local host.

Perhaps too many commas?

A few logs attached. mirror_procmon.log mirror_debugoutput.log

Liryna commented 3 years ago

Thanks for sharing. Running a debut version with the debugger attached would really be helpful to know where it is crashing. Is that something you could do ?

gittyupcowboy commented 3 years ago

Thanks for sharing. Running a debut version with the debugger attached would really be helpful to know where it is crashing. Is that something you could do ?

I should be able to get to that sometime this week. I've got to clone a near blank slate Windows 10 and spin up a trial license for Backblaze - I tried it on the machine with the troubles, but enabling kernel debugging caused some stability issues with other applications.

Liryna commented 3 years ago

@gittyupcowboy It looks like it is the mirror or the library that crash so just debugging from visual studio should do it. Kernel debugging is only needed for the driver which a crash would create a BSOD

gittyupcowboy commented 3 years ago

@Liryna, that makes sense. Today I learned how to dubug *.exe files in Visual Studio.

@gittyupcowboy It looks like it is the mirror or the library that crash so just debugging from visual studio should do it. Kernel debugging is only needed for the driver which a crash would create a BSOD

Debug Output: 

Debug Assertion Failed!

Program: C:\mirror\debug\Win32\mirror.exe
File: minkernel\crts\ucrt\inc\corecrt_internal_string_templates.h
Line: 155

Expression: (L"Buffer is too small" && 0)

For information on how your program can cause an assertion
failure, see the Visual C++ documentation on asserts.

(Press Retry to debug the application)
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\msctf.dll'. 
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\TextInputFramework.dll'. 
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\CoreUIComponents.dll'. 
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\CoreMessaging.dll'. 
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\ntmarta.dll'. 
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\WinTypes.dll'. 
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\WinTypes.dll'. 
'mirror.exe' (Win32): Unloaded 'C:\Windows\SysWOW64\WinTypes.dll'
'mirror.exe' (Win32): Loaded 'C:\Windows\SysWOW64\iertutil.dll'. 
mirror.exe has triggered a breakpoint.

(retried after the fail; debug continued)

Events: 

Event Type Moniker  Event   Time    Duration    Thread
Current Code Context    mirror.c line 92 An exception was caught by the debugger, and user settings indicate that a break should occur. Hyperlink: Go to Source Code    46.36s  46,362ms    [20996]

https://github.com/dokan-dev/dokany/blob/de4c04c449c046b103b1f64b6e3a554b26470020/samples/dokan_mirror/mirror.c#L92

Locals: 

        _Expr_val   0   int
        available   0   unsigned int
        count   243 const unsigned int
 -      destination 0x0327f460 L""  wchar_t * const
            0 '\0'  wchar_t
 -      destination_it  0x0327f668 L"쳌쳌⦃蕉̧Ფ篦뱘ņ̧€" wchar_t *
            52428 '쳌'   wchar_t
        remaining   2   unsigned int
        size_in_elements    260 const unsigned int
 -      source  0x0146bc58 L"\\Music\\cd.rips\\mp3\\The Decca Sound (2011) [MP3 V0]\\CD 26 - Alicia de Larrocha - Granados & De Falla\\Granados - Goyescas, De Falla - Noches en los jardines de España (Alicia de Larrocha, Rafael Frühbeck de...  const wchar_t * const
            92 '\\' const wchar_t
 -      source_it   0x0146be3c L"e" const wchar_t *
            101 'e' const wchar_t

Remote file & path referenced = 262 chars: 

\\remote\folder\Music\cd.rips\mp3\The Decca Sound (2011) [MP3 V0]\CD 26 - Alicia de Larrocha - Granados & De Falla\Granados - Goyescas, De Falla - Noches en los jardines de España (Alicia de Larrocha, Rafael Frühbeck de Burgos, London Philharmonic Orchestra).cue

Call stack: 

>   mirror.exe!common_tcsncat_s<wchar_t>(wchar_t * const destination, const unsigned int size_in_elements, const wchar_t * const source, const unsigned int count) Line 155 C++
    mirror.exe!wcsncat_s(wchar_t * destination, unsigned int size_in_elements, const wchar_t * source, unsigned int count) Line 19  C++
    mirror.exe!GetFilePath(wchar_t * filePath, unsigned long numberOfElements, const wchar_t * FileName) Line 92    C
    mirror.exe!MirrorCreateFile(const wchar_t * FileName, _DOKAN_IO_SECURITY_CONTEXT * SecurityContext, unsigned long DesiredAccess, unsigned long FileAttributes, unsigned long ShareAccess, unsigned long CreateDisposition, unsigned long CreateOptions, _DOKAN_FILE_INFO * DokanFileInfo) Line 224  C
    [External Code] 
    dokan1.dll![Frames below may be incorrect and/or missing, no symbols loaded for dokan1.dll] Unknown

Is it still the file path issue... meaning, did I not perform this correctly?

https://github.com/dokan-dev/dokany/blob/de4c04c449c046b103b1f64b6e3a554b26470020/samples/dokan_mirror/mirror.c#L36-L42

vs. 

//#define WIN10_ENABLE_LONG_PATH
#ifdef WIN10_ENABLE_LONG_PATH
//dirty but should be enough
define DOKAN_MAX_PATH 32768
#else
#define DOKAN_MAX_PATH MAX_PATH
#endif // DEBUG

See previous comment for further details: https://github.com/dokan-dev/dokany/issues/975#issuecomment-799873534

In testing the above was the only way I could get it to compile (again this is not a language I'm even mildly proficient at).

Liryna commented 3 years ago

Oh it is this line that need to be uncomment https://github.com/dokan-dev/dokany/blob/de4c04c449c046b103b1f64b6e3a554b26470020/samples/dokan_mirror/mirror.c#L36 Like that the large size will be used.

Liryna commented 3 years ago

Made an improvement to the doc for this flag https://github.com/dokan-dev/dokany/commit/b4f1be642dce13058533fb231c6c8b750a4494ec

workermaster commented 3 years ago

Oh it is this line that need to be uncomment

https://github.com/dokan-dev/dokany/blob/de4c04c449c046b103b1f64b6e3a554b26470020/samples/dokan_mirror/mirror.c#L36

Like that the large size will be used.

Hi,

If that is the solution could you or @gittyupcowboy give me the updated mirro.exe so I can test it?

gittyupcowboy commented 3 years ago

Made an improvement to the doc for this flag b4f1be6

Thanks @Liryna, went full Python there. I should have perhaps learned how commenting in C works first? The blind leading the blind over here. 🤣

Oh it is this line that need to be uncomment https://github.com/dokan-dev/dokany/blob/de4c04c449c046b103b1f64b6e3a554b26470020/samples/dokan_mirror/mirror.c#L36

Like that the large size will be used.

Hi,

If that is the solution could you or @gittyupcowboy give me the updated mirro.exe so I can test it?

@workermaster try this version out and let me know what you encounter. I've found a strange issue that I can't get to the bottom of.

I have some ~84 files that queue up for transfer, however when Backblaze tries to hit them Procmon shows the result as "File Not Found" (see files with issues.log for the list of files).

When I attempt to access these files via the Mirror drive (M:) in Explorer.exe, Explorer hangs and takes a while to load (near a minute?). However when I browse the UNC (\remote\file network path) or the mapped network drive (Z:), I have no issues accessing, playing or loading the files in question.

I was thinking it might have something to do with file names, but some of the files in the folder have no issues once the Mirror.exe drive (M:) loads, however other files are inaccessible and the Mirror drive (M:) is unable to even display the file properties (again file properties load via UNC/Mounted network share) and they share very similar naming conventions and characters.

For example:

when accessing the Mirror drive (M:) via Explorer.exe the folder containing the following file(s) causes Explorer.exe to hang:

M:\music\cd.rips\111 Years Of Deutsche Grammophon - Collector's Edition\01 - Abbado - Bizet - Carmen - Part 1\17 - Act 1 - 'Voyons, brigadier...' - 'Tra la la la...' (Zuniga, Don José, Carmen - Carmen, Zuniga, Les Cigarières, Un Soldat, Don J.mp3

Additionally tracks 13, 16, 23, 24, 26 have the same issue(s). However, Explorer.exe eventually loads the contents of the folder and the remaining tracks (e.g. 1-12, etc.) are playable.

As far as logs go, Windows Event Logs just show a generic APPHANG for Explore.exe. The debug output inside of Visual Studio shows no exceptions, and the debug output in the console output (/d /s) has pretty standard output regarding accessing files (nothing of note).

I've added every conceivable exception to installed AV solutions, the folder containing Mirror.exe, the network share, the mounted drive (M:), so on and so forth. I've tried with said AV solutions disabled... all yield the same result (and again files are fine via UNC & mapped network drive). Testing indicates it's not a network related issue, but somewhere on the application layer (Windows and above).

I've tried 1 thread (/t) multiple threads, I've changed the timeout (/i) to be minutes etc... no change in results.

I'm running some tests on the Unraid Box (SMART, etc.), and eventually I'll compare file properties of those that load and those that don't. 🤷

My work around was to add those 84 files to a zip file (which will extract them to their full file path of the mounted network share (Z:)) and drop it in the root of that drive (Z:\backblaze_why.zip) which was backed up without issue. 🤣

workermaster commented 3 years ago

Made an improvement to the doc for this flag b4f1be6

Thanks @Liryna, went full Python there. I should have perhaps learned how commenting in C works first? The blind leading the blind over here. 🤣

Oh it is this line that need to be uncomment https://github.com/dokan-dev/dokany/blob/de4c04c449c046b103b1f64b6e3a554b26470020/samples/dokan_mirror/mirror.c#L36

Like that the large size will be used.

Hi, If that is the solution could you or @gittyupcowboy give me the updated mirro.exe so I can test it?

@workermaster try this version out and let me know what you encounter. I've found a strange issue that I can't get to the bottom of.

I have some ~84 files that queue up for transfer, however when Backblaze tries to hit them Procmon shows the result as "File Not Found" (see files with issues.log for the list of files).

When I attempt to access these files via the Mirror drive (M:) in Explorer.exe, Explorer hangs and takes a while to load (near a minute?). However when I browse the UNC (\remote\file network path) or the mapped network drive (Z:), I have no issues accessing, playing or loading the files in question.

I was thinking it might have something to do with file names, but some of the files in the folder have no issues once the Mirror.exe drive (M:) loads, however other files are inaccessible and the Mirror drive (M:) is unable to even display the file properties (again file properties load via UNC/Mounted network share) and they share very similar naming conventions and characters.

For example:

when accessing the Mirror drive (M:) via Explorer.exe the folder containing the following file(s) causes Explorer.exe to hang:

M:\music\cd.rips\111 Years Of Deutsche Grammophon - Collector's Edition\01 - Abbado - Bizet - Carmen - Part 1\17 - Act 1 - 'Voyons, brigadier...' - 'Tra la la la...' (Zuniga, Don José, Carmen - Carmen, Zuniga, Les Cigarières, Un Soldat, Don J.mp3

Additionally tracks 13, 16, 23, 24, 26 have the same issue(s). However, Explorer.exe eventually loads the contents of the folder and the remaining tracks (e.g. 1-12, etc.) are playable.

As far as logs go, Windows Event Logs just show a generic APPHANG for Explore.exe. The debug output inside of Visual Studio shows no exceptions, and the debug output in the console output (/d /s) has pretty standard output regarding accessing files (nothing of note).

I've added every conceivable exception to installed AV solutions, the folder containing Mirror.exe, the network share, the mounted drive (M:), so on and so forth. I've tried with said AV solutions disabled... all yield the same result (and again files are fine via UNC & mapped network drive). Testing indicates it's not a network related issue, but somewhere on the application layer (Windows and above).

I've tried 1 thread (/t) multiple threads, I've changed the timeout (/i) to be minutes etc... no change in results.

I'm running some tests on the Unraid Box (SMART, etc.), and eventually I'll compare file properties of those that load and those that don't. 🤷

My work around was to add those 84 files to a zip file (which will extract them to their full file path of the mounted network share (Z:)) and drop it in the root of that drive (Z:\backblaze_why.zip) which was backed up without issue. 🤣

Hi,

Thanks for the new version! I have downloaded the X64 release version. Backblaze started to backup about 1000 small files that it previousely would not backup. Mirror.exe is not crashing anymore. When I switched back to the previous mirror.exe version the problems returned. It seems that the new version that you shared is working perfectly.

I did notice something strange though. When using the old version of mirror.exe there were 2 files that would cause it to crash the moment Backblaze tried to acces them: UnknownFile_2 UnknownFile The strange thing is that I have never seen these files before. I do not know where they came from and cannot find them with the search function in Windows Explorer.exe. The mirror.exe is only mounting a media share so these files should not be there. But that is not a problem with mirror.exe.

Anyway thanks for the updated version. I am using it now together with the batchfile that restarts it when it crashes.

gittyupcowboy commented 3 years ago

Hi,

Thanks for the new version! I have downloaded the X64 release version. Backblaze started to backup about 1000 small files that it previousely would not backup. Mirror.exe is not crashing anymore. When I switched back to the previous mirror.exe version the problems returned. It seems that the new version that you shared is working perfectly.

I did notice something strange though. When using the old version of mirror.exe there were 2 files that would cause it to crash the moment Backblaze tried to acces them: UnknownFile_2 UnknownFile The strange thing is that I have never seen these files before. I do not know where they came from and cannot find them with the search function in Windows Explorer.exe. The mirror.exe is only mounting a media share so these files should not be there. But that is not a problem with mirror.exe.

Anyway thanks for the updated version. I am using it now together with the batchfile that restarts it when it crashes.

Those are associated with various Microsoft Packages (default apps, Windows Store related items, etc.), you may want to exclude

%LOCALAPPDATA%\Packages\

gittyupcowboy commented 3 years ago

when accessing the Mirror drive (M:) via Explorer.exe the folder containing the following file(s) causes Explorer.exe to hang:

M:\music\cd.rips\111 Years Of Deutsche Grammophon - Collector's Edition\01 - Abbado - Bizet - Carmen - Part 1\17 - Act 1 - 'Voyons, brigadier...' - 'Tra la la la...' (Zuniga, Don José, Carmen - Carmen, Zuniga, Les Cigarières, Un Soldat, Don J.mp3

I solved my riddle... the file length of the UNC path is 260+ chars (\\remote\folder...); causes Mirror.exe to hang (no longer crash).

Liryna commented 3 years ago

Glad that the mirror no longer crash. Is there anything we need to look at for this issue?

gittyupcowboy commented 3 years ago

Glad that the mirror no longer crash. Is there anything we need to look at for this issue?

Just need to figure out why the long file paths mounted via mirror are taking so long to load that it returns a file not found for other applications.

I had already enabled long file paths in Windows, but I can't really seem to find anything else to point me in the right direction.

Liryna commented 3 years ago

@gittyupcowboy Would need to take a look at the mirror logs see if you get the path that has issues and if it processed correctly. Also do not hesitate to use procmon to spy the IO activity.

workermaster commented 3 years ago

@Liryna and @gittyupcowboy,

I have found a new issue but I am not sure if it is caused by the mirror.exe. I want the batchfile that starts mirror.exe to run when Windows boots and every hour after that (to restart it if it crashes). I created a task in the Windows task sheduler to run the batch file when Windows starts.

When I start the batch file manually I get this: Dokan_mapping_ok Drive A is mapped. It reports the wrong size since it is a mapping of the Plex drive Z but it works.

When I enable the task in the sheduler and reboot the VM it does sort of start but I get this: Dokan_mapping_nok Drive A is created but does not work.

Liryna commented 3 years ago

@workermaster Looks like the mirror is not having access to the Plex drive at early stage ? Can try to add some sleep or something to be sure Plex is actually available to see if it helps. Also does the task run with your user permission?

workermaster commented 3 years ago

@workermaster Looks like the mirror is not having access to the Plex drive at early stage ? Can try to add some sleep or something to be sure Plex is actually available to see if it helps. Also does the task run with your user permission?

I tried setting a delay of one minute and 5 minutes. It did not solve the problem. The Backup VM has no password. These are the settings I use: DokanAutoStart

Even manually starting the scheduled task after reboot does not solve the issue. I do not understand why this happens.

EDIT: I managed to get it working in a different way. I placed the batch file in the Windows Startup folder. That seems to work. Though I have no idea why since I tried that before and it didn't work then.

Liryna commented 3 years ago

@workermaster Try with the do not store password option unchecked see if that helps.

workermaster commented 3 years ago

@workermaster Try with the do not store password option unchecked see if that helps.

Hi,

That does not work. When I uncheck that option and click on "Ok" then Windows asks for a password. The VM does not have a password. I can't save an empty password field: Dokanpassword Dokanpassword2

gittyupcowboy commented 3 years ago

@workermaster It's been a long time, but I believe there's a group policy setting for allowing blank passwords somewhere (depending on if you're on home you may not have access).

The easiest solution might be to set a password. If the VM's sole purpose is to back things up and you want windows to auto login to that account at startup, you could try creating a second user account with a password as a workaround.

In my testing using the .bat file via the startup folder (C:\Users\\AppData\Roaming\Microsoft\Windows\Start Menu\Programs\Startup) worked out fine because the cmd console never crashed itself, so it would re-map the drive when the mirror.exe process ended... another viable alternative.

Personally as someone that works in infosec, I find things without passwords bad. That restriction Windows has in place for blank passwords is to protect the system from the operator (i.e. the user). However assuming you have access to that machine locked down pretty good (e.g. RDP only accessible from internal hosts; or a specific host) and you just want the backup to be as automagic as possible, I get it.

@gittyupcowboy Would need to take a look at the mirror logs see if you get the path that has issues and if it processed correctly. Also do not hesitate to use procmon to spy the IO activity.

@Liryna I've attached a small sample from procmon output filtered on a specific file, I failed to run mirror in debug so I do not have the associated logs from the time (I'll have more time this weekend to get more logs).

pathnotfound.log

Again, the mirror.exe mount takes forever to load the files in a folder with a long path name, but it's accessible just fine via UNC. 🤷

marcosscriven commented 3 years ago

Just searched the issues for "mirror buffer" and found this one. I too am trying to map a network drive to a local drive, and although it seems to work mostly, it eventually fails.

What I see in the terminal is:

###GetFileSecurity 0065
GetFileSecurity \\nas.home\media\.bzvol\bzvol_id.xml
        FILE_SHARE_READ
        OWNER_SECURITY_INFORMATION
        GROUP_SECURITY_INFORMATION
        DACL_SECURITY_INFORMATION
        SACL_SECURITY_INFORMATION
  Opening new handle with READ_CONTROL access
  GetUserObjectSecurity error: ERROR_INSUFFICIENT_BUFFER
###GetFileInfo 0065
GetFileInfo : \\nas.home\media\.bzvol\bzvol_id.xml
        GetFileInformationByHandle success, file size = 256
FILE ATTRIBUTE  = 32
        result =  0
        FileBasicInformation
        DispatchQueryInformation result =  0
###QueryVolumeInfo 0065
GetVolumeInformation: unable to query underlying fs, using defaults.  Last error = 123
###GetFileSecurity 0065
GetFileSecurity \\nas.home\media\.bzvol\bzvol_id.xml
        FILE_SHARE_READ
        OWNER_SECURITY_INFORMATION
        GROUP_SECURITY_INFORMATION
        DACL_SECURITY_INFORMATION
        SACL_SECURITY_INFORMATION
  Opening new handle with READ_CONTROL access
  GetUserObjectSecurity return true,  *LengthNeeded = securityDescriptorLength
###GetFileInfo 0065
GetFileInfo : \\nas.home\media\.bzvol\bzvol_id.xml
        GetFileInformationByHandle success, file size = 256
FILE ATTRIBUTE  = 32
        result =  0
        FileStandardInformation
        DispatchQueryInformation result =  0
###Read 0065
ReadFile : \\nas.home\media\.bzvol\bzvol_id.xml
        Byte to read: 4096, Byte read 256, offset 0

Then in event viewer I see this info/error:

Level   Date and Time   Source  Event ID        Task Category
Information     04/05/2021 07:44:48     Windows Error Reporting 1001    None    "Fault bucket 1169492488918548169, type 5
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: mirror.exe
P2: 0.0.0.0
P3: 60001598
P4: mirror.exe
P5: 0.0.0.0
P6: 60001598
P7: 000000000000c408
P8: c0000409
P9: 0000000000000005
P10:

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER4039.tmp.dmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER4059.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER406A.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER4068.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER4079.tmp.txt

These files may be available here:
\\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_mirror.exe_d913b1ed7bd5276d731380f5f3e20be65be9f_05f72e11_6fe00092-a3c3-44cb-943a-c8633d1a438e

Analysis symbol:
Rechecking for solution: 0
Report Id: 3123222a-82c4-46fc-9346-b34c969068a9
Report Status: 268435456
Hashed bucket: 45ff0aa4f0186a2d403adf38f17a3ec9
Cab Guid: 0"
Error   04/05/2021 07:44:48     Application Error       1000    (100)   "Faulting application name: mirror.exe, version: 0.0.0.0, time stamp: 0x60001598
Faulting module name: mirror.exe, version: 0.0.0.0, time stamp: 0x60001598
Exception code: 0xc0000409
Fault offset: 0x000000000000c408
Faulting process ID: 0xfd0
Faulting application start time: 0x01d740b0f496f177
Faulting application path: C:\Program Files\Dokan\Dokan Library-1.4.1\sample\mirror\mirror.exe
Faulting module path: C:\Program Files\Dokan\Dokan Library-1.4.1\sample\mirror\mirror.exe
Report ID: 3123222a-82c4-46fc-9346-b34c969068a9
Faulting package full name:
Faulting package-relative application ID: "

Any ideas what's happening here please, or any other info I can give? I see that buffer size is mentioned in this issue, but there doesn't seem to be a solution/workaround.

Liryna commented 3 years ago

@marcosscriven what makes you think the long path define enable is not the solution? Have you tried to enable it and rebuild the mirror ?

marcosscriven commented 3 years ago

Thanks for your response @Liryna

I assumed it wouldn't make a difference, because the file I'm seeing in the log is \\nas.home\media\.bzvol\bzvol_id.xml, which I don't think is anywhere near the limit?

I can try it if you think it would help.

Liryna commented 3 years ago

It would be great if you could try yes. If this is not the issue, will you be able to attach a debugger and find the source of the crash ?

marcosscriven commented 3 years ago

@Liryna - It'll take me some time to get everything in https://github.com/dokan-dev/dokany/wiki/Build setup (as I don't usually develop in Windows).

I don't know if you happen to have a CI setup for this, or a build with this change already in by any chance please?

EDIT - Sorry, I see you linked it already above https://github.com/gittyupcowboy/dokany_mirror_win10path/releases/tag/v0.2-alpha

Liryna commented 3 years ago

We have a CI but too much changes are at head that are not compatible with the latest driver unfortunately.

Besides that you will need the same env to reproduce and debug the crash.

marcosscriven commented 3 years ago

@Liryna - that seems to have fixed the issue. Thanks!

marcosscriven commented 3 years ago

@Liryna - Just looking at https://github.com/dokan-dev/dokany/discussions/982 regarding the 4TB max issue, you recommend building another commit https://github.com/dokan-dev/dokany/discussions/982#discussioncomment-566632

If I want both the long path and >4TB setup, does this mean I do indeed need to setup a dev environment?

Liryna commented 3 years ago

@marcosscriven Yes that's right. You will need a dev env for now.

The 4TB fix will be in the next release but the long path change not. For the last one it would be great to have a command line option to enable it. If anyone is interested to add it !

Liryna commented 3 years ago

I will probably close this since the main issue was fixed and the long path limited in mirror is already logged in https://github.com/dokan-dev/dokany/issues/511

Liryna commented 3 years ago

I made the long path option enabled by default for now until we get a runtime option. https://github.com/dokan-dev/dokany/commit/e9e331b9fcffbf0ea76f38721ff34ad62fb7292b

marcosscriven commented 3 years ago

I made the long path option enabled by default for now until we get a runtime option. e9e331b

Ahh, thanks so much for your efforts @Liryna - I'd love to be able to contribute, but the whole Windows/Visual Studo dev stack is not something I've dealt with at all, and would probably take me an age to do something simple.

mianghuei commented 3 years ago

This issue does not reoccur after upgrading to 1.5.0.2000. Thanks @Liryna !

workermaster commented 3 years ago

Hi @Liryna

I wanted to report that the fix is working perfectly. However I have run into a new problem. My virtual machine crashed and when trying to setup a new one with Dokan, I get the "Can't install driver" error again. Last time it went away on its own but this time it wont.

I have tried the most recent version and the 1.4 versions I had working last time. Do you know why I get the driver error after a fresh Windows install?

Kind regards, Workermaster.