aleksk / LazyCopy

NTFS minifilter driver that can download file content from a remote location, when it is opened for the first time.
https://git.io/voDg5
MIT License
103 stars 37 forks source link

Slow opening of files from UNC path. #2

Open scottweedon opened 5 years ago

scottweedon commented 5 years ago

When accessing placeholder files across a mapped drive or UNC share, the usermode service takes a long time to write to the target file.

This only occurs when accessing source items that are not local.

aleksk commented 5 years ago

Do you know what exactly is slow?

I assume you create reparse points using similar code: https://github.com/aleksk/LazyCopy/blob/a5334b0aeb7507e1fcc305fd9e3ac5a28fef9d7e/ToolsAndLibraries/SampleClient/Program.cs#L56 Driver can either ask c# service to open the file and then fetch it, or let c# to write content to the file (driver lets such I/O through). Unlike the GitVFS, this service doesn't pass the data back to driver for it to write.

Network path is not optimized there, but I would try:

  1. Make sure c# service reuses connections, where possible. It's specific to your usecase.
  2. Play with both modes: where driver downloads the file, and when service downloads it. https://github.com/aleksk/LazyCopy/blob/a5334b0aeb7507e1fcc305fd9e3ac5a28fef9d7e/Service/LazyCopySvc/LazyCopyDriver.cs#L188
  3. Change chunk size, 128KB might not be optimal in our environment: https://github.com/aleksk/LazyCopy/blob/master/Driver/LazyCopyDriver/Fetch.c#L182

    If you scenario is similar to the following: Build system produces bin output. You create reparse files on test machines and let them lazily download those files while they are executing tests.

This is always a cache-miss. In such cases I usually have an abstraction on top of it that prefetches (just file.open stubs so driver downloads) files that are most likely to be used by the app. You can use those ETW file access logs to build a recommender system. https://github.com/aleksk/LazyCopy/tree/master/ToolsAndLibraries/EventTracing