Limiting memory consumption (Feature request)

[GoogleCodeExporter]

Since Dokan in nature is a Virtual File System (VFS) and thus resides in 
memory, it would be desirable to force a limit on the amount of memory which is 
consumed.

Currently, reading and writing to/from a file is potentially very memory 
consuming due to buffers being allocated. This can obviously be resolved 
programmatically by chopping up large reads and writes in the applications 
using your VFS. Lazy programmers (like me) don't like to write this kind of 
trivial code.

Let me illustrate the issue with an example;
Let's say I have 500 MB of data I want to write to a file in my Dokan-powered 
VFS. I do this by calling the .NET method "File.WriteAllBytes(filename, data)". 
The VFS will then allocate a 500*1024*1024 array of bytes and then perform the 
write-command. 

Now imaging this scenario taking place on a netbook with only 1GB RAM. *ouch*

But ... I gets even worse with the .NET binding, since this relies on the .NET 
Garbage Collector, which is ... erhm ... random at best. OutOfMemory exceptions 
occur frequently (handling large files and/or stress testing) unless one 
forcefully calls the garbage collector regularly.

So, to sum up: some options for maximum read and write buffer sizes would be an 
awesome addition. 

P.S. FUSE has these options, but they are upper-bound by 32 pages of 4096 bytes 
- which I think is too low.

Original issue reported on code.google.com by tokeboy.riis on 8 Aug 2010 at 7:43

[GoogleCodeExporter]

Err, 500MB buffers? What about the good old 32-bit address space? Very risky.

Original comment by gabest11 on 13 Aug 2010 at 3:06

[GoogleCodeExporter]

@tokeboy.riis--"So, to sum up: some options for maximum read and write buffer 
sizes would be an awesome addition."

How to exactly do that? Would you pls elaborate on? Specifically for .net 
applications..

Original comment by madhuric...@gmail.com on 19 Nov 2010 at 10:23

[GoogleCodeExporter]

I'm a bit uncertain on how to answer this question - but here it goes:

As you initialize the Dokan-drive you specify some options;

opt = new DokanOptions();
opt.DriveLetter = "X";
opt.VolumeLabel = "Hello";
opt.DebugMode = false;

The obvious approach would be to extend these with e.g.;

opt.MaxReadSize = 5*1024*1024; //bytes
opt.MaxWriteSize = 5*1024*1024; //bytes

or

opt.MaxBufferSize = 5*1024*1024; //bytes, in one go!

How the exact implementation of this should be done, I have no idea. It seems 
the .Net-binding uses P-invokes to call the driver / C++ version of the 
library. 

My implementation in C# for read file looks like this:

public int ReadFile(string filename, byte[] buffer, ref uint readBytes, long 
offset, Dokan.DokanFileInfo info)
{
   int bytesToRead = Math.Min(buffer.Length, 16 * 1024 * 1024); //Maximum 16 MB

...

The implementation for write file has similar constraints.

Hope my answer was elaborate enough - otherwise refrase the question and try 
again ;-)

/Toke

Original comment by tokeboy.riis on 19 Nov 2010 at 10:50

[GoogleCodeExporter]

Thanks!

Original comment by madhuric...@gmail.com on 22 Nov 2010 at 6:24

[GoogleCodeExporter]

Thanks for the heads up on this..
I have implemented some code that uses the ReadFile / Write file pInvoke API's 
that take the IntPtrs directly from the DokanNet.dll

Have a look at the changeset 
http://liquesce.codeplex.com/SourceControl/changeset/changes/58033
This shows the move from the Filestream over to using the IntPtr from Dokan, 
Therefore no extra memory is allocated above what Dokan has already done. (i.e. 
the Marshal.Copy's have been removed)

Original comment by smurf...@gmail.com on 24 Nov 2010 at 9:26

[GoogleCodeExporter]

I don't know is this a right place but I reimplemented DokanNet 6.0 proxy to 
use .NET default marshalers so the memory usage at least for DokanNetMirror 
dropped by half. 
http://code.google.com/p/dotshed/source/browse/#svn%2Ftrunk%2FDokanNet so if 
you could implement some o this in official release.

Original comment by mladenov...@gmail.com on 4 May 2011 at 10:30