Unable to download big file from many sources

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Join hub
2. Start dowbnloading big file with many sources 4 GB /80 Sources
3. Search for alternatives, add all sources to download item
4. call start transfer for all sources

What is the expected output? What do you see instead?
it should work i think=)

What version of the product are you using?
SVN r482

On what operating system?
Windows 7 7022

Call Stack? ~~(Line numbers and exception type helps much)~~
First IOException:
The process cannot access the file '{File path}' because it is being used 
by another process.
File: transfernmdcprotocol.cs Line: 278

Second SocketException:
remote host has closed connection
File: Connections\TcpConnectionListener.cs Line: 91

Please provide any additional information below.
First of all i got exception №1, i have avoided it using limit for active 
transfers (40)
after that it start downloading correctly (using 30-40 sources), but some 
time later i got second exception. I have changed 
catch (ObjectDisposedException)
to
catch (Exception)
and then it looks like starts downloading normally

Original issue reported on code.google.com by hackw...@gmail.com on 18 Mar 2009 at 5:37

[GoogleCodeExporter]

It looks like I found reason of first bug. In function AllocateFile in line 
using (FileStream fs = File.Create(target, SEGSIZE))
i have changed to following
using (FileStream fs = new FileStream(target, FileMode.OpenOrCreate, 
FileAccess.Write, FileShare.ReadWrite, SEGSIZE))

Original comment by hackw...@gmail.com on 19 Mar 2009 at 6:12

[GoogleCodeExporter]

Something wrong in file transfernmdcprotocol.cs near line 283. Application 
throws 
many exceptions on that line but flowlib handle it. Sometimes it throws 
unhandled 
exception at line 302 fs.Unlock( - telling what this segment already unlocked. 
It 
can be avoided using try-catch block, but after that download goes in 1 or 2 
streams :(

Original comment by hackw...@gmail.com on 19 Mar 2009 at 10:26

[GoogleCodeExporter]

Fresh news=) I think i found bug reason. it around 
DownloadSegment.GetAvailable() 
function. after several threads starts, GetAvailable returns equal segment for 
all 
streams. After that all streams trys to block same part of file. We need 
mechanism 
to prevent this behaviour. Maybe use lock while stream get available segment 
and 
reserving it. And another interesting moment, DownloadItem.Start should mark 
segmentInProgress, but it change segmentsDownloaded. Now trying to fix that... 
=)

Original comment by hackw...@gmail.com on 27 Mar 2009 at 7:03

[GoogleCodeExporter]

I think i solve this problem. I have added locks into transfernmdcprotocol.cs 
and 
set of fix DownloadItem.cs. I dont know, but maybe adcprotocol.cs should be 
fixed 
too. source attached

Original comment by hackw...@gmail.com on 27 Mar 2009 at 8:07

Attachments:

• fix.rar

[GoogleCodeExporter]

1) "It looks like I found reason of first bug. In function AllocateFile in line 
using (FileStream fs = File.Create(target, SEGSIZE))
i have changed to following
using (FileStream fs = new FileStream(target, FileMode.OpenOrCreate, 
FileAccess.Write, FileShare.ReadWrite, SEGSIZE))"

Instead of doing the above you should call Utils.FileOperations.AllocateFile 
before
starting any download.
Otherwise you might overwrite your downloaded content with zeros.
This is also how bittorrent clients does it.

2) "Something wrong in file transfernmdcprotocol.cs near line 283."

What type of exception do you get on this line?

3) "Sometimes it throws unhandled exception at line 302 fs.Unlock[...]"

Try to move the below lines to the line after: trans.CurrentSegment.Position += 
length;

// Saves and unlocks file
fs.Flush();
fs.Unlock(trans.CurrentSegment.Start, trans.CurrentSegment.Length);

4) DownloadItem.GetAvailable() 

In the following functions:
  GetAvailable
  Cancel
  Start
  Finished

add:

lock (this)
{
  [...]
}

where [...] is the content of the function.

Please return with feedback :)

Original comment by blomman84 on 29 Mar 2009 at 7:10

• Changed state: Accepted

[GoogleCodeExporter]

1. I was wrong in first comment, AllocateFile should be called only once at 
download 
starts. It calls several times because GetAvailable() return segment 0 for 
several 
transfers. 
2, 3. this is not necessary now, after my fix in comment 4 all exceptions 
including 
fs.Unlock are gone. I have added object needSegment and have lock it while 
stream 
calls GetAvailable and Start to reserv segment for itself. it is all in 
transfernmdcprotocol.cs.
4. I have restored svn versions of changed files, and have added locks as you 
said. 
This not helps because all threads get their segment before first thread calls 
Start 
and reserved first segment.

Original comment by hackw...@gmail.com on 30 Mar 2009 at 2:10

[GoogleCodeExporter]

Another bug in GetAvailable. bitwise OR operation changes segmentsDownloaded. I 
think it should be like 
BitArray tmp2=new BitArray(segmentsDownloaded);
tmp = tmp2.Or(this.segmentsInProgress);

Original comment by hackw...@gmail.com on 30 Mar 2009 at 2:26

[GoogleCodeExporter]

SVN updated. GetAvailable is now calling Start and i fixed the bug from comment 
7.
Can you test and get back to me.
hopes it work as it should :)

Original comment by blomman84 on 31 Mar 2009 at 8:48

[GoogleCodeExporter]

now its working better. but here 3 moments.
1. in transfernmdcprotocol.cs in OnDownload()
                        // Set that we are actually downloading stuff
                        if (trans.CurrentSegment.Index >= 0)
                        {
                            trans.DownloadItem.Start(trans.CurrentSegment.Index, 
trans.Source);
                        }
is this needed?
2. in DownloadItem.cs bool Start(int pos, Source src, bool lockResources)
Is this function should change segmentsInProgress instead of segmentsDownloaded 
?
3. When downloading 8gb file from 6 sources (after some time working) i got 
exception
on line  using (System.IO.FileStream fs = new System.IO.FileStream
(trans.DownloadItem.ContentInfo.Get(ContentInfo.STORAGEPATH), 
System.IO.FileMode.OpenOrCreate, System.IO.FileAccess.Write, 
System.IO.FileShare.Write))
file locked by other process.

I have worked around it, sometimes this line calls after DownloadItem.Cancel 
called, 
maybe set CurrentSegment=to -1 after DownloadItem.Cancel

Original comment by hackw...@gmail.com on 1 Apr 2009 at 4:53

[GoogleCodeExporter]

Sometimes flowlib dont fires DownloadItem.DownloadCompleted, but all 
segmentsDownloaded set to true

Original comment by hackw...@gmail.com on 9 Apr 2009 at 5:43

[GoogleCodeExporter]

1) Fixed.
2) Fixed.
3) Think this has been fixed.

Do you know the cause of the DownloadCompleted thingy? Any more info to go on?

SVN Updated

Original comment by blomman84 on 13 Apr 2009 at 8:58

[GoogleCodeExporter]

try to download several files from many sources and you will get the problem. 
I'm 
trying to solve it but cant because i dont really know how it works. Also I 
need 
thread-safe access to transfer manager, because i got many exceptions while 
showing 
its data :(

Original comment by hackw...@gmail.com on 17 Apr 2009 at 8:50

[GoogleCodeExporter]

Possible in DownloadItem.Finished segDoneCount should be in lock section to 
avoid 
multiple increment at once. I have made it and it works.

Original comment by hackw...@gmail.com on 24 Apr 2009 at 8:55

[GoogleCodeExporter]

TransferManager should now be more thread safe

Original comment by blomman84 on 17 May 2009 at 1:39

[GoogleCodeExporter]

Sometimes my client stops downloading when it is 1 or 2 segments left. I have 
EMPTY 
TransferManager. It is because flowlib transfernmdcprotocol dont releases 
segment. I 
discovered this after adding debug messages to DownloadItem.Start Finished and 
Cancel 
methods. These segments been released only when I close application.

Original comment by hackw...@gmail.com on 27 May 2009 at 2:41

[GoogleCodeExporter]

Why do you want to lock the segDoneCount in DownloadItem.Finished?

"sometimes my client stops downloading when it is 1 or 2 segments left"

I have found some places where this could happen. (Svn Updated)
Do you still get problems? Have i create some other problems? :)

Original comment by blomman84 on 28 Jun 2009 at 1:25

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

It looks like now it works correclty, but I'm not sure. It needs some time 
while my 
users test it.

*Why do you want to lock the segDoneCount in DownloadItem.Finished?*
to avoid bug when segDoneCount equals to segTotalCount but there is still 
segments wich 
not downloaded

Original comment by hackw...@gmail.com on 30 Jun 2009 at 9:17

[GoogleCodeExporter]

No, some users still get that problem. 1 segment left, TransferManager is empty 
and 
that segment is locked by not-disposed transfer.

Original comment by hackw...@gmail.com on 3 Jul 2009 at 4:55

[GoogleCodeExporter]

I think i know why your TransferManager is empty.
I have found 2 bugs.
One in my example code and one in the actual code.

Update to latest SVN.

Then add this line:
                transferManager.AddTransfer(trans);

to your handling of Protocol.RequestTransfer.
Se example below:

        void Protocol_RequestTransfer(object sender, FmdcEventArgs e)
        {
            ITransfer trans = sender as ITransfer;
            TransferRequest req = e.Data as TransferRequest;
            req = transferManager.GetTransferReq(req.Key);
            if (trans != null && req != null)
            {
                e.Handled = true;
                e.Data = req;
                transferManager.RemoveTransferReq(req.Key);
                transferManager.AddTransfer(trans);
            }
        }

Do you see the transfer now?

Original comment by blomman84 on 4 Jul 2009 at 10:18

[GoogleCodeExporter]

After this fix I cant download anything in active mode. Protocol stops 
receiving 
after lock message.

Original comment by hackw...@gmail.com on 10 Jul 2009 at 3:40

[GoogleCodeExporter]

Where do you call: StartTransfer and AddTransfer in TransferManager?

Original comment by blomman84 on 12 Jul 2009 at 7:07

[GoogleCodeExporter]

StartTransfer Called only at 
void Protocol_Update(object sender, FlowLib.Events.FmdcEventArgs e){
...
                case Actions.TransferStarted:

                    Transfer trans = e.Data as Transfer;
                    Program.Stats.BindStat(trans);

                    if (trans != null)
                    {
                        if (trans.Protocol is TransferNmdcProtocol)
                        {
                            (trans.Protocol as TransferNmdcProtocol).Encoding = 
Encoding.Default;
                        }
                        trans.Protocol.ChangeDownloadItem += new 
FmdcEventHandler(Protocol_ChangeDownloadItem);
                        trans.Protocol.RequestTransfer += new 
FmdcEventHandler(Protocol_RequestTransfer);
                        transferManager.StartTransfer(trans);
                    }
                    break;

AddTransfer called twice:
        void Protocol_RequestTransfer(object sender, FmdcEventArgs e)
        {
            ITransfer trans = sender as ITransfer;
            TransferRequest req = e.Data as TransferRequest;
            req = transferManager.GetTransferReq(req.Key);
            if (trans!=null && req != null)
            {
                e.Handled = true;
                e.Data = req;
                transferManager.RemoveTransferReq(req.Key);
                transferManager.AddTransfer(trans);
            }
        }

and in

        void Connection_Update(object sender, FmdcEventArgs e)
        {
            switch (e.Action)
            {
                case Actions.TransferStarted:
                    Transfer trans = e.Data as Transfer;
                    if (trans != null)
                    {
                        if (trans.Protocol == null)
                        {
                            trans.Protocol = new 
FlowLib.Protocols.TransferNmdcProtocol(trans);

                            trans.Protocol.Encoding = Encoding.Default;
                            Program.Stats.BindStat(trans);
                            trans.Listen();

                            transferManager.AddTransfer(trans);
                        }

                        trans.Protocol.ChangeDownloadItem += new 
FmdcEventHandler(Protocol_ChangeDownloadItem);
                        trans.Protocol.RequestTransfer += new 
FmdcEventHandler(Protocol_RequestTransfer);
                        trans.ProtocolChange += new 
FmdcEventHandler(trans_ProtocolChange);
                        e.Handled = true;
                    }
                    break;
            }
        }

Original comment by hackw...@gmail.com on 13 Jul 2009 at 1:55

[GoogleCodeExporter]

Then remove transferManager.AddTransfer(trans); in Protocol_RequestTransfer 
again.
You should have all transfers in transferManager.

Original comment by blomman84 on 13 Jul 2009 at 7:27

[GoogleCodeExporter]

It is difficult to manage flowlib active/passive downloads in one application 
=). Now 
if I try to download 10+ filelists at once I get my application freezed by 
crossed 
locks on transfer manager and download manager. I think i should rewrite part 
of my 
client wich works with flowlib. I will try today.

Original comment by hackw...@gmail.com on 13 Jul 2009 at 7:58

[GoogleCodeExporter]

Strange behaviour. I'm trying to download all filelists from my hub. I have 
written 
small script. It adds 3 download requests in active mode. After one is finished 
it 
adds another one from next user. After start my program downloads 3-5 
filelists, and 
then it stop to download. TransferManager contains 3 requests. The situation is 
repeated after program restart.

Original comment by hackw...@gmail.com on 13 Jul 2009 at 1:51

[GoogleCodeExporter]

I have prepared this example for you. This is simple console program that 
trying to 
download all filelits from hub using active NMDC connection. It stores data in 
SQLite 
database. It also includes my modification of FlowLib. It uses SQLite share, 
and 
persist filestream for NMDC transfer. All other parts like in last SVN revision.
This program fails after 5-6 files downloaded. All text written in english.

Original comment by hackw...@gmail.com on 14 Jul 2009 at 9:18

Attachments:

• FilelistDownloader.zip

[GoogleCodeExporter]

to compile add reference to flowlib project:
System.Data.SQLite.dll 
file can be found at FilelistsDownloader\FilelistsDownloader\bin\Debug

Original comment by hackw...@gmail.com on 14 Jul 2009 at 1:01

[GoogleCodeExporter]

Excellent example, I will look at the SQLiteShare later when i have more time :)

I can reproduce your problem with your example code but only with your exmple 
code :(
When I try to download 12 filelists from different users with the example I have
attached all seems to work fine. :S Do you see anything that can explain it? 
Can it
be the SQLLite or the Threading?

Side note:
Thanks to your example code i have found a new bug. It seems like FlowLib kept
handling commands from buffer even if the connection was closed. Should be 
fixed in
latest SVN.

Original comment by blomman84 on 15 Jul 2009 at 7:32

Attachments:

• MultiConnections.cs

[GoogleCodeExporter]

Sorry for the long wait with this issue.
I must have been sleeping for a while.
With my last example i forgot to try to download a BIG file ... :S Sorry for 
that.

Now i can reproduce 2 problems.

* It was impossible to download a file bigger then 2GiB if you didn't create 
the file
on your filesystem before starting the download (File keept on growing because
FlowLib used a int instead of a long).

* If you start many transfers (trying to download same file) AND file doesn't 
exist
on your filesystem. Transfers will "collide" as the first one is in 
AllocateFile.

Fix for first problem has been added to svn.
I dont know yet how to fix the second one in a good way.
A way to bypass it however is to call AllocateFile before starting any downloads
(Make sure it has completed before starting downloads).
Example:

ContentInfo ci = new ContentInfo(ContentInfo.TTH,
"WBAVJMJV53N5YSYTV3SYSNTMWCV3NMQ6WDIWX7I");
ci.Set(ContentInfo.STORAGEPATH, currentDir + "some-file.iso");
ci.Size = 2443255643L;

//Allocate file
FileOperations.AllocateFile(ci.Get(ContentInfo.STORAGEPATH), ci.Size);

Original comment by blomman84 on 25 Jul 2009 at 6:46

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

I have fixed second issue  2-3 month ago by adding lock for current 
downloadItem near 
calling of AllocateFile (in transfernmdcprotocol) it works fine.

Original comment by hackw...@gmail.com on 25 Jul 2009 at 3:02

[GoogleCodeExporter]

Also I have changed AllocateFile function now it only creates new file, old 
version  
takes some time to fill file. It makes lag to user. And I think it is not 
needed.

Original comment by hackw...@gmail.com on 25 Jul 2009 at 3:10