Open GoogleCodeExporter opened 9 years ago
It looks like I found reason of first bug. In function AllocateFile in line
using (FileStream fs = File.Create(target, SEGSIZE))
i have changed to following
using (FileStream fs = new FileStream(target, FileMode.OpenOrCreate,
FileAccess.Write, FileShare.ReadWrite, SEGSIZE))
Original comment by hackw...@gmail.com
on 19 Mar 2009 at 6:12
Something wrong in file transfernmdcprotocol.cs near line 283. Application
throws
many exceptions on that line but flowlib handle it. Sometimes it throws
unhandled
exception at line 302 fs.Unlock( - telling what this segment already unlocked.
It
can be avoided using try-catch block, but after that download goes in 1 or 2
streams :(
Original comment by hackw...@gmail.com
on 19 Mar 2009 at 10:26
Fresh news=) I think i found bug reason. it around
DownloadSegment.GetAvailable()
function. after several threads starts, GetAvailable returns equal segment for
all
streams. After that all streams trys to block same part of file. We need
mechanism
to prevent this behaviour. Maybe use lock while stream get available segment
and
reserving it. And another interesting moment, DownloadItem.Start should mark
segmentInProgress, but it change segmentsDownloaded. Now trying to fix that...
=)
Original comment by hackw...@gmail.com
on 27 Mar 2009 at 7:03
I think i solve this problem. I have added locks into transfernmdcprotocol.cs
and
set of fix DownloadItem.cs. I dont know, but maybe adcprotocol.cs should be
fixed
too. source attached
Original comment by hackw...@gmail.com
on 27 Mar 2009 at 8:07
Attachments:
1) "It looks like I found reason of first bug. In function AllocateFile in line
using (FileStream fs = File.Create(target, SEGSIZE))
i have changed to following
using (FileStream fs = new FileStream(target, FileMode.OpenOrCreate,
FileAccess.Write, FileShare.ReadWrite, SEGSIZE))"
Instead of doing the above you should call Utils.FileOperations.AllocateFile
before
starting any download.
Otherwise you might overwrite your downloaded content with zeros.
This is also how bittorrent clients does it.
2) "Something wrong in file transfernmdcprotocol.cs near line 283."
What type of exception do you get on this line?
3) "Sometimes it throws unhandled exception at line 302 fs.Unlock[...]"
Try to move the below lines to the line after: trans.CurrentSegment.Position +=
length;
// Saves and unlocks file
fs.Flush();
fs.Unlock(trans.CurrentSegment.Start, trans.CurrentSegment.Length);
4) DownloadItem.GetAvailable()
In the following functions:
GetAvailable
Cancel
Start
Finished
add:
lock (this)
{
[...]
}
where [...] is the content of the function.
Please return with feedback :)
Original comment by blomman84
on 29 Mar 2009 at 7:10
1. I was wrong in first comment, AllocateFile should be called only once at
download
starts. It calls several times because GetAvailable() return segment 0 for
several
transfers.
2, 3. this is not necessary now, after my fix in comment 4 all exceptions
including
fs.Unlock are gone. I have added object needSegment and have lock it while
stream
calls GetAvailable and Start to reserv segment for itself. it is all in
transfernmdcprotocol.cs.
4. I have restored svn versions of changed files, and have added locks as you
said.
This not helps because all threads get their segment before first thread calls
Start
and reserved first segment.
Original comment by hackw...@gmail.com
on 30 Mar 2009 at 2:10
Another bug in GetAvailable. bitwise OR operation changes segmentsDownloaded. I
think it should be like
BitArray tmp2=new BitArray(segmentsDownloaded);
tmp = tmp2.Or(this.segmentsInProgress);
Original comment by hackw...@gmail.com
on 30 Mar 2009 at 2:26
SVN updated. GetAvailable is now calling Start and i fixed the bug from comment
7.
Can you test and get back to me.
hopes it work as it should :)
Original comment by blomman84
on 31 Mar 2009 at 8:48
now its working better. but here 3 moments.
1. in transfernmdcprotocol.cs in OnDownload()
// Set that we are actually downloading stuff
if (trans.CurrentSegment.Index >= 0)
{
trans.DownloadItem.Start(trans.CurrentSegment.Index,
trans.Source);
}
is this needed?
2. in DownloadItem.cs bool Start(int pos, Source src, bool lockResources)
Is this function should change segmentsInProgress instead of segmentsDownloaded
?
3. When downloading 8gb file from 6 sources (after some time working) i got
exception
on line using (System.IO.FileStream fs = new System.IO.FileStream
(trans.DownloadItem.ContentInfo.Get(ContentInfo.STORAGEPATH),
System.IO.FileMode.OpenOrCreate, System.IO.FileAccess.Write,
System.IO.FileShare.Write))
file locked by other process.
I have worked around it, sometimes this line calls after DownloadItem.Cancel
called,
maybe set CurrentSegment=to -1 after DownloadItem.Cancel
Original comment by hackw...@gmail.com
on 1 Apr 2009 at 4:53
Sometimes flowlib dont fires DownloadItem.DownloadCompleted, but all
segmentsDownloaded set to true
Original comment by hackw...@gmail.com
on 9 Apr 2009 at 5:43
1) Fixed.
2) Fixed.
3) Think this has been fixed.
Do you know the cause of the DownloadCompleted thingy? Any more info to go on?
SVN Updated
Original comment by blomman84
on 13 Apr 2009 at 8:58
try to download several files from many sources and you will get the problem.
I'm
trying to solve it but cant because i dont really know how it works. Also I
need
thread-safe access to transfer manager, because i got many exceptions while
showing
its data :(
Original comment by hackw...@gmail.com
on 17 Apr 2009 at 8:50
Possible in DownloadItem.Finished segDoneCount should be in lock section to
avoid
multiple increment at once. I have made it and it works.
Original comment by hackw...@gmail.com
on 24 Apr 2009 at 8:55
TransferManager should now be more thread safe
Original comment by blomman84
on 17 May 2009 at 1:39
Sometimes my client stops downloading when it is 1 or 2 segments left. I have
EMPTY
TransferManager. It is because flowlib transfernmdcprotocol dont releases
segment. I
discovered this after adding debug messages to DownloadItem.Start Finished and
Cancel
methods. These segments been released only when I close application.
Original comment by hackw...@gmail.com
on 27 May 2009 at 2:41
Why do you want to lock the segDoneCount in DownloadItem.Finished?
"sometimes my client stops downloading when it is 1 or 2 segments left"
I have found some places where this could happen. (Svn Updated)
Do you still get problems? Have i create some other problems? :)
Original comment by blomman84
on 28 Jun 2009 at 1:25
[deleted comment]
It looks like now it works correclty, but I'm not sure. It needs some time
while my
users test it.
*Why do you want to lock the segDoneCount in DownloadItem.Finished?*
to avoid bug when segDoneCount equals to segTotalCount but there is still
segments wich
not downloaded
Original comment by hackw...@gmail.com
on 30 Jun 2009 at 9:17
No, some users still get that problem. 1 segment left, TransferManager is empty
and
that segment is locked by not-disposed transfer.
Original comment by hackw...@gmail.com
on 3 Jul 2009 at 4:55
I think i know why your TransferManager is empty.
I have found 2 bugs.
One in my example code and one in the actual code.
Update to latest SVN.
Then add this line:
transferManager.AddTransfer(trans);
to your handling of Protocol.RequestTransfer.
Se example below:
void Protocol_RequestTransfer(object sender, FmdcEventArgs e)
{
ITransfer trans = sender as ITransfer;
TransferRequest req = e.Data as TransferRequest;
req = transferManager.GetTransferReq(req.Key);
if (trans != null && req != null)
{
e.Handled = true;
e.Data = req;
transferManager.RemoveTransferReq(req.Key);
transferManager.AddTransfer(trans);
}
}
Do you see the transfer now?
Original comment by blomman84
on 4 Jul 2009 at 10:18
After this fix I cant download anything in active mode. Protocol stops
receiving
after lock message.
Original comment by hackw...@gmail.com
on 10 Jul 2009 at 3:40
Where do you call: StartTransfer and AddTransfer in TransferManager?
Original comment by blomman84
on 12 Jul 2009 at 7:07
StartTransfer Called only at
void Protocol_Update(object sender, FlowLib.Events.FmdcEventArgs e){
...
case Actions.TransferStarted:
Transfer trans = e.Data as Transfer;
Program.Stats.BindStat(trans);
if (trans != null)
{
if (trans.Protocol is TransferNmdcProtocol)
{
(trans.Protocol as TransferNmdcProtocol).Encoding =
Encoding.Default;
}
trans.Protocol.ChangeDownloadItem += new
FmdcEventHandler(Protocol_ChangeDownloadItem);
trans.Protocol.RequestTransfer += new
FmdcEventHandler(Protocol_RequestTransfer);
transferManager.StartTransfer(trans);
}
break;
AddTransfer called twice:
void Protocol_RequestTransfer(object sender, FmdcEventArgs e)
{
ITransfer trans = sender as ITransfer;
TransferRequest req = e.Data as TransferRequest;
req = transferManager.GetTransferReq(req.Key);
if (trans!=null && req != null)
{
e.Handled = true;
e.Data = req;
transferManager.RemoveTransferReq(req.Key);
transferManager.AddTransfer(trans);
}
}
and in
void Connection_Update(object sender, FmdcEventArgs e)
{
switch (e.Action)
{
case Actions.TransferStarted:
Transfer trans = e.Data as Transfer;
if (trans != null)
{
if (trans.Protocol == null)
{
trans.Protocol = new
FlowLib.Protocols.TransferNmdcProtocol(trans);
trans.Protocol.Encoding = Encoding.Default;
Program.Stats.BindStat(trans);
trans.Listen();
transferManager.AddTransfer(trans);
}
trans.Protocol.ChangeDownloadItem += new
FmdcEventHandler(Protocol_ChangeDownloadItem);
trans.Protocol.RequestTransfer += new
FmdcEventHandler(Protocol_RequestTransfer);
trans.ProtocolChange += new
FmdcEventHandler(trans_ProtocolChange);
e.Handled = true;
}
break;
}
}
Original comment by hackw...@gmail.com
on 13 Jul 2009 at 1:55
Then remove transferManager.AddTransfer(trans); in Protocol_RequestTransfer
again.
You should have all transfers in transferManager.
Original comment by blomman84
on 13 Jul 2009 at 7:27
It is difficult to manage flowlib active/passive downloads in one application
=). Now
if I try to download 10+ filelists at once I get my application freezed by
crossed
locks on transfer manager and download manager. I think i should rewrite part
of my
client wich works with flowlib. I will try today.
Original comment by hackw...@gmail.com
on 13 Jul 2009 at 7:58
Strange behaviour. I'm trying to download all filelists from my hub. I have
written
small script. It adds 3 download requests in active mode. After one is finished
it
adds another one from next user. After start my program downloads 3-5
filelists, and
then it stop to download. TransferManager contains 3 requests. The situation is
repeated after program restart.
Original comment by hackw...@gmail.com
on 13 Jul 2009 at 1:51
I have prepared this example for you. This is simple console program that
trying to
download all filelits from hub using active NMDC connection. It stores data in
SQLite
database. It also includes my modification of FlowLib. It uses SQLite share,
and
persist filestream for NMDC transfer. All other parts like in last SVN revision.
This program fails after 5-6 files downloaded. All text written in english.
Original comment by hackw...@gmail.com
on 14 Jul 2009 at 9:18
Attachments:
to compile add reference to flowlib project:
System.Data.SQLite.dll
file can be found at FilelistsDownloader\FilelistsDownloader\bin\Debug
Original comment by hackw...@gmail.com
on 14 Jul 2009 at 1:01
Excellent example, I will look at the SQLiteShare later when i have more time :)
I can reproduce your problem with your example code but only with your exmple
code :(
When I try to download 12 filelists from different users with the example I have
attached all seems to work fine. :S Do you see anything that can explain it?
Can it
be the SQLLite or the Threading?
Side note:
Thanks to your example code i have found a new bug. It seems like FlowLib kept
handling commands from buffer even if the connection was closed. Should be
fixed in
latest SVN.
Original comment by blomman84
on 15 Jul 2009 at 7:32
Attachments:
Sorry for the long wait with this issue.
I must have been sleeping for a while.
With my last example i forgot to try to download a BIG file ... :S Sorry for
that.
Now i can reproduce 2 problems.
* It was impossible to download a file bigger then 2GiB if you didn't create
the file
on your filesystem before starting the download (File keept on growing because
FlowLib used a int instead of a long).
* If you start many transfers (trying to download same file) AND file doesn't
exist
on your filesystem. Transfers will "collide" as the first one is in
AllocateFile.
Fix for first problem has been added to svn.
I dont know yet how to fix the second one in a good way.
A way to bypass it however is to call AllocateFile before starting any downloads
(Make sure it has completed before starting downloads).
Example:
ContentInfo ci = new ContentInfo(ContentInfo.TTH,
"WBAVJMJV53N5YSYTV3SYSNTMWCV3NMQ6WDIWX7I");
ci.Set(ContentInfo.STORAGEPATH, currentDir + "some-file.iso");
ci.Size = 2443255643L;
//Allocate file
FileOperations.AllocateFile(ci.Get(ContentInfo.STORAGEPATH), ci.Size);
Original comment by blomman84
on 25 Jul 2009 at 6:46
[deleted comment]
I have fixed second issue 2-3 month ago by adding lock for current
downloadItem near
calling of AllocateFile (in transfernmdcprotocol) it works fine.
Original comment by hackw...@gmail.com
on 25 Jul 2009 at 3:02
Also I have changed AllocateFile function now it only creates new file, old
version
takes some time to fill file. It makes lag to user. And I think it is not
needed.
Original comment by hackw...@gmail.com
on 25 Jul 2009 at 3:10
Original issue reported on code.google.com by
hackw...@gmail.com
on 18 Mar 2009 at 5:37