Open GoogleCodeExporter opened 9 years ago
I think i know where you get the problem.
It is on Protocols\transfernmdcprotocol.cs:line 270.
Here the file is created and filled with zeros.
This is done to allow segmented downloading (where you could download last part
before the first part).
To verify that this is the cause, can you replace AllocateFile in
Utils\FileOperations.cs with the following method and report back?
public static void AllocateFile(string target, long size)
{
if (!PathExists(target))
{
//File.WriteAllBytes(target, new byte[ size]);
// Doing it as below is working on compact .Net and we can handle big
files (if you didnt have 2gb of free ram and wanted to write a 3 gb big file
this
made a problem before)
int SEGSIZE = 1024 * 1024;
using (FileStream fs = File.Create(target, SEGSIZE))
{
int i = 0;
int segmentSize = SEGSIZE;
do
{
// Set length to segment size
segmentSize = SEGSIZE;
// Is segment size bigger then content size?
if (segmentSize > size)
{
segmentSize = (int)size;
}
// Are we at last segment
else if ((1 + i) * segmentSize > size)
{
segmentSize = (int)(size % segmentSize);
}
fs.Write(new byte[segmentSize], 0, segmentSize);
System.Threading.Thread.Sleep(1);
i++;
} while ((i * SEGSIZE) < size);
}
}
}
Original comment by blomman84
on 14 Mar 2009 at 9:55
I have made changes, but this dont solves problem. I set brakepoint to
System.Threading.Thread.Sleep(1); line, it fires only once at download begin.
But I
have 85-100% processor usage during all download time.
Original comment by hackw...@gmail.com
on 15 Mar 2009 at 2:20
ok :)
Next try then.
In Connections\TcpConnection.cs
Change:
protected byte[] buffer = new byte[1024];
To:
protected byte[] buffer = new byte[8024];
In Protocols\transfernmdcprotocol.cs
Change:
if (!e.Handled)
To:
System.Threading.Thread.Sleep(1);
if (!e.Handled)
Original comment by blomman84
on 15 Mar 2009 at 3:24
in second file if (!e.Handled) meets several times i have changed following:
in void OnTimer
in both void ParseRaw
total 3 delay insert
now downloading takes 60-70% as TaskManager said
Original comment by hackw...@gmail.com
on 15 Mar 2009 at 3:49
Some additional information here. 60-70% it shows on Amd Athlon X2 6000+, 2 GB
ram.
On Pentium 4 it still shows 80-100% processor load. Usual download speed is
between
500KB/s and 3 MB/s and very low latency. My client works in local network.
Original comment by hackw...@gmail.com
on 16 Mar 2009 at 7:44
Another part of information. While testing multisource download i found that
number
of transfers does not make infuence to processor load. It shows 60-70% with 1
source
and with 40 sources.
Original comment by hackw...@gmail.com
on 18 Mar 2009 at 5:44
What happens if you change the below?
In Connections\TcpConnection.cs
Change:
protected byte[] buffer = new byte[8024];
To:
protected byte[] buffer = new byte[1048576];
Original comment by blomman84
on 18 Mar 2009 at 7:03
70-100% but now flowlib shows download speed 100-300 MBytes/s -it is unreal my
network works on 100 Mbit full duplex. if size is 1024 speed shows correctly
Original comment by hackw...@gmail.com
on 19 Mar 2009 at 2:41
maybe you can give me some start point to solve this issue? I'm not so good
understand all mechanism of download working. It is interesting that uploads
connections does not take significant processor time.
Original comment by hackw...@gmail.com
on 26 Mar 2009 at 5:20
I have realized single filestream for transfer (it was dynamic for each data
part),
but it is not a reason of problem.
Original comment by hackw...@gmail.com
on 27 Mar 2009 at 10:57
[deleted comment]
After solving issue 29 it shows 20-50% processor usage.
I'm using protected byte[] buffer = new byte[102400];
Original comment by hackw...@gmail.com
on 9 Apr 2009 at 5:03
WOW! That is a huge difference :)
Is that line the only difference between the svn version and your local copy?
Original comment by blomman84
on 9 Apr 2009 at 5:13
no, a have changed transfernmdcprotocol to support multisource download, all
changes
in issue 30
Original comment by hackw...@gmail.com
on 9 Apr 2009 at 5:22
now it is all except the buffer? :)
Original comment by blomman84
on 13 Apr 2009 at 9:18
I'm not sure because I have changed many files in flowlib to make possible its
use
in client. I will try to download new copy and test it in my client.
Original comment by hackw...@gmail.com
on 17 Apr 2009 at 8:53
Full change list
DownloadManager.cs
1. Added struct KeyValuePairS<TKey, TValue> to support xml serialization
(save/load)
2. Implemented DownloadManager save/load methods
3. Implemented DownloadManager method FlowSortedList<Source>
GetSources(DownloadItem
d)
4. Added property: public SortedList<DownloadItem, FlowSortedList<Source>>
DownloadItems
DownloadItem.cs
1. Added properties:
public long TotalBytesReceived {get; set; }
public DownloadPriority Priority { get; set; }
Connections\Hub.cs
1. Added properties:
public long TotalShare { get; set; }
2. Modified property RegMode added in set section: if (regMode == -1)
TotalShare = 0;
3. Modified protected void OnProtocolUpdate:
Actions.UserOnline: TotalShare += long.Parse(ui.Share);
Actions.UserOffline: TotalShare -= long.Parse(userInfo.Share);
4. Added function: public User GetUserByStoreID(string storeID)
transfernmdcprotocol.cs
1. In ActOnInMessage: where message is MyNick
trans.Source.UserId = trans.User.StoreID; changed to
trans.Source.UserId = "o-go.ru411" + trans.User.StoreID;
this needs to allow passive download (it dont works else, i dont know how to do
this
better)
2. Added after fs.Unlock:
if (trans.DownloadItem.Priority != DownloadItem.DownloadPriority.Pause)
trans.DownloadItem.TotalBytesReceived += length;
FileOperations.cs
1. AllocateFile function modified to skip filling with zeros, its only sets
needed
size. I'm controlling file integrity using bitarray segmentsDownloaded.
(if I start download without this change I got exception because second stream
starts writing before first has finished filling file)
TcpConnection.cs - buffer size set to 102400
After these changes i have 30-60% CPU usage.
Original comment by hackw...@gmail.com
on 20 Apr 2009 at 7:11
I have realized persist filestream object for protocol, now it takes 8-15% CPU
usage,
i think this percent got by my gui animation draw functions and other
processes. It
takes 15% when flowlib throws SocketException. I think this is enough
perfomance.
Thanks for help =)
Original comment by hackw...@gmail.com
on 24 Apr 2009 at 1:57
after that I can't debug it=) client correctly downloads the file but don't
unlocking
it and I can't use it. This happens because protocol handler continues
recieving data
after segment has finished.
Here is another problem i found: I'm trying to download 1,4 Gb file from 40
sources,
my client timer every 10 seconds checking sources which not in
transferManager.Transfers and calls StartTransfer for each of them.
It works and flowlib establish connection with 18-20 sources (its all in
transferManager) other sources has no free slots or something else.
While download windows taskmanager shows constantly increase of system used
memory
and when it takes all available memory windows gets huge lags. After download
has
been finished windows releases memory. BUT my client used memory dont grows!
wtf?
Can you give me any advise or idea to solve this?
Original comment by hackw...@gmail.com
on 24 Apr 2009 at 9:18
I have returned to old not persist filetream version of
transfernmdcprotocol.cs. It
anyway eats memory when downloading. Most of cpu usage going to manage this
memory.
Original comment by hackw...@gmail.com
on 27 Apr 2009 at 3:12
I have found the problem - it because transfer continues reciving data after
DownloadItem.Cancel was called, and it calls DownloadItem.Finished() after
recieving.
Single filestream works fine, on Windows 7100 it dont eats memory.
Original comment by hackw...@gmail.com
on 3 May 2009 at 1:25
It appears the main reason of this issue is wrong behaviuor of disconnect.
HEEEELP :(
this is last thing separating me from relese
Original comment by hackw...@gmail.com
on 14 May 2009 at 2:50
Sadly my time is limited right now but if you can give me all source files
needed for
your client i can have a look when i get time over.
Original comment by blomman84
on 17 May 2009 at 1:30
I dont get a high processor usage when downloading anymore.
Can you download latest svn and confirm this?
Original comment by blomman84
on 7 Sep 2009 at 9:29
Hi Mattias, nice to see you=)
I'm trying to use pure svn library in my client. Here the problems:
First of all I can not use IShare interface because it is not completly
realized.
For example hubnmdcprotocol.cs line 360. This is code that handles search
messages.
In my case there is no need to iterate through all items in share because
search can
be done much faster in SQL (I'm using SQL share)
Next BaseFileList.cs function CreateFilelist() - this function requires share
object
to implement ICollection interface. And then it trying to sort content. This
work
also can be done by SQL.
My suggestion is to create two virtual functions in Share class(or in IShare).
First
is Search and second is GetVirtualList. I have already done this work, you can
see it
in my example (when I tried to download all filelists om hub) Both functions
located
at document bottom.
Now about Issue. I have prepared special version of my client to work with
latest SVN
version. When I downloading from 10 sources (1Mbyte/s) I get 20-30% constant
CPU
usage (Athlon X2 6000!!!)
I suggest you to implement constant filestream for writing operation of
protocol. (In
my version it takes 5-10% when downloading from 13 sources (1Mbyte/s)) You can
find
my implementation in my last example.
Original comment by hackw...@gmail.com
on 8 Sep 2009 at 1:58
Hi :)
As you may have noticed, my time is limited :(
Can you open a new issue about IShare?
It is not related to this issue.
I will be back about this issue.
Original comment by blomman84
on 10 Sep 2009 at 7:56
Original issue reported on code.google.com by
hackw...@gmail.com
on 11 Mar 2009 at 9:13