hungdluit / flowlib

Automatically exported from code.google.com/p/flowlib
0 stars 0 forks source link

High processor usage when active download #28

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
What steps will reproduce the problem?
1. Start download any big file

When download one file from one source TaskManager shows 100% processor 
usage.

What version of the product are you using?
SVN r478

On what operating system?
Windows Vista Sp1

Please provide any additional information below.
Then download starts, my gui works very slow, Windows Forms Timers pass 
their ticks. If I move my main form over screen it lags. I have Pentium 4 
3.2 GHz dual core processor, and 1 GB RAM. I'm tryed to find place where 
thread created for each transfer but failed.

Original issue reported on code.google.com by hackw...@gmail.com on 11 Mar 2009 at 9:13

GoogleCodeExporter commented 8 years ago
I think i know where you get the problem.
It is on Protocols\transfernmdcprotocol.cs:line 270.

Here the file is created and filled with zeros.
This is done to allow segmented downloading (where you could download last part
before the first part).
To verify that this is the cause, can you replace AllocateFile in
Utils\FileOperations.cs with the following method and report back?

        public static void AllocateFile(string target, long size)
        {
            if (!PathExists(target))
            {
                //File.WriteAllBytes(target, new byte[ size]);
                // Doing it as below is working on compact .Net and we can handle big
files (if you didnt have 2gb of free ram and wanted to write a 3 gb big file 
this
made a problem before)
                int SEGSIZE = 1024 * 1024;
                using (FileStream fs = File.Create(target, SEGSIZE))
                {
                    int i = 0;
                    int segmentSize = SEGSIZE;
                    do
                    {
                        // Set length to segment size
                        segmentSize = SEGSIZE;
                        // Is segment size bigger then content size?
                        if (segmentSize > size)
                        {
                            segmentSize = (int)size;
                        }
                        // Are we at last segment
                        else if ((1 + i) * segmentSize > size)
                        {
                            segmentSize = (int)(size % segmentSize);
                        }
                        fs.Write(new byte[segmentSize], 0, segmentSize);
                        System.Threading.Thread.Sleep(1);
                        i++;
                    } while ((i * SEGSIZE) < size);
                }
            }
        }

Original comment by blomman84 on 14 Mar 2009 at 9:55

GoogleCodeExporter commented 8 years ago
I have made changes, but this dont solves problem. I set brakepoint to 
System.Threading.Thread.Sleep(1); line, it fires only once at download begin. 
But I 
have 85-100% processor usage during all download time.

Original comment by hackw...@gmail.com on 15 Mar 2009 at 2:20

GoogleCodeExporter commented 8 years ago
ok :)
Next try then.

In Connections\TcpConnection.cs
Change:
        protected byte[] buffer = new byte[1024];
To:
        protected byte[] buffer = new byte[8024];

In Protocols\transfernmdcprotocol.cs
Change:
                    if (!e.Handled)
To:
                    System.Threading.Thread.Sleep(1);
                    if (!e.Handled)

Original comment by blomman84 on 15 Mar 2009 at 3:24

GoogleCodeExporter commented 8 years ago
in second file if (!e.Handled) meets several times i have changed following:
in void OnTimer
in both void ParseRaw
total 3 delay insert
now downloading takes 60-70% as TaskManager said

Original comment by hackw...@gmail.com on 15 Mar 2009 at 3:49

GoogleCodeExporter commented 8 years ago
Some additional information here. 60-70% it shows on Amd Athlon X2 6000+, 2 GB 
ram.
On Pentium 4 it still shows 80-100% processor load. Usual download speed is 
between 
500KB/s and 3 MB/s and very low latency. My client works in local network.

Original comment by hackw...@gmail.com on 16 Mar 2009 at 7:44

GoogleCodeExporter commented 8 years ago
Another part of information. While testing multisource download i found that 
number 
of transfers does not make infuence to processor load. It shows 60-70% with 1 
source 
and with 40 sources.

Original comment by hackw...@gmail.com on 18 Mar 2009 at 5:44

GoogleCodeExporter commented 8 years ago
What happens if you change the below?
In Connections\TcpConnection.cs
Change:
        protected byte[] buffer = new byte[8024];
To:
        protected byte[] buffer = new byte[1048576];

Original comment by blomman84 on 18 Mar 2009 at 7:03

GoogleCodeExporter commented 8 years ago
70-100% but now flowlib shows download speed 100-300 MBytes/s -it is unreal my 
network works on 100 Mbit full duplex. if size is 1024 speed shows correctly

Original comment by hackw...@gmail.com on 19 Mar 2009 at 2:41

GoogleCodeExporter commented 8 years ago
maybe you can give me some start point to solve this issue? I'm not so good 
understand all mechanism of download working. It is interesting that uploads 
connections does not take significant processor time.

Original comment by hackw...@gmail.com on 26 Mar 2009 at 5:20

GoogleCodeExporter commented 8 years ago
I have realized single filestream for transfer (it was dynamic for each data 
part), 
but it is not a reason of problem.

Original comment by hackw...@gmail.com on 27 Mar 2009 at 10:57

GoogleCodeExporter commented 8 years ago
[deleted comment]
GoogleCodeExporter commented 8 years ago
After solving  issue 29  it shows 20-50% processor usage.
I'm using protected byte[] buffer = new byte[102400];

Original comment by hackw...@gmail.com on 9 Apr 2009 at 5:03

GoogleCodeExporter commented 8 years ago
WOW! That is a huge difference :)
Is that line the only difference between the svn version and your local copy?

Original comment by blomman84 on 9 Apr 2009 at 5:13

GoogleCodeExporter commented 8 years ago
no, a have changed transfernmdcprotocol to support multisource download, all 
changes 
in issue 30

Original comment by hackw...@gmail.com on 9 Apr 2009 at 5:22

GoogleCodeExporter commented 8 years ago
now it is all except the buffer? :)

Original comment by blomman84 on 13 Apr 2009 at 9:18

GoogleCodeExporter commented 8 years ago
I'm not sure because I have changed many files in flowlib to make possible its 
use 
in client. I will try to download new copy and test it in my client.

Original comment by hackw...@gmail.com on 17 Apr 2009 at 8:53

GoogleCodeExporter commented 8 years ago
Full change list

DownloadManager.cs

1. Added struct KeyValuePairS<TKey, TValue> to support xml serialization 
(save/load)
2. Implemented DownloadManager save/load methods
3. Implemented DownloadManager method FlowSortedList<Source> 
GetSources(DownloadItem 
d)
4. Added property: public SortedList<DownloadItem, FlowSortedList<Source>> 
DownloadItems

DownloadItem.cs

1. Added properties:
public long TotalBytesReceived {get; set; }
public DownloadPriority Priority { get; set; }

Connections\Hub.cs

1. Added properties:
public long TotalShare { get; set; }

2. Modified property RegMode added in set section: if (regMode == -1) 
TotalShare = 0;
3. Modified protected void OnProtocolUpdate: 
Actions.UserOnline: TotalShare += long.Parse(ui.Share);
Actions.UserOffline: TotalShare -= long.Parse(userInfo.Share);
4. Added function: public User GetUserByStoreID(string storeID)

transfernmdcprotocol.cs

1. In ActOnInMessage: where message is MyNick 
trans.Source.UserId = trans.User.StoreID; changed to
trans.Source.UserId = "o-go.ru411" + trans.User.StoreID;
this needs to allow passive download (it dont works else, i dont know how to do 
this 
better)

2. Added after fs.Unlock:
if (trans.DownloadItem.Priority != DownloadItem.DownloadPriority.Pause)
   trans.DownloadItem.TotalBytesReceived += length;

FileOperations.cs

1. AllocateFile function modified to skip filling with zeros, its only sets 
needed 
size. I'm controlling file integrity using bitarray segmentsDownloaded.
(if I start download without this change I got exception because second stream 
starts writing before first has finished filling file)

TcpConnection.cs  - buffer size set to 102400

After these changes i have 30-60% CPU usage.

Original comment by hackw...@gmail.com on 20 Apr 2009 at 7:11

GoogleCodeExporter commented 8 years ago
I have realized persist filestream object for protocol, now it takes 8-15% CPU 
usage, 
i think this percent got by my gui animation draw functions and other 
processes. It 
takes 15% when flowlib throws SocketException. I think this is enough 
perfomance. 
Thanks for help =)

Original comment by hackw...@gmail.com on 24 Apr 2009 at 1:57

GoogleCodeExporter commented 8 years ago
after that I can't debug it=) client correctly downloads the file but don't 
unlocking 
it and I can't use it. This happens because protocol handler continues 
recieving data 
after segment has finished. 
Here is another problem i found: I'm trying to download 1,4 Gb file from 40 
sources, 
my client timer every 10 seconds checking sources which not in 
transferManager.Transfers and calls StartTransfer for each of them.
It works and flowlib establish connection with 18-20 sources (its all in 
transferManager) other sources has no free slots or something else.
While download windows taskmanager shows constantly increase of system used 
memory 
and when it takes all available memory windows gets huge lags. After download 
has 
been finished windows releases  memory. BUT my client used memory dont grows! 
wtf? 
Can you give me any advise or idea to solve this?

Original comment by hackw...@gmail.com on 24 Apr 2009 at 9:18

GoogleCodeExporter commented 8 years ago
I have returned to old not persist filetream version of 
transfernmdcprotocol.cs. It 
anyway eats memory when downloading. Most of cpu usage going to manage this 
memory.

Original comment by hackw...@gmail.com on 27 Apr 2009 at 3:12

GoogleCodeExporter commented 8 years ago
I have found the problem - it because transfer continues reciving data after 
DownloadItem.Cancel was called, and it calls DownloadItem.Finished() after 
recieving.  
Single filestream works fine, on Windows 7100 it dont eats memory.

Original comment by hackw...@gmail.com on 3 May 2009 at 1:25

GoogleCodeExporter commented 8 years ago
It appears the main reason of this issue is wrong behaviuor of disconnect. 
HEEEELP :(  
this is last thing separating me from relese

Original comment by hackw...@gmail.com on 14 May 2009 at 2:50

GoogleCodeExporter commented 8 years ago
Sadly my time is limited right now but if you can give me all source files 
needed for
your client i can have a look when i get time over.

Original comment by blomman84 on 17 May 2009 at 1:30

GoogleCodeExporter commented 8 years ago
I dont get a high processor usage when downloading anymore.
Can you download latest svn and confirm this?

Original comment by blomman84 on 7 Sep 2009 at 9:29

GoogleCodeExporter commented 8 years ago
Hi Mattias, nice to see you=)
I'm trying to use pure svn library in my client. Here the problems:
First of all I can not use IShare interface because it is not completly 
realized.
For example hubnmdcprotocol.cs line 360. This is code that handles search 
messages. 
In my case there is no need to iterate through all items in share because 
search can 
be done much faster in SQL (I'm using SQL share)
Next BaseFileList.cs function CreateFilelist() - this function requires share 
object 
to implement ICollection interface. And then it trying to sort content. This 
work 
also can be done by SQL.
My suggestion is to create two virtual functions in Share class(or in IShare). 
First 
is Search and second is GetVirtualList. I have already done this work, you can 
see it 
in my example (when I tried to download all filelists om hub) Both functions 
located 
at document bottom.
Now about Issue. I have prepared special version of my client to work with 
latest SVN 
version. When I downloading from 10 sources (1Mbyte/s) I get 20-30% constant 
CPU 
usage (Athlon X2 6000!!!)
I suggest you to implement constant filestream for writing operation of 
protocol. (In 
my version it takes 5-10% when downloading from 13 sources (1Mbyte/s)) You can 
find 
my implementation in my last example.

Original comment by hackw...@gmail.com on 8 Sep 2009 at 1:58

GoogleCodeExporter commented 8 years ago
Hi :)
As you may have noticed, my time is limited :(

Can you open a new issue about IShare?
It is not related to this issue.

I will be back about this issue.

Original comment by blomman84 on 10 Sep 2009 at 7:56