vbauer / manet

Website screenshot service powered by Node.js, SlimerJS and PhantomJS
MIT License
577 stars 102 forks source link

concurrent time does not work #6

Closed rijing2000 closed 9 years ago

rijing2000 commented 9 years ago

Used in the project, 4 concurrent time does not work, we would like to use, and he saves us a lot of time, but concurrency issues, I do not know whether any improvements Timeout error results are screenshots actually take screenshots of the address can be quickly opened

vbauer commented 9 years ago

@rijing2000 Could you please provide more details? Is it possible to reproduce this issue? I'm interested to improve Manet too.

rijing2000 commented 9 years ago

We deployed on a Linux Manet, the configuration of this server is a dual-core 4G, just deploy a Manet, for screenshots, and then we have another server called Manet, take screenshots of the results back. Here needs note of is, we needs cut screen of address is with a address, will according to different of parameter show different of content, roughly of application scene is these, then problem is, several calls words, are can normal work, but due to we this cut screen of workload than larger, so needs concurrent multiple thread to ran, peak is 20 a concurrent, less of is 4 a concurrent, such, if concurrent words, Manet on cannot normal work has, he will large receives information, but basically not feedback has, Last is 60 seconds timeout errors

vbauer commented 9 years ago

Could you please also provide:

UPD: Have you tried to switch rendering engine from PhantomJS to SlimerJS (using "--engine" parameter)?

rijing2000 commented 9 years ago
  1. Manet's version is new,use engine is PhantomJS
  2. default.JSON no change
  3. URL example to Manet server:http://xxx.XX.com/snap.aspx?ID=XXXX, is submitted by post, I use my method, paste:
HttpWebResponse myResponse = null;
System.Drawing.Image iSource = null;
Stream strem = null;
try
{
string data = string. Format("url={0}", HttpUtility.UrlEncode(param));
byte[] postdata = System.Text.Encoding.UTF8.GetBytes(data);

string url = string. Format("{0}?width=770&height=562", SnapUrl);
//string url = "http://192.168.1.103:8891";
HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(url);
if (myRequest == null)
{
throw new Exception("myRequest is null");
}

myRequest.Method = "POST";
myRequest.ContentType = "application/x-www-form-urlencoded";
myRequest.UserAgent = "MSIE 7.0; Windows NT 5.1";
myRequest.Timeout = 1000 * 60*2;

Stream newStream = myRequest.GetRequestStream();
if (newStream == null)
{
throw new Exception("newStream is null");
}

newStream.Write(postdata, 0, postdata. Length);
newStream.Close();

myResponse = (HttpWebResponse)myRequest.GetResponse();

if (myResponse == null)
{
throw new Exception("myResponse is null");
}
strem = myResponse.GetResponseStream();

iSource = System.Drawing.Image.FromStream(strem, true);
iSource.Save(filename);
strem. Close();

}
finally
{
if (iSource != null)
{
iSource.Dispose();
}
if (strem != null)
{
strem. Dispose();
}
}
vbauer commented 9 years ago

I could not reproduce this issue. I have just tried to test using siege and got the following results:

Transactions:                    205 hits
Availability:                 100.00 %
Elapsed time:                114.96 secs
Data transferred:            1059.60 MB
Response time:                 10.17 secs
Transaction rate:               1.78 trans/sec
Throughput:                     9.22 MB/sec
Concurrency:                   18.13
Successful transactions:         205
Failed transactions:               0
Longest transaction:           17.36
Shortest transaction:           5.75

Command for stress testing:

siege.exe -c 20 "http://localhost:8891/?width=270&height=562&force=true&url=github.com"

I used 20 concurrent clients and as you could see all transactions were finished without fails.

Ofc, it works much faster without "force=true" parameter:

Transactions:                   1037 hits
Availability:                 100.00 %
Elapsed time:                  52.43 secs
Data transferred:           10999.61 MB
Response time:                  0.49 secs
Transaction rate:              19.78 trans/sec
Throughput:                   209.78 MB/sec
Concurrency:                    9.67
Successful transactions:        1037
Failed transactions:               0
Longest transaction:            0.89
Shortest transaction:           0.06

If you know the certain place in code with possible bug you can just send me pull request with fix.

PS: I have the following hardware and software configuration:

vbauer commented 9 years ago

I added possibility to use async requests. You can use this feature to prevent problem with hanging. See "callback" parameter in the README file: https://github.com/vbauer/manet/blob/master/README.md