search

toshsan / caliper

Automatically exported from code.google.com/p/caliper
Apache License 2.0
0 stars 0 forks source link

Release the webapp #255

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
After printing my config, I realized the benchmark is trying to upload results 
to https://microbenchmarks.appspot.com.
while this is generally a nice idea, there are some problems:

1. this is not documented anywhere.
2. the uploaded data may be sensitive.
3. some development servers have no internet access, there is currently zero 
information as to why an upload files right now.

I realized results are being dumped into a json file in ~/.caliper/results.
is there a way to visualize those results?
also, Is there a way to run the webapp locally?

I found the the default run prints not a word about run lengths, here is a 
sample output:

Experiment selection: 
  Instruments:   [allocation, micro]
  User parameters:   {}
  Virtual machines:  [default]
  Selection type:    Full cartesian product

This selection yields 2 experiments.
Starting experiment 1 of 2: {instrument=allocation, method=Div, vm=default, 
parameters={}}
Complete!
Starting experiment 2 of 2: {instrument=micro, method=Div, vm=default, 
parameters={}}
ERROR: Hotspot compilation occurred during timing. Warmup is likely insufficent.
Complete!

Execution complete: 25.29s.
Collected 27 measurements from:
  2 instrument(s)
  1 virtual machine(s)
  1 benchmark(s)
Some trials failed to upload. Consider uploading them manually.

Original issue reported on code.google.com by o...@yadan.net on 24 May 2013 at 7:12

GoogleCodeExporter commented 9 years ago 
s/upload files right now/upload FAILS right now

Original comment by o...@yadan.net on 25 May 2013 at 7:38

GoogleCodeExporter commented 9 years ago 
I'm observing the same problems. I found the results in ~/.caliper/results but 
how can I visualize it?
Please give us the possibility to evaluate these results locally! (run the 
webapp locally?)

Execution complete: 3.488m.
Collected 30 measurements from:
 1 instrument(s)
 1 virtual machine(s)
 6 benchmark(s)
Some trials failed to upload. Consider uploading them manually.

How can I upload them manually?

Thanks!

Original comment by martin.i...@gmail.com on 15 Jun 2013 at 1:35

GoogleCodeExporter commented 9 years ago 
Since I just always have failed uploads to the web UI, I build my own tool for 
analyzing caliper results. It's quite spartanic right now; text mode only. It 
will just summarize the results by the variable parameters it detects, compute 
appropriately weighted averages etc.
It will also now try to predict a trend for numerical parameters. But as of 
now, this is not yet really satisfactory yet. You may need extensive series for 
this to really work, so I will probably make this optional at some point when I 
start adding command line parameters.

https://code.google.com/p/caliper-analyze/

I'd also appreciate someone contributing e.h. a JFreeChart based visualization 
of the results, or similar UI niceness. I'm not really a UI guy.

Original comment by erich.sc...@gmail.com on 7 Jul 2013 at 10:31

GoogleCodeExporter commented 9 years ago 
I have created a R script [1] which parses the JSON output and creates multiple 
graphics. (bar and box plot)
But you can also use it for other R statistic analysis.

[1] 
https://github.com/millecker/applications/blob/master/util/caliper/CaliperResult
s.R

Original comment by v.keusch...@gmail.com on 7 Jul 2013 at 2:13

GoogleCodeExporter commented 9 years ago 
Somehow I missed this issue the first time through.  Sorry about that.

The webapp will be runnable locally.  It was designed to do so, but we just 
haven't worked out the final bit of deployment.  You'll also be able to run it 
in your own servlet container.

The 3rd-party tools are great!  I've even done a bit of analysis with R.  
That's exactly the type of thing that we were hoping to enable with the JSON 
output.  Glad to see that it's working out for others as well.

Anyway, this will be the tracking bug for releasing the webapp.

Original comment by gak@google.com on 8 Jul 2013 at 5:37

GoogleCodeExporter commented 9 years ago 
[deleted comment]
GoogleCodeExporter commented 9 years ago 
This R tool is really great - one drawback is that when you haven't got any 
parameters it throws error:
Error in setNames(data.frame(data), value.name) : 
  'names' attribute [1] must be the same length as the vector [0]
Calls: t ... melt.list -> l

Second thing - You're not using weight value (it's number of reps for each 
trial) - in fact your results are almost always not accurate, because it's just 
sum of all execution times...

Original comment by jkubryn...@gmail.com on 14 Jul 2013 at 8:31

GoogleCodeExporter commented 9 years ago 
I've created veeery simple parser in Groovy just to show average time of single 
iteration: https://gist.github.com/jkubrynski/5995874

Original comment by jkubryn...@gmail.com on 14 Jul 2013 at 8:33

GoogleCodeExporter commented 9 years ago 
Hi jkubrynski,
thanks for your comments!

You are right, I have calculated the arithmetic mean by 
 avg += magnitude / amountOfMeasurements

I didn't see the weight value. You think this would be much more accurate
  avg += magnitude/weight ?

Do you know how the weight is defined exactly? 
Weight of measurement depending on what?

I haven't used the R tool without parameters, maybe I will fix this later.
If you have some ideas for further improvements, I would be happy to hear from 
you. Thanks!

Original comment by martin.i...@gmail.com on 14 Jul 2013 at 8:56

GoogleCodeExporter commented 9 years ago 
Starting this morning around 7 AM pacific time, I've been unable to upload my 
caliper trial data. The online web app appears to be overloaded or broken. In 
either case, the dependency on this public web app makes caliper practically 
unusable and extremely frustrating to work with. The fact that this issue's 
priority is set to "medium" is very discouraging. Can the maintainers *please* 
prioritize publishing the web app source code and/or host the public app on a 
more robust service?

Thank you.

Original comment by mich...@eclipse.io on 15 Jul 2013 at 3:46

GoogleCodeExporter commented 9 years ago 
I've checked that logs and aside from 1 or 2 suprious 500s, there appear to 
have been no outages.  https://code.google.com/status/appengine/ shows no 
site-wide outage either.  Have you checked your connectivity?

Original comment by gak@google.com on 15 Jul 2013 at 5:08

GoogleCodeExporter commented 9 years ago 
I'm not having any connectivity issues. In fact, I've tried uploading with two 
different networks. I just tried uploading two new trials and they both failed. 
Accessing the results of previous trials is very slow. Are you not seeing any 
performance problems when you access the site?

Cheers.

Original comment by mich...@eclipse.io on 15 Jul 2013 at 5:16

GoogleCodeExporter commented 9 years ago 
Are there any errors reported in ~/.caliper/log?

Original comment by gak@google.com on 15 Jul 2013 at 5:53

GoogleCodeExporter commented 9 years ago 
Yeah, there are a bunch of errors that look like this:

com.sun.jersey.api.client.UniformInterfaceException: POST 
https://microbenchmarks.appspot.com/data/trials?key=bc60ec9c-2e0d-41d1-9bf7-88a0
0993d581 returned a response status of 500 Internal Server Error

As another data point, I just tried loading 
https://microbenchmarks.appspot.com/runs/9454582f-081f-46a6-82e8-530c976e74b4 
in my browser. It took about 30 seconds. I'm not having trouble accessing any 
other web sites.

Cheers.

Original comment by mich...@eclipse.io on 15 Jul 2013 at 5:59

GoogleCodeExporter commented 9 years ago 
That link loaded quickly for me, but I've found some errors in the server logs. 
 Looking into it.

Original comment by gak@google.com on 15 Jul 2013 at 6:08

GoogleCodeExporter commented 9 years ago 
Sometimes the web app loads quickly for me, too. I'm confident if you try that 
link several times, you'll soon see it drag.

Thanks for helping!

Original comment by mich...@eclipse.io on 15 Jul 2013 at 6:13

GoogleCodeExporter commented 9 years ago 
It looks like it's a bad interaction with the AE load balancer.  The requests 
that are 500ing are ones that cause the load balancer to start up a new 
instance and those virtually always cause the request to time out.  I got a bit 
of advice from somebody on the AE team, that we'll try and hopefully, that'll 
fix some of the issues.

Original comment by gak@google.com on 15 Jul 2013 at 6:34

GoogleCodeExporter commented 9 years ago 
We now have several resident instances.  This should hopefully alleviate a lot 
of the issues.

Original comment by gak@google.com on 15 Jul 2013 at 8:38

GoogleCodeExporter commented 9 years ago 
I was able to upload several trials starting about an hour ago, however the 
last 6 out of 8 trials have failed to upload results because of HTTP 500 errors.

Original comment by mich...@eclipse.io on 15 Jul 2013 at 10:09

GoogleCodeExporter commented 9 years ago 
I've now bumped the instance class as well.

Original comment by gak@google.com on 16 Jul 2013 at 5:12

GoogleCodeExporter commented 9 years ago 
Issue 267 has been merged into this issue.

Original comment by gak@google.com on 22 Jul 2013 at 3:54

GoogleCodeExporter commented 9 years ago 
Issue 268 has been merged into this issue.

Original comment by gak@google.com on 22 Jul 2013 at 3:37

GoogleCodeExporter commented 9 years ago 
Hi Martin,

please have a look at this code fragment:

long nanos = invokeTimeMethod(reps);
        log.notifyMeasurementEnding(measurementBuilder
            .value(Value.create(nanos, "ns"))
            .weight(reps)
            .build());

As you can see, if you want to count time of one iteration to have to use value 
(magnitude) and reps (weight)

Regards,
Kuba

Original comment by jkubryn...@gmail.com on 23 Jul 2013 at 3:21

GoogleCodeExporter commented 9 years ago 
As an update, I'm working on redistributing the workload using AE modules 
(https://developers.google.com/appengine/docs/java/modules/).  I'll get this 
tested and deployed ASAP.

Original comment by gak@google.com on 23 Jul 2013 at 3:50

GoogleCodeExporter commented 9 years ago 
This was a pain, but I think we're finally done with band-aids and should have 
a much more reliable webapp now.  Let me know if anybody is still having 
trouble.

Original comment by gak@google.com on 16 Aug 2013 at 11:42

GoogleCodeExporter commented 9 years ago 
I'm having trouble -- I hit the quota just now.  I can't log in to create an 
API key.

Original comment by David.W....@gmail.com on 19 Aug 2013 at 6:02

GoogleCodeExporter commented 9 years ago 
I'm still observing the same issue, no upload possible!!

Execution complete: 1.087h.
Collected 160 measurements from:
1 instrument(s)
1 virtual machine(s)
32 benchmark(s)
Some trials failed to upload. Consider uploading them manually.

$ cat ~/.caliper/config.properties 
# Caliper config file
# Run with --print-config to see all of the options being applied

# INSTRUMENT CONFIG
# instrument.micro.options.warmup=10s
# instrument.micro.options.timingInterval=500ms
# instrument.micro.options.reportedIntervals=7
# instrument.micro.options.maxRuntime=10s

# VM CONFIG
vm.args=-Xmx4g -Xms4g

# See the Caliper webapp to get a key so you can associate results with your 
account
results.upload.options.key=26******************************

Original comment by martin.i...@gmail.com on 19 Aug 2013 at 9:57

GoogleCodeExporter commented 9 years ago 
From the looks of the logs, all of the errors were right at the end of the 
(billing) day.  I think these new, more expensive instances gobbled up the 
quota.  Luckily, we had overhead.  I've reduced it by an instance.  We'll try 
again for another day and see if things are fixed.

Original comment by gak@google.com on 20 Aug 2013 at 12:36

GoogleCodeExporter commented 9 years ago 
When will the webapp be released so that we can run it locally ? I am from 
China, for some reasons,  the network to appspot is often blocked. So will  the 
webapp released or or will the source be opened ?

Original comment by Jafe...@gmail.com on 6 Apr 2014 at 12:22