Support for static images

[rtoma]

Could you support for PNG images in Graph-explorer?

I know Javascript rendered graphs is fancier than PNG, but transferring large JSON datafiles takes time & resources to create, transfer and render, resulting in worse user experience than using PNG.

Your excellent tool allows for Google-style searching, but when using too generic terms (every new user will start with such queries) results in requesting way too many graphs for way too many metrics and/or datapoints.

Because I noticed a big delay between starting a query and getting graphs on my screen I did some performance testing by firing off an API call for 42 metrics/targets, each having 1 datapoint per 10 seconds, ranging the from-param from 2 to 50 hours, ranging the output-param and executing every test 3 times.

Graph below contains the test results. The from-param is put on the X-axis. The total-time in seconds (incl. download) is put on the Y-axis.

I have used less-than-optimal hardware, so please ignore the Y-axis value and focus on the difference between JSON, SVG and PNG.

[Dieterbe]

how do you get these numbers? and what does svg mean? (graphite has no svg format? http://graphite.readthedocs.org/en/1.0/url-api.html) assuming it's mostly network transfer times, we could look into compression of data, a less verbose graphite format (i.e. raw instead of json), serverside aggregation or even client-side caching of metrics

since i will be adding more interactive features, i focus on client side rendering using flot (which uses canvas; i found it quite fast, esp. much faster than svg based solutions like d3/rickshaw). but as you can see i abstracted the rendering of the graphs through a little plugin called graphitejs: https://github.com/Dieterbe/graphitejs it supports png's, rickshaw and flot through an almost identical api, so switching to PNG's should be trivial. you'll also notice the graphitejs source code is simple to get into, i'm certainly open for pull requests that make it more efficient.

[rtoma]

Thanks for your answer. I will look into the abstraction layer and will try to contribute.

Regarding your questions:

The 1.0 url-api page seems incomplete. If you look at http://graphite.readthedocs.org/en/latest/render_api.html#data-display-formats you will find the following formats mentioned:

&format=png
&format=raw
&format=csv
&format=json
&format=svg

Note that I don't really care about svg. I simply used it as a 3rd format the test (besides json and png).

For performancetesting I used this iterator:

for n in 1 2 3 ; do
  for h in `seq 2 4 50`; do
    for o in svg png json; do
      ./query.sh $o $h
    done
  done
done

And "query.sh" is:

#!/bin/sh

format=$1
[ -z "$format" ] && format=json
from=$2
[ -z "$from" ] && from=24

output=$format-$from.out

echo -n "format=$format from=$from "

curl -w 'rc=%{http_code} time_total=%{time_total} time_starttransfer=%{time_starttransfer} size_download=%{size_download} speed_download=%{speed_download}\n' -s -o $output "<large-graphite-test-url-using-$from-and-$output>"

Running the for-loops result in output like:

format=svg from=2 rc=200 time_total=1.460 time_starttransfer=1.062 size_download=553878 speed_download=379292.000
format=png from=2 rc=200 time_total=1.322 time_starttransfer=1.144 size_download=145056 speed_download=109685.000
format=json from=2 rc=200 time_total=1.519 time_starttransfer=0.969 size_download=600604 speed_download=395282.000
format=svg from=6 rc=200 time_total=2.180 time_starttransfer=1.658 size_download=764447 speed_download=350673.000
format=png from=6 rc=200 time_total=1.003 time_starttransfer=0.880 size_download=108287 speed_download=107972.000
format=json from=6 rc=200 time_total=3.513 time_starttransfer=2.297 size_download=1781514 speed_download=507105.000
...

Which I copied to Excel to do a pivot and make the graph.

Below you'll find a graph using time_starttransfer ("first byte time") instead of time_total, thus ignoring the download size + speed.

The time Graphite needs before sending the 1st byte is most likely related to the size of the result it is preparing. Graph below shows you the download_size.

If you have any questions, please let me know!

[Dieterbe]

https://github.com/Dieterbe/graphitejs/issues/1 here, I made this ticket in case someone wants to make graphitejs faster. looking at your graphs, it seems the slowdown is mostly caused by the extra time graphite needs to generate the output, and not to actually transfer the data. that's interesting. have you checked mem/cpu load of your graphite machine when you do the json requests?

[rtoma]

Graphite machine was at 100% CPU during the handling of the requests. As I mentioned, the hardware is below par.

When requesting data for 50 hours for 42 metrics, each having 1 point per 10 seconds, results in 50 * 3600 * 42 * 10/60 = 1.26M datapoints. A PNG will not really get bigger when requesting more metrics or more history, but JSON/RAW/CSV will. Maybe add a summarize function to downsample the data and get as much datapoints as the final graph has resolution on the X-axis? That way clientside rendering scales no matter the number of metrics or history selected. Just a thought.