Export to JSON/CSV

[edwardchanjw]

Does panda-panopticon can export the data show on graph as JSON / CSV? Haven look closely to how the underlying work, is that already possible to inspect the JSON data point if the graph is generated based on JSON?

Thanks for this great nodejs + metrics apps.

[freeformflow]

Hi there. Thank you for checking this out.

To be honest, I was still learning Node while I was building this app, so it is definitely a little rough around the edges, haha. The data from AWS is provided as a deeply nested (and non-time-sequenced) JSON object. In the proxy server code, I pull data points out of this AWS object and package it into a delimited string. At the time, I did this to make it easier on me to place data into the function provided by HighCharts. Unfortunately, this is irresponsible and not sustainable in the long-term, so it's something I would refactor in later versions.

What would be most helpful for you would probably be to pull data from Amazon directly. You need use the AWS module,

var AWS = require('aws-sdk');

and to fill out this configuration object to access the module.

AWS.config = {
        accessKeyId: creds[0],
        secretAccessKey: creds[1],
        region: creds[2],
        sslEnabled: true
      }

Then you create a new object from the AWS module that lets you pull data from CloudWatch.

// Create a handle to access Amazon services... Amazon CloudWatch utility.
      var cloudwatch = new AWS.CloudWatch();

Then you have to tell CloudWatch what you are asking for.

// Now we may create a parameter object, "params", that will placed in the
      // request function.  It specifies exactly what we are asking for.

      var params = {
        StartTime: String(time[0]),
        EndTime: String(time[1]),
        Period: String(time[2]),
        Namespace: 'AWS/EC2',
        MetricName: String(mini_chunks[3]),
        Statistics: ['Average'],
        Dimensions: [
          {
            Name: 'InstanceId',
            Value: String(mini_chunks[2])
          }
        ]
      };

Then you are all set to make a request to Amazon.

      // Make the request to Amazon.
      cloudwatch.getMetricStatistics(params, function(error, data){
        if (error)
        {
          // Send error response to browser.
          response.writeHead(502);
          response.write(error.message);
          response.end();
        }
        else
        {
          // The object "data" is structured and contains the response from Amazon.
          // Do stuff with it here.
        }
}

Amazon provides great documentation for their API. To pull data from CloudWatch, check out this page. You can read in-general about the SDK or click on CloudWatch and read about the function I showed above, getMetricStatistics().

I hope this helps!

[edwardchanjw]

Sure it does help a lot!

Thanks for the detailed guide, since your are so nice here, can I ask a not related question forward? For the TimeSpan we can select up to 14 days, is that correct?

Does AWS throttle the data point to certain number like 7 day of data for 5 minute resolution etc, 1 day of data point of data for 1 minute resolution etc Does a call to the getMetricStatistics get all the data contain all the 14 days's data point for selection metrics?

[edwardchanjw]

Spam haha, will be last non-constructive comment for me.

I already setup panda-panopticon and found out the data point can only up to 1440 too, because of AWS SDK limit.

If i am correct, you are using High Charts, It seem to relative easy for include Export CSV. http://www.highcharts.com/plugin-registry/single/7/Export-CSV

I would try to include the export CSV if you confirm with me :) I am new to NodeJS and relative new to graphing type software.