taskcluster / ec2-manager

Mozilla Public License 2.0
2 stars 14 forks source link

Store the EBS totals in Postgres and fetch these totals in an API call #45

Closed SamanthaYu closed 6 years ago

SamanthaYu commented 6 years ago

Store the EBS totals in Postgres

We decided to store the EBS totals in Postgres. In the table ebsusage, the columns are region, volumetype, state, totalcount, totalgb, and touched with the primary key a combination of region, volumetype, and state. When reportEbsUsage() gets called, it inserts the totals into their respective columns. If there's already an existing record with that primary key, then that record's totalcount and totalgb gets replaced with the reported totalcount and totalgb respectively, and touched is set to the current timestamp.

Fetch these totals in an API call

The internal API call, ebsUsage(), returns the entire ebsusage table. In other words, it returns a list of objects in the form:

{
   region: string,
   volumetype: string,
   state: string,
   totalcount: string,
   totalgb: string,
   touched: timestamp,
}

Updated documentation for AMI API call

Because the API calls for EBS usage and AMI usage are similar, I updated the AMI API call's for consistency with the EBS API call and for clarity of what values get returned:

/**
 * List AMIs and their usage by returning a list of objects in the form:
 * {
 *   region: string,
 *   volumetype: string,
 *   lastused: timestamp,
 * }
 */

@rafvasq and @auni53 What do you think? Is this comment accurate?

Other comments

@rstiyer and I decided to combine our two parts into this PR. Since @rstiyer worked on the database side and I worked on the EBS API call, the tests for each of our parts would rely on each other so it would be hard to separate them completely.

@djmitche and @imbstack Can you review this PR when you get the chance?

djmitche commented 6 years ago

I think -- please correct me -- the current state is that this is waiting for an update from rachel to overwrite the DB in a single transaction. With that change (and with the drop-database call restored in test_housekeeping.js) this should be good to go.

rstiyer commented 6 years ago

I added a check within the function reportEbsUsage(rowData) that does nothing if rowData is an empty array, but I wanted to confirm if everyone thinks this makes sense. The alternative would be to assume that the empty list is correct, so just clear the table and insert nothing.

djmitche commented 6 years ago

I think if there's no row data then we're provisioning in 0 regions, which doesn't seem right. So I think it's fine as it is.