Storage metrics usage for billing

[gshif]

• Add tests and necessary libraries for storage reports of metrics usage for billing

[gshif]

• [x] Create a library function to be able to parse prometheus metrics returned by the /metrics/usage endpoint.
  • The handling of results of /metrics endpoint for various services is covered in the different issue https://github.com/influxdata/Litmus/issues/111
  • This library will only handle /metrics/usage for a storage component.
• [ ] Use test data with different data types: float (default), integer, string, boolean and timestamp.
• [ ] Define tests for metrics/usage endpoint. First it will be tested on a cluster with a single storage/node pods. The same set of tests would be eventually run on a cluster with more than one storage/kafka node, so tests should be written in a such a way so they should be runnable on singe/multi node storage/kafka deployments.
  • [ ] Write a point(s) to different orgs/buckets
    • [x] Write point to same organization, different buckets. Storage's cache age is set to 1 minute:
      • Create organization org with two buckets, bucket1 and bucket2.
      • Write 6 points to org->bucket1 and 6 points to org->bucket2:

        h2o_feet,location=coyote_creek water_level=8.120,level\ description="between 6 and 9 feet" 
        1439856000
        h2o_feet,location=coyote_creek water_level=8.005,level\ description="between 6 and 9 feet" 
        1439856010
        h2o_feet,location=coyote_creek water_level=7.887,level\ description="between 6 and 9 feet" 
        1439856020
        h2o_feet,location=coyote_creek water_level=7.762,level\ description="between 6 and 9 feet" 
        1439857030
        h2o_feet,location=coyote_creek water_level=7.635,level\ description="between 6 and 9 feet" 
        1439857040
        h2o_feet,location=coyote_creek water_level=7.500,level\ description="between 6 and 9 feet" 
        1439857050

      • Wait for 1 minute for tsm file to be created. (Currently we do not have any tools that would let us verify data was written to a disk).
      • Get storage usage metrics per org/bucket and verify they are correct. The same set of points would be written to a database using 1.7 influxdb and using influx_inspect tool the size of the data stored on a disk would be calculated. The assumption is that 1.7 gives us the correct results. After first 6 points were written to an influxdb 1.7 and then written to a disk, run:

        influx_inspect dumptsm -all 000000001-000000001.tsm 

        and that is what returned by the above command (even though I am seeing that times are wrong):

           Summary:
  File: 000000001-000000001.tsm
  Time Range: 1970-01-01T00:00:01.439856Z - 1970-01-01T00:00:01.43985705Z
  Duration: 1.05µs   Series: 2   File Size: 306

Index:

  Pos   Min Time            Max Time            Ofs Size    Key             Field
  1 1970-01-01T00:00:01.439856Z 1970-01-01T00:00:01.43985705Z   5   53  h2o_feet,location=coyote_creek  level description
  2 1970-01-01T00:00:01.439856Z 1970-01-01T00:00:01.43985705Z   58  78  h2o_feet,location=coyote_creek  water_level
Blocks:
  Blk   Chk     Ofs Len Type    Min Time            Points  Enc [T/V]   Len [T/V]
  0 3385718256  5   49  string  1970-01-01T00:00:01.439856Z 6   s8b/snpy    17/30
  1 2769078932  58  74  float64 1970-01-01T00:00:01.439856Z 6   s8b/gor     17/55

Statistics
  Blocks:
    Total: 2 Size: 131 Min: 49 Max: 74 Avg: 65
  Index:
    Total: 2 Size: 162
  Points:
    Total: 12
  Encoding:
    Timestamp:  none: 0 (0%)    s8b: 2 (100%) 
    Float:  none: 0 (0%)    gor: 1 (50%) 
    String:     none: 0 (0%)    snpy: 1 (50%) 
  Compression:
    Per block: 10.92 bytes/point
    Total: 25.50 bytes/point

• [ ] Delete point(s) from different orgs/buckets. Deletion of the points can be accomplished in a few ways:
  • [ ] Delete bucket using delete bucket REST API.
  • [ ] Delete organization, that would delete all of the buckets that belong to the org that is being deleted using delete organization REST API
  • [ ] enforcing of retention policies. If bucket points time past RP, points would be deleted.
    • [ ] Reducing the size of the data stored on disk using compactions. Since there are many levels of compactions, need to be able to test all. (TBD)