search

TACC / tacc_stats

TACC Stats is an automated resource-usage monitoring and analysis package.
GNU Lesser General Public License v2.1
41 stars 15 forks source link

Port pickler from previous versions into timescale workflow #20

Closed stephenlienharrell closed 1 year ago

stephenlienharrell commented 1 year ago

The pickler has been a (in)famous part of tacc stats, and while we move to a new pipeline architecture, this is will be important for historical data going forward.

stephenlienharrell commented 1 year ago

As well, we will need to be archiving raw stats files, which will be the primary data source, then the pickler can use those files.

A library should be created to be able to load the raw stats archives in memory, search for the relevant stats files and pull them out without pulling them all out. Then the pickler can use this library to pull the data it needs and convert it.

The pickler should use the existing product-metrics code as to mirror what we do on the website.

stephenlienharrell commented 1 year ago

Not going to do this.