search

breuner / elbencho

A distributed storage benchmark for file systems, object stores & block devices with support for GPUs
GNU General Public License v3.0
161 stars 24 forks source link

InfluxDB or other time series database out option #62

Closed philcanman closed 1 week ago

philcanman commented 1 month ago

I am wanting to capture the data in a time series database so that I can visualize the data in Grafana.

breuner commented 1 month ago

hi @philcanman , would the elbencho --livecsv /path/to/some/output.csv option work for you in combination with csv file import option of influx as described here? https://docs.influxdata.com/influxdb/cloud/write-data/developer-tools/csv/#example-write-command

philcanman commented 1 month ago

The live CSV file in its current form will not work with the CSV tool built into Influx as I understand it. Is there a way to output the live CSV data to the terminal with each update on a new line with fixed columns? Then I can pipe that data off to a script for real time metrics.

Would also be good with a timestamps in the livecsv file. Then I could write something to import the livecsv data into influx.

breuner commented 1 month ago

Ok, no problem to add an option to have the CSV lines as console output. (The side effect will be that the human-readable end result summary will no longer get printed to the console, because the csv reader into which you pipe the output wouldn't like that non-CSV tail - but I assume that's not a problem here.)

When you say "fixed columns", are you referring to fixed column width or am I getting this wrong?

Adding timestamps will also be no problem.

philcanman commented 1 month ago

Awesome glad to hear that. When I say fixed columns I mean the header or column value never changes. For instance if the third column is MB/s it is always MB/s kinda deal. That and a time stamp should make writing a script to import the data a breeze. Thanks!

breuner commented 1 week ago

Hi @philcanman , the new elbencho version v3.0.19 (released a few hours ago) supports a new special value "stdout" (for elbencho --livecsv stdout) to print the stats in csv format to the console. The MiB/s column in integer format is sometimes not fine-grained enough and thus the values of this column might get decimal places at some point in the future, but the column unit MiB/s will stay the same, so I assume this still counts as "fixed columns".

Also, the live csv stats now have a new column for "ISO Date" timestamps (as you requested), so the columns did actually change in this release, but hopefully for the last time :-)

If you create any script or guide of which you think it would be good to get hosted in the elbencho repo (under the contrib/ subdir), so that others can also benefit from it, then of course please just let me know.

philcanman commented 1 week ago

Thanks a ton for this. I have it working now using graphite see repo: https://github.com/philcanman/elbencho_graphite