INFLUXDB: Measurement or tag, and we need an agreement on "schema"

[wishbone1138]

[REMOVED AND CHANGED MY MIND BELOW]

[wishbone1138]

One other thing, I noticed in your pool-nanopool.sh you were creating a measurement for each API/measurement. We could you have a measurement per pool type and then use a different tag per API call (and data) so that even if the timestamp overlaps influx will still treat is as a different record.

Here's the schema I suggest: MEASUREMENT: pool_ TAGS ( I suggest to use the list for the POOL_LIST array) plus a report tag (e.g. stats, payments, shares, etc... This will be used to distinguish data from different APIs for the same pool: pool_type,crypto,label,,api_token,wallet_addr,report FIELDS TIMESTAMP

Quick comment on using TAGs for API calls. I've experimented with it and if you add new key fields then it will put a lot of empty data per series (it appears), even if the tag is different. I think that "Show series" is a bit misleading because if you do a call from the database from a measurement it will include all of the key values used in ANY tag for that measurement. For example, even though I have two different series with different tags it still seems to hold locations for those key fields that are in all series in that measurement. This also shows up in the Grafana UI as well. I don't know how these are actually being stored, but this is the output of what I"m talking about:

rig_stats,coin=ZEC,gpu_id=0,miner=EWBF,rig_id=KINKO_NAVI
rig_stats,coin=eth,dcoin=sia,gpu_id=0,miner=claymore,rig_id=Kinko1

time                coin dcoin gpu_fan gpu_hashrate gpu_hashrate_dcoin gpu_id gpu_power gpu_rej_shares gpu_shares gpu_shares_dcoin gpu_status gpu_temp miner rig_id
----                ---- ----- ------- ------------ ------------------ ------ --------- -------------- ---------- ---------------- ---------- -------- ----- ------
1505611272370728495 ZEC        0       561                             0      204       0              94                          2          72       EWBF  KINKO_NAVI

I like the idea of using the module name in the measurement since each module might be storing different data for this reason. I think making a new series for every API call might result in a lot of empty fields being stored vs making more measurements. I could be wrong about this?

Here's an idea.

pool_$API$_$POOLNAME$
TAGS- coin, (dynamic per module really)
stat_$API$_$MINER$
TAGS- coin, dcoin(for claymore only?), gpu_id, rig_id (dyanmic per module)

I don't think we need a separate miner stat measurement for "rig" outside of the individual gpu stats? I know miners (like claymore) can report their own averages and righ hash rate, but if we have all the gpus do we really need that? If we don't have individual gpu stats then we just write it as gpu_id=0. If you think I'm leaving out major data here then no problem, lets make a statSYSTEM$MINER$ and a statGPU$MINER$

That begs the question where do we write "static" data on a rig? Which leads me to what I've seen in other places. Perhaps we create a "measurement" that is just a single time series. Give it a special name like _rig_info_$RIGNAME$ and a particular time that never changes. That way when we want to update that particular measurement we just use the same time to update it. That could house things like expected hash rates, max temperatures, gpu configurations, etc... Updated with update-rig-info.sh. NOTE: this will need a special retention policy assigned to it. These single time series might be:

_rig_info_$RIGNAME$
_gpu_info_$RIGNAME$

I think this is going to make things difficult for creating stats across different miners or pools, but I haven't tried yet. Sorry for all the TEXT, let me know which way you want to go and we'll make it happen. More series with empty fields and less measurements or more measurements specific to their applications and more concise field lists.

[rodneymo]

I like the idea of using the module name in the measurement since each module might be storing different data for this reason. I think making a new series for every API call might result in a lot of empty fields being stored vs making more measurements. I could be wrong about this?

Here's an idea.

pool$API$$POOLNAME$ TAGS- coin, (dynamic per module really) stat$API$$MINER$ TAGS- coin, dcoin(for claymore only?), gpu_id, rig_id (dyanmic per module)

I see what you mean. It sounds like a good principle. I'd like to propose some minor tweaks:

pool$POOLNAME$$REPORT TAGS- coin, label (we don't need API_TOKEN nor WALLET_ADDR. The label can be made different if those differ)

Swapped $API and $POOLNAME vars to keep some kind of hierarchy and also in line with the respective pool script name. Also renamed the API to REPORT, reason being that some pools return stats and payouts (in an array) in the same response (see my MPOS script) so we can use either the API or another label to specify the data

I don't think we need a separate miner stat measurement for "rig" outside of the individual gpu stats? I know miners (like claymore) can report their own averages and righ hash rate, but if we have all the gpus do we really need that? If we don't have individual gpu stats then we just write it as gpu_id=0. If you think I'm leaving out major data here then no problem, lets make a statSYSTEM$MINER$ and a statGPU$MINER$

I's still like to keep them so I can track things like mining time, installed GPUs, rig specific power usage. Should we keep the naming convention aligned with your proposal above? So for miners we'd have:

minersystem$miner minergpu$miner

That begs the question where do we write "static" data on a rig? Which leads me to what I've seen in other places. Perhaps we create a "measurement" that is just a single time series. Give it a special name like _riginfo$RIGNAME$ and a particular time that never changes. That way when we want to update that particular measurement we just use the same time to update it. That could house things like expected hash rates, max temperatures, gpu configurations, etc... Updated with update-rig-info.sh. NOTE: this will need a special retention policy assigned to it. These single time series might be:

_riginfo$RIGNAME$ _gpuinfo$RIGNAME$

I think this is going to make things difficult for creating stats across different miners or pools, but I haven't tried yet. Sorry for all the TEXT, let me know which way you want to go and we'll make it happen. More series with empty fields and less measurements or more measurements specific to their applications and more concise field lists.

Sounds good. Given that we'll have a single data point per rig do we need rig or GPU specific measurements? I am just being cautious because of the number of series (and performance implications of it). If you agree we could use 2:

MEASUREMENT: _rig_info TAGS: rig_name, miner TIME: 0 (FIXED)

MEASUREMENT: _gpu_info TAGS: rig_id, gpu_id, miner TIME: 0 (FIXED)

Also it makes it easier to maintain dashboards. Otherwise we'd have to do multiple joins when using rig/gpu info on a multiple GPU/RIG dashboard

[wishbone1138]

All looks great to me, I agree on all points. I like it!

The only question I have is how to choose fixed time fields for every entry we want to add in the gpu and rig "static" measurements. We'll need some sort of convention for that to update those fields. I don't think they can all be the same?

[wishbone1138]

Also, we need a standard naming convention for tags and fields. rig_id, gpu_id, hashrate, gpu_hashrate, etc... Whatever you want, just let me know.

[rodneymo]

sounds good, but how do we deal we some of the pool specific items e.g. I am currently integrating the dwarfpool API and it has both hashrate and calculated hashrate?

[wishbone1138]

Just from a naming convention standpoint? There will probably be variance in modules (for both miner and pool) that are different. I think that's just going to be best effort there. If the project takes off enough for people to really start contributing we just have to make sure that any contributions match up when there is overlap to do so.

I'd say most fields could start with the generic term and get specific.

hashrate hashrate_current hashrate_average (if you wanted to use the average from a pool or miner) hashrate_reported (ethermine.org does this one depending on the miner you're using). etc.

If it's completely new metric, then I guess the module author gets to make it up?

[rodneymo]

I will propose something for the naming convention. I'll use the claymore miner as basis for it and try to make those field names as generic as possible.

What do you mean with

The only question I have is how to choose fixed time fields for every entry we want to add in the gpu and rig "static" measurements. We'll need some sort of convention for that to update those fields. I don't think they can all be the same?

[wishbone1138]

Sorry. Im just not clear on how the update/overwrite work. When we update the rig and gpu info we will want to replace what is there. That means that our update command must also match whatever we used for time I order to overwrite those entries. We won't know what that time is without a convention (or pulling from the DB first). I wonder if we can write zero time entries? I haven't tried. Make sense?

[wishbone1138]

The time entries could possibly be all the same if the tags are different.

[rodneymo]

Right. Influx uses the measurement name + tags + timestamp as "index" so if those match you'll override the fields. I have tested this. Do you wish to track different static data (configurations) based on time?

[rodneymo]

or we can explicitly delete the existing data if the timestamp is known e.g. 0

[wishbone1138]

nope, nothing beyond what we've already discussed. Overwriting is fine. I suppose if a rig line disappears from the configuration file we'll probably need the logic to remove it as well.

Does "0" work in the time field?

[rodneymo]

it does. The problem is it's possible to combine data from different measurements with a single query. Have you tested it? See here https://community.influxdata.com/t/solved-writing-a-tickscript-to-aggregate-multiple-measurements-into-one/1630

[wishbone1138]

Hmmm.. Looks like there's no way https://docs.influxdata.com/influxdb/v1.3/troubleshooting/frequently-asked-questions/#how-do-i-query-data-across-measurements

Older versions had join and it looks like they removed it.

Maybe that's why I had the miner and pool as tags. Compound queries are possible but I don't think that will let you pull from more than one measurement. So two options?

1. Go back to using tags for pool name and miner 2., Setup a new measurement for system wide calculations. Like total eth hashrate, etc. Might need to do this for profit calculations anyway? Would be a new script. Not a huge deal but more hits on the DB and more storage usage.

[rodneymo]

I think we'll need a combination of both, It's probably the best is if we start with the graphs we would like to have and work our way down to the (simple) measurements and system measurements (combining data from different sources) we need to have. I'll create a wiki page for that I think it's easier to update it

https://github.com/rodneymo/rig-monitor/wiki/List-of-reports-and-panes-in-Grafana-dashboards

[wishbone1138]

I don't think I can edit the wiki, I think you need to make me a collaborator in order to do that.

[rodneymo]

done

[wishbone1138]

Speaking more to schema again. I noticed you're using gpu_specs as a tag. My spec has spaces in it and if you include the quotes it will store the actual quotes in influxdb along with the tag.
SEE: https://docs.influxdata.com/influxdb/v1.3/write_protocols/line_protocol_tutorial/#quoting Two options I can see.

1. replaces spaces with _
2. move to field (seems like a bad place to put something that doesn't change)

My vote is #1, what do you think?

[rodneymo]

I didn't include the quotes. I escaped the "space" chars but apparently grafana quotes those fields on the (json) output. Using the command line client they are not shown. Is that a problem? I noticed that grafana does remove them

(I also notice the gnu_id in the same measurement (miner_gpu_claymore) is also in quotes. It's probably a null char put by awk.)

[wishbone1138]

interesting. I'll take another look at it, maybe it was an assumption I made based on what I saw from the jq/awk output in my ewbf script.

[wishbone1138]

Gotcha, I missed the gsubs you were doing in the awk file.

I'm still playing around with the idea of moving a lot of this data to single entry time series. Perhaps I'm trying to over complicate it. The case below might be a good reason to keep all the data in the individual series/measurements.

coin/dcoin might need to be on miner_gpu_claymore? You can switch it and it will be reflected on miner_system_claymore, but how do you correlate the data in the gpu stats to only show the new coin you're mining?

[wishbone1138]

At this point we're doing two different setups. For miner we're including the miner in the measurement, for pool we're only including it in a tag. In addition, I noticed you're including miner=claymore in your miner_gpu_claymore measurement. That might be redundant, but I guess that depends if you want to use the data in a table or graph it could be useful now that I think about it. I dont' think there's anyway to do that with measurement name dynamically. Just circling back to this subject to see if you've given it any more thought. The current setup hasn't prevented me from doing anything I want yet, but I don't have very many systems.

As a side note, if you do a table as s time series (does not work for format as "table") you can display multiple measurements in a single table. (if you didn't know that already). I'm using this to display the hashrate of each miner in a single table.

[rodneymo]

I noticed it today while finalizing the sgmmier script. I think using tags provides more flexibility if we want to run miner-aggregated queries in grafana and is more future proof. It's also better in order to minimise the number of series. Do you agree?

As a side note, if you do a table as s time series (does not work for format as "table") you can display multiple measurements in a single table. (if you didn't know that already). I'm using this to display the hashrate of each miner in a single table.

I didn't. Thanks for the tip.

[wishbone1138]

I think the flexible move is definitely to use tags. If we move away from using it in measurements then we just have to put up with potential extra blank fields, but I believe influx is pretty good at compression so it probably does not matter. (ie dcoin is meaningless to anything but claymore). So I guess my vote is tags and we've come full circle =)

[rodneymo]

all miners updated. miner TAG added to missing one, plus measurement names renamed to: _minersystem and _minergpu

[wishbone1138]

sounds good!

[rodneymo]

@wishbone1138 we have implemented this in all miner scripts. Can we close it?

[wishbone1138]

agreed. closed