add profitability dashboard

[wishbone1138]

I started thinking about this one. Do we have everything we need?

Profitability = current fiat cost of coin X current coin output per day - cost power usage per day

current fiat cost of coin you grab and store with coinmarket.sh current coin output per day - we already grab balance (at least from nanopool), this would probably be the most accurate to determine how much we make per day. The pools may also have an api to their calculator. power usage per day - I calculate this for ST, I don't have a power plug but I'm guessing this would be simple for you as well.

I'm not sure if you can do complex metrics calculations between data sources in grafana, so once this is all in influxdb I believe we have everything we need. Doing these types of queries in mysql is ridiculously complex imo, or I'm just horrible at SQL (or both).

[wishbone1138]

Nanopool also has a calculator API that you can use for their coins that allows you to calculate based on whatever hashrate you feed it. It's another option but not a particularly extensible one. Maybe I'm over thinking this one but it's turning out to be tougher than I thought. I think we're going to need a local profit script that processes the data in the DB and creates a new series for profit? Thoughts?

[rodneymo]

that sounds good. We already have the coinmarket API integrated so we can make some revenue projections based on the previous days (hashrate and/or shares). Using future revenue calculation on shares is probably more accurate, but not all pools provide the number of shares.

[wishbone1138]

I think almost all pools provide a balance. I was planning on tracking that over a 24/hr period to define how much coin you were currently producing on average.. I also really like the new progress bars the pools are using now. More data to crunch to make that happen.

[wishbone1138]

I built an extremely basic profit calculator based on the last 24 hours of actual coin earnings and power usage. It wasn't really worthy of committing so I thought I'd throw it up here first to see if you had any comments. All pools should provide balance, which we're already capturing over time. It seems to be the best way to calculate an accurate idea of what our current earnings look like going forward?

        CURLOUT=`curl -sG "http://$INFLUXDB_IP:8086/query" --data-urlencode "db=rigdata" --data-urlencode \
                "q=select sum(non_negative_difference) from (select non_negative_difference(balance) from pool_acct where time > now() - 24h AND \"coin\"='"${COIN,,}"')"`
        EARNINGS=`echo $CURLOUT | jq -r '.results | .[] | .series | .[] | .values | .[] | .[1]'`

        echo $EARNINGS

        CURLOUT=`curl -sG "http://$INFLUXDB_IP:8086/query" --data-urlencode "db=rigdata" --data-urlencode \
                "q=select last(price_usd) from coinmarketcap where \"base_currency\" = '"$COIN"'"`
        COINPRICE=`echo $CURLOUT | jq -r '.results | .[] | .series | .[] | .values | .[] | .[1]'`

        echo $COINPRICE

        GROSS=`awk "BEGIN {print $EARNINGS * $COINPRICE}"`

        echo $GROSS

        CURLOUT=`curl -sG "http://$INFLUXDB_IP:8086/query" --data-urlencode "db=SmartThings" --data-urlencode \
                "q=select sum(non_negative_difference) from (select non_negative_difference(value) from energy where time > now() - 24h)"`

        ENERGY=`echo $CURLOUT | jq -r '.results | .[] | .series | .[] | .values | .[] | .[1]'`

        echo $ENERGY

        PROFIT=`awk "BEGIN {print $GROSS - ($ENERGY * 0.107)}"`

        echo $PROFIT

        IDB_PROFIT="profit,coin=$COIN profit=$PROFIT"

        #write out to DB
        curl -s -m 5 -XPOST "http://$INFLUXDB_IP:8086/write?db=rigdata&precision=s" --data-binary "$IDB_PROFIT"

Eth just had another difficulty jump. =T

[rodneymo]

Looks good. A couple of comments/updates: 1) I have modified coinmarket.sh (and renamed it colindata)to collect only data for the currencies configured in the pool_list 2) calculating past profitability looks good, but how about future profitability? 24h,7d,30d revenue. In order to do that we'll need difficulty and block time. The values I am getting from each pool seem to be different from the actual values so I have opted to use the api from whattomine.com and consolidate that data together with the coin market data in a single measurement (see coin_data.sh). Of course calculating future profits is highly inaccurate unless the diff doesn't change, but I think we need to have it an note the caveats. Thoughts?

[wishbone1138]

I like the idea of future profitability. I think these days it's hard to do that with any real accuracy, but I use whattomine for exactly that reason.

I definitely think we need to pull down the data for avg block time and difficulty so people could setup alerts for their pools if the values change dramatically. These should probably be specific to the pools because it may indicate an issue with your pool (avg block time on that pool for example). For example, ETH just took a big jump in difficulty and has dramatically effected profitability, but I had to check my profit graphs and dig into the why before I understood that.

I'm not sure about the calculations associated with predictions. I guess I'd need to see the math on how you plan on accomplishing that (unless you're just going to use whattomine's data). I'll take a look at coin_data.sh.

[rodneymo]

I have already integrated the whattomine API into the coin_data script. Basically I am merging the data from both WTM and coin market into a single series which include: price, volume, market cap in configured quote currency (see changes in config file) plus the difficulty, block reward and block time. The latter 3 values will be used to calculate future earnings (based on current difficulty)

[wishbone1138]

Awesome!! I'll take a look.

[rodneymo]

About the profitability calculation, we need to subtract the electricity costs (stored in the rig series) from the pool revenue (stormed in the pool_payments series). The problem is, in case I am mining multiple coins using different pools, there's no way to tell which (rig power costs) are related so a given pool/label. We could tackle this by querying the workers but those would have to match the names in the RIG_LIST config. An alternative is to add the pool label to the rig definition. Thoughts?

[wishbone1138]

I was thinking about that as I created my script. I think it might be more work, but perhaps we need to start breaking things out a bit more and linking them together.

Everything should key off RIGNAME. RIGNAME should be included (and match) in both POOL_LIST and RIG_LIST. That way we know exactly what rig is mining in what pool. The reason I want to key off of RIGNAME instead of poolname is because you can have one rig mine in multiple pools at the same time (claymore dual miner is a perfect example). RIGNAME should probably match worker name in order to provide more fine grained stats from pools that support that?

In addition, I think we need to start thinking about creating an environmental array. Create a ENV_LIST would also key off of RIGNAME. I'm not clear on how this would work, but we need to separate that out to support different modules somehow for environmental stats. Maybe this is futile since the various APIs for this will all be so different. I guess we may need to draw a line on what you want to support there. e.g. RIGNAME,MODULE,IP,STATSLIST rig1x01,PLUGIP,10.1.1.10,[POWER ENERGY TEMP]

We should then be able to just pull up everything we need for RIGNAME. Power, the pools it's using, the coin it's mining, and the power associated with that RIG. If a rig is doing more than just mining in one pool, the profits will be pulled from multiple pools before the power is subtracted. This would be a profit per rig idea, but they certainly could be combined into total profits as well.

[rodneymo]

Everything should key off RIGNAME. RIGNAME should be included (and match) in both POOL_LIST and RIG_LIST. That way we know exactly what rig is mining in what pool. The reason I want to key off of RIGNAME instead of poolname is because you can have one rig mine in multiple pools at the same time (claymore dual miner is a perfect example). RIGNAME should probably match worker name in order to provide more fine grained stats from pools that support that?

possible solutions we could work with: 1) add reference to (POOL) LABEL (up to 2 for rigs dual mining) to every RIG_LIST entry 2) add references to RIG_ID(s) to POOL_LIST. This will require adding an array which could get confusing 3) read worker names from pool. Currently all support pools (nano, ether and mpos) support them

option 3) is probably the most elegant, but requires more work. What do you think?

In addition, I think we need to start thinking about creating an environmental array. Create a ENV_LIST would also key off of RIGNAME. I'm not clear on how this would work, but we need to separate that out to support different modules somehow for environmental stats. Maybe this is futile since the various APIs for this will all be so different. I guess we may need to draw a line on what you want to support there. e.g. RIGNAME,MODULE,IP,STATSLIST rig1x01,PLUGIP,10.1.1.10,[POWER ENERGY TEMP]

You mentioned that before. I should have listed to you ;-) Let's make an "environment" series and the ENV_LIST as suggested. We should use the same master-script approach. So for the ENV_LIST:

<rig_id>, <plug_type>, <plug_ip>, <currenct power>, <max power>, <temp>

[rodneymo]

power monitoring moved out to env scripts. Just pushed it to the repo

[rodneymo]

I am implementing option 3. Latest commit already includes support in ethermine.

[rodneymo]

Support of option 3) added to all pool monitors

[wishbone1138]

Awesome, sorry I havent' had much time to work on it. I'll grab the latest and check it out!

[rodneymo]

slow progress on this one. Hope I can finish it in a couple of days

[rodneymo]

I am struggling with this one, mainly because there are multiple scenarios e.g. 1) all rigs mining one coin on same pool 2) all rigs mining 2 coins, one coin per pool 3) mixed scenarios: some rigs doing dual mining in 2 pools (like scenario 1), while other rigs mining only 1 coin in one of the pools

Calculating the revenue is straightforward using the data from payouts, but cost (and profitability) calculation for 2) and 3) can be a pain. For scenario 2) profitability we could make (pool_coin + pool_dcoin) payments divided per sum of rig power costs (since all rigs would be dual mining) but for scenario 3) how do we allocate power costs per pool? Maybe we could add a global var called power_allocation_ratio (e.g. 30%) So for rigs doing dual mining: 70% of power costs would be allocated to the 1st coin (and pool) and the remaining 30% to the 2nd coin (and pool). Thoughts?

One other thing, I have been trying to accurately report on list of rigs allocated each pool using the worker stats, but that's very unreliable. Often workers will still show on the workers' stats because most mining pools will report offline workers/rigs up to 24h after they have been disconnected. So I think the best option is to reference the pool (or pools in case of dual mining) in the RIG_LIST.

[wishbone1138]

I'm with you, I'm really not sure how to handle these pretty dynamic scenarios. It does't help that we have tools like claymore that do dual mining and some that do single. Of course you can run two miners at once on the same machine on different GPUs if you want as well. The best way I can think of accomplishing this is to do one line per coin per rig and add a new value for percentage total expected use on the machine (similar to what you suggest). For claymore you'd have to handle that in the claymore module for it to work right, but I think it could be done.

I would also like to suggest a "manual" plug type and either creatively using the same field or another in the same line to manually specify power. Most rigs run 24/7 and it won't be as accurate but it'll give people something. (I can open this as a new issue if you want)

RE rig per pool: Why not reference the rig name in the pool monitoring list instead?

[rodneymo]

I'm with you, I'm really not sure how to handle these pretty dynamic scenarios. It does't help that we have tools like claymore that do dual mining and some that do single. Of course you can run two miners at once on the same machine on different GPUs if you want as well. The best way I can think of accomplishing this is to do one line per coin per rig and add a new value for percentage total expected use on the machine (similar to what you suggest). For claymore you'd have to handle that in the claymore module for it to work right, but I think it could be done.

I have tried multiple alternatives to address this. The one I find the most elegant (and simple) is, in case of dual mining, to allocate a % of every rig power consumption to a specific pool e.g. if rig01 is mining ETH+SIA then there's a new config param (PWR_RATIO_DUAL_MINING=0.30), which in this case allocates 70% of the power to ETH POOL and 30% to the SIA POOL. This is also the reason why I added 2 extra fields to the RIG_LIST definition (POOL_LABEL and POOL_LABEL_DCOIN).

I would also like to suggest a "manual" plug type and either creatively using the same field or another in the same line to manually specify power. Most rigs run 24/7 and it won't be as accurate but it'll give people something. (I can open this as a new issue if you want)

I have added support for this already ;-) It's the plug-no-plug.sh script. It uses the figure specified by the MAX_POWER param in the RIG_LIST.

RE rig per pool: Why not reference the rig name in the pool monitoring list instead?

We could but that would mean adding an array to the POOL LIST, which will, in my pov, be more prone to user errors. On the other hand, I had to add the POOL_LABEL to the RIG_LIST in order to fix the power consumption (and profitability) calculations (see 1st point above).

All these fixes are in my last commit.

[rodneymo]

there were several issues with the profitability. Most of them have been fixed in last commit

[rodneymo]

implemented. profitability script and example dashboard added.