BigManDave commented 2 years ago

Good Morning,

Thanks for the work you've put in so far. I was wondering what are the chances on being able to get more info out of this integration? My configuration is solar and battery with force charge and export to grid. So I'd be hoping to get things like Production (kwh), Battery charge (kwh), battery discharge (kwh), grid usage (kwh), grid export (kwh), energy draw (kwh), battery SoC (%), I think.

I'm not a developer but I've had a poke around the API and can see that there are a lot of variables available. The list can be seen using GET https://www.foxesscloud.com/c/v1/device/variables?deviceID=[xxxxx]. However I can only see that these are usable with a POST and a date range. I think there's a limit on how many can be in each POST too.

Would love to hear your feedback. Thanks in advance.

macxq commented 2 years ago

I will investigate what the api allows us and if possible I will implements this!

BigManDave commented 2 years ago

From our email thread:

What I’ve noticed is that currently the integration uses a power generation statistic for the sensor. Looking at my system this is including battery discharge and is an ‘Inverter Generation’ value, rather than ‘Solar Generation’.

Based on the values I added into the Github request and the values available in the HA Energy Dashboard if the API allows I think we’d like the following:

Daibutt commented 2 years ago

Just a quick observation, my export only shows on feedinPower. meter2Power for me seems to always be zero.

BigManDave commented 2 years ago

Just a quick observation, my export only shows on feedinPower. meter2Power for me seems to always be zero.

Thanks for contributing. I've not got Export running yet and figured it had to be one of those! 🙂 I've edited my original post to reflect.

Daibutt commented 2 years ago

I'm haven't worked out what meter2 is. Maybe for say diesel generator input into your home AC circuit?

It's great to see @macxq developing this and happy to help in whichever why I can. I'm not proficient in HA/yaml things but have blundered my way through setting up Hive, Meross, Hildebrand Glow MQTT, zoneminder etc... so not totally inept, but still definitely a noob lol. I have two PV arrays, battery storage & Fox inverter, so this custom add is very welcome!

BigManDave commented 2 years ago

Think I might be on to something fiddling with the API.

POST https://www.foxesscloud.com/c/v0/device/history/raw -with an authorization token header RAW Body: {"deviceID":"[INVERTER_ID]","variables":["batChargePower","SoC","batDischargePower","batTemperature","gridConsumptionPower","pv1Power","pv2Power","feedinPower"],"timespan":"hour","beginDate":{"year":2021,"month":10,"day":25,"hour":12,"minute":0,"second":0}}

Changing the year, month, day, hour values as applicable will return readings for the whole hour at 5 minute intervals. I'm thinking(hoping) it might be possible to program the integration to fetch this every 5 minutes and take the last value from each of our vairables?

Daibutt commented 2 years ago

Not sure if this will help? https://www.dropbox.com/s/k3kjti6spp40xtz/Fox%20Cloud%20API%20Protocol.pdf?dl=0

I found it at this thread; https://community.home-assistant.io/t/foxess-hybrid-solar-invertor/347225

The PDF seems to suggest that you need to have a FoxCloud account first, then request access to MQTT. I have emailed FoxESS for confirmation, and how access is requested.

BigManDave commented 2 years ago

Oh wow, excellent find! MQTT will be far better than fetching from the cloud. Look forward to hearing if you were successful.

macxq commented 2 years ago

@BigManDave I have added more sensor to the integration:

HA Entity	Measurement
Generation Power	kW
Grid Consumption Power	kW
Feeding Power	kW
Bat Discharge Power	kW
Bat Charge Power	kW
Energy Generated	kWh
Grid Consumption	kWh
Feedin	kWh
Bat Charge	kWh
Bat Discharge	kWh

Please validate if this works for you!

To do this you need to fetch fetching-more-data branch executing two commands: git checkout fetching-more-data and then git pull

Please remember to use Energy measurements on HA Energy Dashboard as described here https://www.home-assistant.io/docs/energy/faq/

Looking forward for you feedback!

FYI. i'm willing to implement more sensor in the future but right now i would like to focus that those 10 are working as expected

BigManDave commented 2 years ago

@macxq Great work and many thanks! Looks mostly good to me, assuming the kwh figures are cumulative of the day? (see grid consumption below) One issue (I think) that is still present from the main release is that the Energy Generated stats are using the Foxcloud Generated variable. This stat in foxcloud is Energy in/out of the inverter so does not include solar power charging the battery, in an evening will continue to then show output from the battery, and overnight on an empty battery will actually show a negative value for powering the inverter. It therefore means it can't be used by HA Energy to properly track Solar Production.

Really appreciate your work on this, you've been amazing! :)

macxq commented 2 years ago

@BigManDave so how should i calculate this properly? Should I increases Energy Generated by Bat Charge Thx for feedback 🙇

BigManDave commented 2 years ago

@macxq I'd think you need to add together the pv1power and pv2power. HA Energy would allow us to add each pvXpower as a separate entity but I think it'd be neater if you can combine them. That way we'd clearly have an entity in HA for Solar Generation, Battery In/Out and Grid In/Out. Also I spotted that FeedIn power seems to be named Feeding power, not sure if that's a typo? :')

macxq commented 2 years ago

@BigManDave pv1power and pv2power is measurement of power not energy 😬 As it is explained in the HA FAQ https://www.home-assistant.io/docs/energy/faq/ it cannot be used by HA Energy

Feeding power is a a typo , thx for catching it!

BigManDave commented 2 years ago

@macxq Sorry if I'm missing something. When I'm looking at the API pvXpower is giving me a value in kw the same as generationpower, batchargepower, gridconsumptionpower etc. I figured you'd just be able to do the same 'conversion' to all of them?

EDIT: Oh actually I see you're grabbing the energy figures from elsewhere in the API. My bad!

Daibutt commented 2 years ago

@macxq - Awesome - working for me too :). Thank you so much. Just a minor point, I believe the power measurement units should be kW not W.

Added battery in/out to energy dashboard and that works BUT the energy balance looks screwy at the moment. Battery delta looks wrong for my data and solar generated is too low. It may be due to the issue @BigManDave mentions. I've only had the one update of the dashboard so far, so I'll let it run and see what it looks like tomorrow. Can't get my head around the numbers just yet, but may need to make a new sensor for solar generation something like (solar generated = energy generated - bat discharge + bat charge - grid consumption) - I'll have a think about that tho! Or potentially creating a Riemann sum integral sensor of pv1power +pv2power to turn kW into kWh? I think I read something about that somewhere.

So far updating every 5minutes perfectly. I haven't dared use the FoxCloud WebUI yet, I don't want to break this and lose all the fab data! So in the short term I'll be setting force charge and checking SoC on the inverter panel :)

Daibutt commented 2 years ago

https://www.home-assistant.io/integrations/integration/

Energy

An integration sensor is quite useful in energy billing scenarios since energy is generally billed in kWh and many sensors provide power in W (Watts).

If you have a sensor that provides you with power readings in Watts (uses W as unit_of_measurement), then you can use the integration sensor to track how much energy is being spent. Take the next configuration as an example:


> sensor:
>   - platform: integration
>     source: sensor.current_power
>     name: energy_spent
>     unit_prefix: k
>     round: 2

YAML

This configuration will provide you with sensor.energy_spent which will have your energy in kWh.

BigManDave commented 2 years ago

@macxq Sorry, edited my last post. Figured out you were grabbing energy from another part of the API. We might be able to get a better solar generation figure as @Daibutt suggested by generatedenergy - batdischarge + batcharge. This should be fine if we have a way to return 0 if the calculated value is negative, possibly? My battery won't ever charge from the grid whilst solar is producing, which is probably the same for almost everyone. This way generatedenergy would only be negative when force charging the battery from grid overnight, so wouldn't impact solar figures if we count that as zero. Hope that makes some sense :)

P.s. @Daibutt good spot at the W instead of KW. Your numbers are making me jealous!

Daibutt commented 2 years ago

@BigManDave Haha, yes it's end of October in South Wales, so I am a little bit surprised myself! Makes up for this last week when I averaged just under 4kWh solar generation per day.

BigManDave commented 2 years ago

@Daibutt East Yorkshire here, sunny spells aren't lasting particulalry long haha I see from your screenshot you've got your Gas consumption in HA, mind if I ask what you're using to get that measurement? ---Sorry to derail the thread!

Daibutt commented 2 years ago

Not sure if this will help? https://www.dropbox.com/s/k3kjti6spp40xtz/Fox%20Cloud%20API%20Protocol.pdf?dl=0

I found it at this thread; https://community.home-assistant.io/t/foxess-hybrid-solar-invertor/347225

The PDF seems to suggest that you need to have a FoxCloud account first, then request access to MQTT. I have emailed FoxESS for confirmation, and how access is requested.

Update on MQTT - FoxESS replied saying MQTT access is not for End-Users! They say that my "distrbutor" can get access to the data (presumably via MQTT?) and supply it to me (presumably via MQTT?) - seems a bit long winded. I have emailed my supplier/installer to see if this is indeed true. Again will update if/when I hear anything.

Daibutt commented 2 years ago

@Daibutt East Yorkshire here, sunny spells aren't lasting particulalry long haha I see from your screenshot you've got your Gas consumption in HA, mind if I ask what you're using to get that measurement? ---Sorry to derail the thread!

@BigManDave Hildebrand Glow stick https://shop.glowmarkt.com/products/glow-stick

MQTT access to data following this method (MQTT bridge to Glowmarkt MQTT broker etc..) https://brunty.me/post/hildebrand-glow-home-assistant-mqtt/

Added MQTT sensor and a utiilty meter sensor to feed into energy dashboard; `- platform: mqtt name: "Gas Import Meter3 MQTT" state_topic: "SMART/HILD/[GlowstickID]"

last_reset_topic: "SMART/HILD/[Glowstick ID]"

last_reset_value_template: homeassistant.util.dt.utc_from_timestamp(0) unit_of_measurement: 'kWh' device_class: energy unique_id: "xxxxxxxxxxxx" state_class: total_increasing icon: 'mdi:counter'

Following code added as meter value occasionally dropped to zero and when value returned gave me daily consumption in excess of 42000kWh gas!

value_template: > {% if value_json['gasMtr']['0702']['00']['00']|int(base=16) value_json['gasMtr']['0702']['03']['01']|int(base=16) / value_json['gasMtr']['0702']['03']['02']|int(base=16) < 42000 %} {{ states('sensor.gas_import_meter3_mqtt') }} {% else %} {{ value_json['gasMtr']['0702']['00']['00']|int(base=16) value_json['gasMtr']['0702']['03']['01']|int(base=16) / value_json['gasMtr']['0702']['03']['02']|int(base=16) }} {% endif %}

utility_meter: daily_gas_import2: source: sensor.gas_import_meter3_mqtt cycle: daily`

Ditto for electric import & export.

Daibutt commented 2 years ago

My battery won't ever charge from the grid whilst solar is producing, which is probably the same for almost everyone. This way generatedenergy would only be negative when force charging the battery from grid overnight, so wouldn't impact solar figures if we count that as zero. Hope that makes some sense :)

The way I was thinking was the energy values are daily totals. If your battery gains 8kWh in a day but 4kWh is from grid, you need to subtract that to get the solar energy generated number. i.e. FoxCloud numbers just report how much energy the battery has charged/discharged in kWh. I'm not sure energy generated goes negative? I will be importing tomorrow early hours as price is either 0 or < 10p/kWh, so I can see what the generated/charge/discharge numbers show first thing in morning before any significant solar generation happens. @BigManDave @macxq

BigManDave commented 2 years ago

@Daibutt I think it is HA that is accumulating the energy totals, as foxess shows 30min time slots on the data the integration is using. Therefore I was thinking we should be able to do the math before it reaches HA

Force charge has only just been enabled on my battery today, but I'm expecting my energy in the early hours of the night will be higher going into the battery from the grid than comes out of the battery for a while, therefore probably giving the inverter energy generation a negative reading for a bit. But this might be wrong too.

BigManDave commented 2 years ago

Decided to stay up to make sure my forced charge from grid was working. See negative generation power that I've been referring to. Looks like it might not even affect generation energy since it's at zero. Unless the negative subtracts from whatever value was already there, but fairly sure that isn't the case.

Daibutt commented 2 years ago

@BigManDave Yes you are right, the generated energy figure does not go negative. Energy values (kWh) are what is required for the energy dashboard, not power. The accumulated energy sensor values do seem to come from FoxCloud. I think if they were accumulated in HA they would update every sensor scan, and have multiple decimal places. They actually only increase by 0.1 increments (same as FoxCloud values) and only update when the value increases by 0.1.

@macxq I have had a think about the solar generation calculation. Either the Riemann sum of pv1power + pv2 power as mentioned earlier and/or ...

Energy balance: In - Out = Accumulation (Solar generation + Grid consumption) - (loads consumption + feed in) = (battery charge - battery discharge) or rewritten as; Solar Generation = battery charge - battery discharge - Grid consumption + loads consumption + feed in. I'm not sure if the loads consumption energy kWh is available in the API though?

macxq commented 2 years ago

@BigManDave @Daibutt these all measurements goes directly for FoxCloud :) I'm just fetching them and saving to HA. The negative Generation Power is what you can also observe in your foxesscloud instance:

Screenshot 2021-11-01 at 09 50 50

There is a lot of comment for you both i appreciate it but I'm sinking in them to have conclusions :)

DoesEnergy Generated match what you see in foxesscloud as Today Yield ?
Is foxesscloud correctly calculating Today Yield?
Is Today Yield the same as you are refering to Solar Generation?

Daibutt commented 2 years ago

@macxq Apologies for the many comments! In answer to your questions;

1) I have only had FoxESS-HA running for two complete days. One of those days Today Yield matched Energy Generated, and the other day it did not - but we changed clocks from daylight savings time, so maybe that screwed the numbers a little?

2) Possibly - but I'm not sure it is useful! See answer 3

3) Today yield is not the same as what I am callling solar generation; For Today Yield/Energy Generated, it seems FoxCloud draws the "boundary" of the PV system around the panels, batteries & inverter ;

As end users, we are likley more interested in understanding output of PV arrays, and our "boundary" is like this;

As far as I can tell, that energy figure is not available on FoxCloud (kWh daily accumulated energy from PV arrays). But the real time PV power figures are available though, as pv1power, pv2power or pvPower;

I hope this helps!

macxq commented 2 years ago

Now I get it and see it, let me think on it!

Daibutt commented 2 years ago

@macxq If it is not possible to integrate/accumulate pv1Power, pv2Power, pvPower to get solar generation in kWh, the other idea I mentioned was doing an energy balance to infer solar generated energy kWh. Add the green arrows and subtract the red ones in this picture.

Obviously this needs an energy value for loads consumption, which I think is availble on the FoxCloud overview/statistics diagram as Load Yield

This won't give exactly the same as integrating the pvPower method because of inverter losses but will be close.

macxq commented 2 years ago

@Daibutt I have implemented fetching Load and Load Power but i cannot test it because my instance does not have battery and does not report any value under this.

it is on the main branch so you need to:

git checkout main
git pull

jasperbom commented 2 years ago

Not sure if this will help? https://www.dropbox.com/s/k3kjti6spp40xtz/Fox%20Cloud%20API%20Protocol.pdf?dl=0 I found it at this thread; https://community.home-assistant.io/t/foxess-hybrid-solar-invertor/347225 The PDF seems to suggest that you need to have a FoxCloud account first, then request access to MQTT. I have emailed FoxESS for confirmation, and how access is requested.

Update on MQTT - FoxESS replied saying MQTT access is not for End-Users! They say that my "distrbutor" can get access to the data (presumably via MQTT?) and supply it to me (presumably via MQTT?) - seems a bit long winded. I have emailed my supplier/installer to see if this is indeed true. Again will update if/when I hear anything.

I got this file from my supplier and said: good luck hahaha

Daibutt commented 2 years ago

@macxq Fantastic! New entities working;

And match FoxCloud.

So now you have all values for my green and red arrows! Green: Red:

Can you make a new Solar Generation energy entity by adding and subtracting these?

macxq commented 2 years ago

@Daibutt done 👍

check out how it looks for you, because for me (i do not have battery) data looks wrong ;)

probably the is needed some logic to distinction two two types of installation

Daibutt commented 2 years ago

Hi @macxq OK, so I gave this a try, maybe I did it wrong, I'm not very good at this stuff! This is what I did;

Then restarted HA server.

I do not see the new solar entities.

Should I reclone the whole foxess-ha thing?

P.S. Thank you for working so hard on this. What you have done is awesome. I know FoxESS are relatively new, but compared to others such as Solax, getting access to their/our PV data is not great. You are making it possible :)

Daibutt commented 2 years ago

@Daibutt done 👍

check out how it looks for you, because for me (i do not have battery) data looks wrong ;)

probably the is needed some logic to distinction two two types of installation

@macxq Load kWh for me at end of today looked correct. 11.5 kWh and my last 2 month daily average is 11.3kWh.

macxq commented 2 years ago

@Daibutt I did not pushed all the things to repo 🤦‍♂️
Please do git pull one more time and the new sensor should apperen

Daibutt commented 2 years ago

@macxq Both new solar entities now working :)

Numbers make sense for me so far today; Grid Consumption + Battery Discharge = 7.9kWh Batttery Charge + Loads = 7.9 kWh (Feed In=0) Solar Generation = 0 kWh This makes sense as sun is only just hitting PV array.

Update edit: @macxq This has worked great all day. Figures seem to match up and where I would expect them to be. I do get some negative values for solar power/energy. I'm guessing that's inverter losses.

I think this is totally great. @BigManDave is this working for you too? You have batteries as well I believe?

@macxq I know you have worked super hard on this, but another important add when you get time would be battery level (SoC %). And less important but useful to understand performance of each array is pv1Power (kW) and pv2Power (kW)

Thank you so much for your work on this. I believe this has totally opened up FoxESS systems to the HA community. (And personally I am amazed as I would not even know where to start in doing this !)

BigManDave commented 2 years ago

@Daibutt Yep I'm seeing similar to you. Number mostly making sense apart from the negatives I think. Would also love to see SoC and ideally battery temp too, just to be aware of how hot it gets in the loft! 🙂 @macxq as always fantastic work here, really, really appreciate it.

Daibutt commented 2 years ago

@BigManDave @macxq Glad it's working for you too Dave. I think the way it's calculated any system losses end up in the solar generated number, as everthing else is measured. But I'll have a think about that!! BTW, if the temp in my living room in South Wales is anything to go by, your loft in Yorkshire is going to be pretty cool at the moment :)

Daibutt commented 2 years ago

@macxq @BigManDave I'm importing as we speak. Solar power is showing -ve (-170W). This seems representative of when I am charging battery. I import 5kW & battery charges at 4.8kW. I know this is more work but is there any way to feed -ve solar values to an " inverter losses" entity? This would tidy up the numbers (a bit...maybe only when there is no PV generation) and also give us an indication of inverter losses.? Edit: On 2nd thoughts this would probably just complicate things. Ignore me, just thinking aloud!

BigManDave commented 2 years ago

@macxq @Daibutt So with the negative solar values it looks like it's making HA wait until the Solar Energy is positive before it will begin plotting anything on the graph. I.e. Inverter losses overnight mean by the time the sun rises I've got -1kwh to make up before I'll get any production on the graph. I guess reporting-wise it's not actually that bad, since it means the generated solar energy reflects the actual AC energy you're getting out of the solar system. Just a shame it means less pretty graphs!

I've been working on grabbing SoC and BatTemp. Hopefully I'll have a pull request ready later for review, if that's ok :)

Daibutt commented 2 years ago

@macxq @BigManDave Fantastic work guys! Love the update. I didn't have/use HACS previously so this forced me to do a bit of reading and get it installed! The SoC & battery temp entities are welcome additions. Would it also be possible to add pv1Power & pv2Power entities please? It would be useful for understanding performance of each array.

macxq commented 2 years ago

@Daibutt yes, it wold be :) more sensors we have, more I think some some configuration around this is required to menage the haos :) not everyone will interesting in all the sensors especially if have small installation without battery

Daibutt commented 2 years ago

Hi @macxq, spotted the new beta version with PV1-4 sensors, so I installed that 👍, nice work. It looked like it was working great, values increasing as sun moved round. But when it became cloudy & solar reduced, the values "froze".

Looking at the values, could it be they are picking the max values from this chart, rather than the latest/real time (lower) values?

macxq commented 2 years ago

@Daibutt I have got the same - need to validate that

macxq commented 2 years ago

@Daibutt with the v0.9.2-beta it should be fixed

Daibutt commented 2 years ago

@macxq this seems to be working great 👍 I created Riemann sum integrals for my PV1 & PV2 and now feed these into the HA energy dashboard (instead of sensor.foxess_solar). This gets round the problem with the energy balance method sometimes throwing out negative values. I'm guessing inverter losses will now end up assigned to "Home" by the dashboard.... It also means you can assign Forecast.Solar predictions to each array and get multiple prediction lines in the dashboard.

platform: integration source: sensor.foxess_pv1_power name: pv1_sum unit_time: h round: 2

platform: integration source: sensor.foxess_pv2_power name: pv2_sum unit_time: h round: 2

Really great work. Thank you!

BigManDave commented 2 years ago

@Daibutt I've followed your comment to get those sensors set up. It looks like it didn't reset overnight. Did you figure that bit out? Thanks in advance 🙂

Daibutt commented 2 years ago

@BigManDave those sensors will carry on accumulating forever which is OK for input to HA energy dashboard, the same as the inputs for gas & import/export elec meters.

If you want a daily sum outside of the HA dashboard add them as utility meters, sourced from the ones you just set up;

utility_meter: daily_pv1_sum: source: sensor.pv1_sum cycle: daily

daily_pv2_sum: source: sensor.pv2_sum cycle: daily

This should work.

Edit: the indents didn't come out as I pasted! This has the format https://www.home-assistant.io/integrations/utility_meter/

BigManDave commented 2 years ago

@Daibutt Thanks! I hadn't confgiured them in the dashboard yet, wanted to verify the values looked right first so didn't realise it'd work without reset. Thanks for the utility meter config too :)

macxq / foxess-ha

[FEAUTRE REQUEST] More Stats - Battery, Force Charge, Export to Grid etc. #2

last_reset_topic: "SMART/HILD/[Glowstick ID]"

Following code added as meter value occasionally dropped to zero and when value returned gave me daily consumption in excess of 42000kWh gas!