Different format to old Fox cloud data?

[Daibutt]

I think I am seeing an issue with older Fox cloud data;

I was pulling data from Sept 2021 to upload to PVOutput and received an error when uploading. It looks like PV data (generation) and Export figures are incorrect. Load (Consumption) and Import seem about right. Using the Google colab method this is what I get for 7 Sep 2021;

20210907,567661,10900,531200,06:46,,,,,0,0,199,0,10299,,,,

Which PVOutput did not like as values in position 2 and 4 were too large.

Manipulating the Fox data report in Excel gets me these values which seem more likely;

BUT.... if I run the Google colab method on recent data (I tried 30 Aug 2023) , the values come out more or less the same as data that Foxcloud & HA are reporting (so not an issue with the code).

Something at the back of my head is remembering the format of the Fox data report changing at some point, but I can't be sure, or pin down a date!

I posted this as an issue but it is more just for info if anybody is using it on an older Fox system. It's not impacting me as I'm getting round it by downloading Fox data reports, manipulating them in Excel and creating CSV data to upload.

[TonyM1958]

Thanks. I've seen a couple of issues with PV Output not loading my historic data. It definitely does not like it if you export more than you generate - so this will be an issue if/when anyone tries to charge their battery from the grid and then export that power back.

I got around this by just editing the CSV values that it objected to before I uploaded them.

The Grid Import, Export and House consumption are taken directly from the Fox Stats data - so you see the same things if you look at a particular day in EnergyStats, for example. The PV Power and CT2 values are more complex as Fox does not expose this data in it's Stats, so what the code does is to get the raw_data values for pvPower and meterPower2. It then integrates these to approximate the PV generation. It also adds 8% to CT2 to allow for inverter losses as this is measured at the output of the inverter and we want to upload the power generated by the solar panels.

I've checked these values against my HA data and they are generally within a few%.

The code you have is a general toolkit for using the Fox cloud API and can do a lot more than I have included in the Colab notebook so far. If you look at the README file for this site, the various API calls you can use are listed and explained. You can also have a look in the tests folder - there is a notebook there that I run through to test various API calls when I make code changes.

For example, if there is a day you are interested in looking at, add a cell and try running this:

f.report_data('month', '2021-09-07', v=f.report_vars)

This will throw out the JSON that the Fox Cloud API delivers with all the variables they provide (plus a summary that I add). The report will have a value for each variable and for each day of the month.

(use the month report to look at the data for days - because the values by hour that Fox report for a day do not add up to same as you get when you ask for the day of the month. So far as I can, they loose data when they try to break down energy by hour).

You can also do this:

f.raw_data('day', '2021-09-07', v=f.power_vars)

And get the power values for the same day, again plus a summary You can put individual variables or a complete list (f.raw_vars) or pick items from the list...

And now you are really deep into the rabbit hole...

f.get_battery()

[TonyM1958]

BTW - looking at your data, there appears to be a spike in the PV Power 06:40. These are the max power and time from the Fox cloud:

531200,06:46

So, Fox is saying you generated 531.2 kW at 06.46 in the monning...

I've added a max_power setting to the next release that gets checked for raw_data with kW units - it will print a warning with the time and value if the power exceeds 100kW and set the value to 0 so the integration should still get the right result.

The value here makes me wonder if Fox was loading a signed int from the inverter as unsigned? We have seen -ve PV Power on occassions, as I understand it. The code ignores -ve PV power values as well.

[TonyM1958]

I've uploaded v0.3.5 that includes a check on the max solar power generation. Do you want to check again to see if you get a sensible value now for 2012-09-07? It should generate some warnings so we can see how many bad data points there are.

To run it, go to Runtime menu in Colab and select Restart Runtime and re-run cell 1. It should sign on with v0.3.5. Then set start_date to '2021-09-07 and generate the upload and see if the pv generation data looks more sensible?

I'd be interested to see what output it produces as I don't know how many other people might get this same problem.

[Daibutt]

Hi Tony, Thanks for this I won't be able to get on to this today but will give it a try Sunday evening or Monday. Cheers

On Fri, 1 Sept 2023 at 20:52, TonyM1958 @.***> wrote:

I've uploaded v0.3.5 that includes a check on the max solar power generation. Do you want to check again to see if you get a sensible value now for 2012-09-07? It should generate some warnings so we can see how many bad data points there are.

To run it, go to Runtime menu in Colab and select Restart Runtime and re-run cell 1. It should sign on with v0.3.5. Then set start_date to '2021-09-07 and generate the upload and see if the pv generation data looks more sensible?

I'd be interested to see what output it produces as I don't know how many other people might get this same problem.

— Reply to this email directly, view it on GitHub https://github.com/TonyM1958/FoxESS-Cloud/issues/1#issuecomment-1703253243, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQQRDKMHQCXVUQK5C4RVMJLXYI4H3ANCNFSM6AAAAAA4HWKRBA . You are receiving this because you authored the thread.Message ID: @.***>

[Daibutt]

Hi @TonyM1958, good news and bad news....! I quickly ran the v0.3.5 yesterday on some of my earlier data (June 2021) and it seemed to work. Originally (with v0.3.4) all the PV data I was missing from 16/6/21 to 07/09/21 was incorrect, as shown here;

But the first running of 0.3.5 fixed that for June 👍 , so I reran it for June-Sept and broke Google colab somehow with various errors.

Just taken another look today, now on v0.3.6 and get this error trying to get those 84 values from 16/6/21 - 7/9/21....

Code in input cell matches your example (as far as I can see), any thoughts?

But I can confirm that the first time I ran it it worked and values looked sensible. I didn't get chance to check against values I dumped from Fox data report as I ran it again and came up with the error.

[TonyM1958]

OK, I'm away for a few days so can't easily update the code.

The error is my fault because i added the code to ignore very large power values. Its complex but I temporarily zero'd out the data with values over 50000 in your data. The code then looks for the minimum value and picks up the temporary zero and then after scanning the values goes back to work out the time the data was zero. But the zero value does not exist anywhere in the original list - hence the error zero not in list when getting the index to get the time...

The bizarre thing is that means even when it's dark, your PV power was never zero over a complete 24 hour period.. so I think I really need to get a dump of the data that's causing this problem to see what is really going on and how best to handle it.

Can I get back to you on Wednesday?

[Daibutt]

Absolutely not a problem Tony. I have now got my older data (pre my manual records/HA) in PVOutput anyway, using the FoxCloud data reports & a bit of Excel manipulation/integration. I was just letting you know in case anyone else runs across this problem. Cheers

On Sun, 3 Sept 2023 at 16:41, TonyM1958 @.***> wrote:

OK, I'm away for a few days so can't easily update the code. The error is my fault- its complex but I temporarily zero'd out the data with values over 50000 in your data. The code then looks further minimum value and picks up zero and then after scanning the values goes back to work out the time the data was zero. But this value does not exist anywhere in the original list - hence the error zero not in list...

The bizarre thing is that means even when it's dark, your PV power was never zero over a complete 24 hour period.. so I think I really need to get a dump of the data that's causing this problem to see what is really going on and how best to handle it.

Can I get back to you on Wednesday?

— Reply to this email directly, view it on GitHub https://github.com/TonyM1958/FoxESS-Cloud/issues/1#issuecomment-1704338032, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQQRDKJYNQ2ZVD3BIC3FVG3XYSQLRANCNFSM6AAAAAA4HWKRBA . You are receiving this because you authored the thread.Message ID: @.***>

[TonyM1958]

I've updated and added some code to v0.3.7.

If you open your notebook in Colab after you haven't used it for a few hours, hopefully it will sign on with v0.3.7 when you run the first cell. If not, you may need to go to Runtime, Disconnect and delete. Close the notebook and then re-open it in Colab. It should load a fresh runtime and download the latest code.

Once you have this running, please can you add a line to a new, empty cell at the bottom of the notebook:

f.get_raw('day', d='2021-06-16', v='pvPower', save='pvPower')

I've made some changes, so hopefully this won't now crash getting the min or max values.

You can change the date if you wish. Each time it runs, it print some data in the output window and writes a file named pvPower_rawday\<date>.json. This contains the raw data that the Fox cloud is returning for the day. You can download this file and attach it and I can then see the 5 minute sample PV power data for that day - 288 samples per file.

It can take a few minutes before Colab rescans the folder and it appears in the file viewer (click the folder in the LH margin to show/hide the file viewer). Feel free to have a look at the downloaded file in a text editor to check you are happy with the data and there is no personal data before attaching.

Also note, if you disconnect or leave the runtime idle for a period, it will probably delete the files if you didn't download them - but you can run the cell again to recreate the data any time.

[TonyM1958]

ps - it would also help to know what you think your generation actually might have been for the day you provide data for!

[Daibutt]

Thanks @TonyM1958 Followed your instructions and it worked great. This is for 16/06/21. Had to rename the JSON to TXT as upload wouldn't allow it. pvPower_raw_day_20210616.txt

This is what I think my data is for that day...... (Fox data report into Excel, then P=IV for each array and summing them to get total PV for both arrays, evaluating import or export power based on meter power, then integrating all power values based on time between readings).

Date | Sum of TotPV | Sum of Load kW | Sum of Exp | Sum of Imp 16/06/2021 | 25.7 | 13.0 | 8.3 | 0.7

Apols for the formatting, it's a paste from a Pivot Table. Let me know if you want it with more granularity as I have the raw data like this;

[TonyM1958]

Thanks for this, glad it worked.

I was really hoping there would be something sensible in the data and I could apply a relatively simple transform to make sense of it but, my first impression is that it doesn't make much sense at all!

Here's what it looks like if you pull the json data into Excel and graph pvPower values against time through the day:

At midnight, the value is 6.9, rising to about 150 at 4am, dropping to 6.9 again between 09:40 and 11:40 (possible inverter down time due to a problem?), then going to just over 500 until about 9pm, dropping to 300 and then tailing off around 10.30pm.

To me, this looks like it might be PV1 + PV2 Voltage. Its the middle of summer, so the days are quite long, I suppose. Does this make any sense to you?

What do you see if you look at the same day via foxesscloud.com?

Judging by what you said above, you ignored pvPower and calculated power from PV1 and PV2 Voltage and Current?

Not quite sure where to go with this from here - can you work out how many dates are affected and when the power data I'm extracting starts to make sense?

Really sorry - seems to raise more questions that it answers - let me know how far you are happy to go looking into this...

[Daibutt]

@TonyM1958 Thanks for looking at this, and no, it doesn't make any sense to me either !!

I haven't got data from the API before, just the Fox data download. So no PvPower in that just the array I & V data.

I am OK with the data I have, and the calculated import/export data ties in with my supplier (+/-), so the cloud data and my integration method seem ok.

Don't spend any more time on this Tony! Appreciate all your hard work getting us access to the data. I raised the issue just in case anyone else was seeing similar issues, and suggesting there is a way around it using the cloud data.

There is still someting nagging me about the Foxcloud data reports though ! A few months after my install I wanted to start recording data and used the Fox data reports to get this. I vaguely remember some extra fields appearing in the CSV at some point after that & I had to change my XLS accordingly. I then got HA working with macxq integration so ditched the XLS analysis. So maybe Fox changed format of the data report and the API output changed too?

[TonyM1958]

To me, this looks like it might be PV1 + PV2 Voltage. Its the middle of summer, so the days are quite long, I suppose. Does this make any sense to you?

Don't know if you saw this edit I made shortly after posting? Fox report this data as pvPower with a unit of kW but as the value at midnight is 6.9 and PV1 Voltage = 3.6 and PV2 Voltage = 3.3, it looks like their data is actually PV1 Voltage + PV2 Voltage. Still doesn't make sense but may point to a historic database error of some sort.

I don't think this data is fixable for you, and you are happy with what you have - but I am concerned about how many others might come across a similar problem so would like to frame this in the code so it returns good data where possible.

The code is pretty flexible... here are a couple of cells I'd like you to run if that's OK with you?

for d in f.date_list('2021-06-16', '2021-09-30'):
    result = f.get_raw('day', d, v=['pvPower'], summary=2)
    max = result[0].get('max')
    if max is not None and max > 0 and max < 100:
        print(f"{d} is valid, max pvPower = {max}kW")
        break

This looks at your pvPower data from 2021-06-16 to 2021-09-30 and checks the maximum value in the day. If it's voltage, the max will be over 100v but power generation should never be over 100kW - so it should show the first day when the power data is sensible.

for d in f.date_list('2021-06-16', '2021-09-30'):
    result = f.get_raw('day', d, v=['pv1Volt', 'pv1Current', 'pv2Volt', 'pv2Current', 'pvPower'], summary=1)
    wh = 0.0
    for i,data in enumerate(result[0]['data']):
        wh += result[0]['data'][i]['value'] * result[1]['data'][i]['value'] / 12   # pv1Volt * pv1Current / 12
        wh += result[2]['data'][i]['value'] * result[3]['data'][i]['value'] / 12   # pv2Volt * pv2Current / 12
    print(f"{d}: generation from IV = {round(wh/1000,3)}kWh, generation from pvPower = {result[4]['kwh']}kWh")

This cell replicates what you did and downloads the PV1/PV2 Voltage and Current data and integrates this to produce the pv Power and compares this with what the Fox data says. Just want to check that this agrees with your calculated data.

You can change the dates in this cell to check your data for any date if you want.

I've added a check of max generation in the next version of the code that will highlight data that PV Output won't upload anyway.

Depending on the result of the above, I may just check the dates people put in and not generate data before a cut off date (start of 2022?) unless someone explicitly over-rides that date...

Then I think we've taken this as far as we can?

[Daibutt]

Hi @TonyM1958, This is great! It does look like the values are PV1+PV2 voltages as you say. The drop 09:40-11:40 makes sense now, installers had to come back as they couldn't complete the PV setup on the first day. I'd forgotten that!

Ran the 2 cells above and it looks like data becomes normal on 8th Sept 2021;

And values tie up with my Excel integration values.

Yep I agree can't take this any further, and I have all my data pre 8th Sept 2021 so that's all good. Thanks for all your work on this 👍

[TonyM1958]

OK - I think a check if pvPower is > 100 kW should work. A bad value getting through if max total PV voltage in a day is less than 100v should be a very rare occurence (panels covered in snow?) but a max PV Power of more than 100kW in a day is clearly wrong for the size of solar arrays / inverters we are talking about.

So, the next version will flag the date as invalid if pvPower is more than 100kW.