No "Current Consumption" via Fronius EMS

Nachtkapp commented 3 years ago

Hi there. This is Sven. I used the dracoventions/TWCManager for year now with a hacky connection to my Fronius inverter. I just migrated to your fork and wanted to use the Fronius EMS.

The connection to the datalogger of the Fronius inverter works fine and I get the "Current Generation" value on the web interface on port 8080. However the value for "Current Consumption" on the same dashboard stays at 0.

I looked into the Fronius.py file and saw that in the Function "getMeterData" the URL is using the DeviceId=0. This does not work for my environment as my Inverter has the Id=1. As in the function "getInverterData" the value is set to DeviceID=1 in the URL, I guess this is maybe is wrong and should be DeviceId=1 in general.

I changed the DeviceId to 1 in the url but the dashboard still shows 0 for "Current Consumption". I added some logging to the function "update" and saw that "self.consumedW" now gets the correct value. I was not able to trace it any further a my Python knowledge is very limited.

Can you help here to find out what is going wrong? I am happy to test or provide more information if needed.

Thank you in advance.

Regards

Sven

Nachtkapp commented 3 years ago

What I forgot to mention: I tried it with version 1.1.8 and 1.2.0 with the same result.

Nachtkapp commented 3 years ago

Hi there and happy new year

I was able to solve the problems in the Fronius.py:

As mentioned above the MeterData URL has the wrong DeviceID
The consumedW value that the field PowerReal_P_Sum delivers has negative values for the power value. Therefore it needs to be negated (*-1); values below 0 are discarded. That was the reason the value was always 0 in the dashboard
When no power is generated, the used field ["Body"]["Data"]["PAC"]["Value"] is not available in the JSON response. Thefore I added an if contdition so the values for generatedW and voltage are set to 0 if the field is missing in the JSON data

As I was not able to add a branch, here are the changes of my changed Fronius.py:

/lib/TWCManager/EMS/Fronius.py

     def getMeterData(self):
         url = "http://" + self.serverIP + ":" + self.serverPort
         url = url + "/solar_api/v1/GetMeterRealtimeData.cgi?Scope=Device&DeviceId=1"

         return self.getInverterValue(url)

     def update(self):
....
             inverterData = self.getInverterData()
             if inverterData:
                 try:
                     if "PAC" in inverterData["Body"]["Data"]:
                         self.generatedW = inverterData["Body"]["Data"]["PAC"]["Value"]
                     else:
                         self.generatedW = 0
                 except (KeyError, TypeError) as e:
                     self.debugLog(4, "Exception during parsing Inveter Data (PAC)")
                     self.debugLog(10, e)

                 try:
                     if "PAC" in inverterData["Body"]["Data"]:
                         self.voltage = inverterData["Body"]["Data"]["UAC"]["Value"]
                     else:
                         self.voltage = 0
                 except (KeyError, TypeError) as e:
                     self.debugLog(4, "Exception during parsing Inveter Data (UAC)")
                     self.debugLog(10, e)

             meterData = self.getMeterData()
             if meterData:
                 try:
                     self.consumedW = float(meterData["Body"]["Data"]["PowerReal_P_Sum"]) * -1
                 except (KeyError, TypeError) as e:
                     self.debugLog(
                         4, "Exception during parsing Meter Data (Consumption)"
....

Hope that helps. If you have any questions, let me know.

Regards Sven

ngardiner commented 3 years ago

Hey there,

Apologies for the late response, it has been a busy period.

Thanks for the patch, I will take a look however I suspect more investigation is required, as I use a Fronius inverter and meter myself and I am quite confident changing the parsing will break my setup, so I will need to identify what differs between your and my setup and adjust for it.

Are you able to see any version number in your web interface that I can compare to mine? Given none of these things (negative generation number or meter offset) exist in my setup, it may be a difference in installation. For example, in my setting consumption is always a positive value and I do not see the PAC value node missing in the JSON response when I am not generating energy, however the check that you have added is universally useful and I will add that in.

Perhaps we make inverting the consumption value and changing the meter index call configurable options for now until we work out how they could be detected?

ngardiner commented 3 years ago

In fact, having thought about it some more, most of these can be automated.

In terms of inverted consumption values, we can check for a negative number and invert it if so. In the case of the node missing from the JSON output I will merge your check which will take care of that, it's really just the meter index that I need to find a way to identify - I'll see if there's any API documentation that might help there. Ultimately what I'd like is to enumerate all meter devices and then add their consumption together, in case we come across a complex multi-meter installation at some point (assuming that logic is correct, it's hard to comprehend what a multi-meter install might look like without getting some feedback).

Nachtkapp commented 3 years ago

Thank you very much for your reply. I checked the versions of my environment:

Inverter: Fronius Symo 8.2-3-M SW Version: 3.14.1-10 (Update to 3.16.7-1 available but not applied jet)

Get API version http://MyIP/solar_api/GetAPIVersion.cgi

{
    "APIVersion" : 1,
    "BaseURL" : "/solar_api/v1/",
    "CompatibilityRange" : "1.5-16"
}

I found the API documentation here: http://files.ushiro.cat/fronius_solar_api_v1.pdf

There is a scope parameter in the URL. If you change this from Scope=Device to Scope=System, you can skip the DeviceID. Here is what the documentation says about this parameter: Parameter: Scope Type: String Range/Values/Pattern: ”Device” or ”System” Description: Query specific device(s) or whole system (uses collection ”CumulationInverterData”)

This is the output when I call the inverter URL with the scope set to system: http://MyIP/solar_api/v1/GetInverterRealtimeData.cgi?Scope=System&DataCollection=CommonInverterData

Output:

{
   "Body" : {
      "Data" : {
         "DAY_ENERGY" : {
            "Unit" : "Wh",
            "Values" : {
               "1" : 882
            }
         },
         "PAC" : {
            "Unit" : "W",
            "Values" : {
               "1" : 85
            }
         },
         "TOTAL_ENERGY" : {
            "Unit" : "Wh",
            "Values" : {
               "1" : 43217404
            }
         },
         "YEAR_ENERGY" : {
            "Unit" : "Wh",
            "Values" : {
               "1" : 4224
            }
         }
      }
   },
   "Head" : {
      "RequestArguments" : {
         "DeviceClass" : "Inverter",
         "Scope" : "System"
      },
      "Status" : {
         "Code" : 0,
         "Reason" : "",
         "UserMessage" : ""
      },
      "Timestamp" : "2021-01-03T14:27:50+01:00"
   }
}

The same works for the meter URL: http://MyIP/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System

Output:

{
   "Body" : {
      "Data" : {
         "1" : {
            "Details" : {
               "Manufacturer" : "Fronius",
               "Model" : "S0 Meter at inverter 1",
               "Serial" : "n.a."
            },
            "Enable" : 1,
            "EnergyReal_WAC_Minus_Relative" : 62,
            "Meter_Location_Current" : 1,
            "PowerReal_P_Sum" : -750.62546469808296,
            "TimeStamp" : 1609680450,
            "Visible" : 1
         }
      }
   },
   "Head" : {
      "RequestArguments" : {
         "DeviceClass" : "Meter",
         "Scope" : "System"
      },
      "Status" : {
         "Code" : 0,
         "Reason" : "",
         "UserMessage" : ""
      },
      "Timestamp" : "2021-01-03T14:27:31+01:00"
   }
}

I suggest a third URL. That's the one I used in the old script and has all the values we need in one URL: http://MyIP/solar_api/v1/GetPowerFlowRealtimeData.fcgi

Output:

{
   "Body" : {
      "Data" : {
         "Inverters" : {
            "1" : {
               "DT" : 123,
               "E_Day" : 901,
               "E_Total" : 43217404,
               "E_Year" : 4242.7001953125,
               "P" : 76
            }
         },
         "Site" : {
            "E_Day" : 901,
            "E_Total" : 43217404,
            "E_Year" : 4242.7001953125,
            "Meter_Location" : "load",
            "Mode" : "vague-meter",
            "P_Akku" : null,
            "P_Grid" : 706.35353904777924,
            "P_Load" : -782.35353904777924,
            "P_PV" : 76,
            "rel_Autonomy" : 9.7142782906691227,
            "rel_SelfConsumption" : 100
         },
         "Version" : "12"
      }
   },
   "Head" : {
      "RequestArguments" : {},
      "Status" : {
         "Code" : 0,
         "Reason" : "",
         "UserMessage" : ""
      },
      "Timestamp" : "2021-01-03T14:39:29+01:00"
   }
}

Generation: Body\Data\Site\P_Load Consumptioin: Body\Data\Site\P_PV

What do you think?

mattiasclaesson commented 3 years ago

I have a fronius inverter also and the consumption values seem off in TWCManager, maybe its not updating that often?

When installing the primary smart meter you have to choose if its located in the feed-in position or in the consumption path. Its also possible to install more smart meters assigned to external generators and individual consumers.

I would not negate the consumption value without investigating more.

mattiasclaesson commented 3 years ago

Symo 17.5-3-M fro31331.upd Datamanager 3.16.7-1

{
    "APIVersion" : 1,
    "BaseURL" : "/solar_api/v1/",
    "CompatibilityRange" : "1.5-18"
}

http://myip/solar_api/v1/GetInverterRealtimeData.cgi?Scope=System&DataCollection=CommonInverterData

{
    "Body": {
        "Data": {
            "DAY_ENERGY": {
                "Unit": "Wh",
                "Values": {
                    "1": 6718
                }
            },
            "PAC": {
                "Unit": "W",
                "Values": {
                    "1": 0
                }
            },
            "TOTAL_ENERGY": {
                "Unit": "Wh",
                "Values": {
                    "1": 96479
                }
            },
            "YEAR_ENERGY": {
                "Unit": "Wh",
                "Values": {
                    "1": 20798
                }
            }
        }
    },
    "Head": {
        "RequestArguments": {
            "DeviceClass": "Inverter",
            "Scope": "System"
        },
        "Status": {
            "Code": 0,
            "Reason": "",
            "UserMessage": ""
        },
        "Timestamp": "2021-01-04T18:06:09+01:00"
    }
}

http://myip/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System

{
    "Body": {
        "Data": {
            "0": {
                "Current_AC_Phase_1": 4.5609999999999999,
                "Current_AC_Phase_2": 5.9509999999999996,
                "Current_AC_Phase_3": 3.782,
                "Current_AC_Sum": 14.294,
                "Details": {
                    "Manufacturer": "Fronius",
                    "Model": "Smart Meter TS 65A-3",
                    "Serial": "601489885"
                },
                "Enable": 1,
                "EnergyReactive_VArAC_Sum_Consumed": 44278,
                "EnergyReactive_VArAC_Sum_Produced": 179535,
                "EnergyReal_WAC_Minus_Absolute": 11916,
                "EnergyReal_WAC_Plus_Absolute": 2426829,
                "EnergyReal_WAC_Sum_Consumed": 2426829,
                "EnergyReal_WAC_Sum_Produced": 11916,
                "Frequency_Phase_Average": 50,
                "Meter_Location_Current": 0,
                "PowerApparent_S_Phase_1": 1010.7,
                "PowerApparent_S_Phase_2": 1323.7,
                "PowerApparent_S_Phase_3": 835.20000000000005,
                "PowerApparent_S_Sum": 3169.6999999999998,
                "PowerFactor_Phase_1": 0.98699999999999999,
                "PowerFactor_Phase_2": 0.999,
                "PowerFactor_Phase_3": -0.999,
                "PowerFactor_Sum": 0.998,
                "PowerReactive_Q_Phase_1": -160.19999999999999,
                "PowerReactive_Q_Phase_2": -63.600000000000001,
                "PowerReactive_Q_Phase_3": 39.100000000000001,
                "PowerReactive_Q_Sum": -184.69999999999999,
                "PowerReal_P_Phase_1": 997.89999999999998,
                "PowerReal_P_Phase_2": 1322.2,
                "PowerReal_P_Phase_3": 834.29999999999995,
                "PowerReal_P_Sum": 3154.5,
                "TimeStamp": 1609781386,
                "Visible": 1,
                "Voltage_AC_PhaseToPhase_12": 387.39999999999998,
                "Voltage_AC_PhaseToPhase_23": 383.5,
                "Voltage_AC_PhaseToPhase_31": 387.60000000000002,
                "Voltage_AC_Phase_1": 221.59999999999999,
                "Voltage_AC_Phase_2": 225.80000000000001,
                "Voltage_AC_Phase_3": 221.59999999999999
            }
        }
    },
    "Head": {
        "RequestArguments": {
            "DeviceClass": "Meter",
            "Scope": "System"
        },
        "Status": {
            "Code": 0,
            "Reason": "",
            "UserMessage": ""
        },
        "Timestamp": "2021-01-04T18:29:46+01:00"
    }
}

http://myip/solar_api/v1/GetPowerFlowRealtimeData.fcgi

{
    "Body": {
        "Data": {
            "Inverters": {
                "1": {
                    "DT": 114,
                    "E_Day": 6718,
                    "E_Total": 96479.0078125,
                    "E_Year": 20798,
                    "P": 0
                }
            },
            "Site": {
                "E_Day": 6718,
                "E_Total": 96479.0078125,
                "E_Year": 20798,
                "Meter_Location": "grid",
                "Mode": "meter",
                "P_Akku": null,
                "P_Grid": 1330.5999999999999,
                "P_Load": -1330.5999999999999,
                "P_PV": null,
                "rel_Autonomy": 0,
                "rel_SelfConsumption": null
            },
            "Version": "12"
        }
    },
    "Head": {
        "RequestArguments": {},
        "Status": {
            "Code": 0,
            "Reason": "",
            "UserMessage": ""
        },
        "Timestamp": "2021-01-04T18:32:05+01:00"
    }
}

Nachtkapp commented 3 years ago

Hi Mattias thank you for your feedback. I agree that the negation of the consumption value should not be done at all times as it seems to deliver correct values in some scenariaos. With an if condition a negation could be made if the returned value is negative.

I did not unterstand what you mean with "the consumption values seem off in TWCManager"? As far as I can see there is no switch in the script code to turn reading the consumtion on or off.

If the returned value for consumption is smaller than 0, a 0 is returned to TWCManager. With the changes named above, I now get the correct consumption values in my environment. What value is shown is your TWCManager dashboard on port 8080 for "Current Consumption"?

Your json output above shows the values of the inverter URL. Can you call the meter URL (http://MyIP/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System) and post your output for this?

Regards

Sven

mattiasclaesson commented 3 years ago

Hi Mattias thank you for your feedback. I agree that the negation of the consumption value should not be done at all times as it seems to deliver correct values in some scenariaos. With an if condition a negation could be made if the returned value is negative.

I did not unterstand what you mean with "the consumption values seem off in TWCManager"? As far as I can see there is no switch in the script code to turn reading the consumtion on or off.

Sorry, I was unclear. I meant that the value shown in TWCManager was "off" thus not the same as the values shown in the fronius web solar web and local webserver. If the returned value for consumption is smaller than 0, a 0 is returned to TWCManager. With the changes named above, I now get the correct consumption values in my environment. What value is shown is your TWCManager dashboard on port 8080 for "Current Consumption"?

Looked now and it say Current Consumption | 3214.40 | watts | 13.39 | amps The first value may actually be correct, but there is no way I'm pulling 13Amps! The current consumption seem to be stuck at one value in TWCManager.

Your json output above shows the values of the inverter URL. Can you call the meter URL (http://MyIP/solar_api/v1/GetMeterRealtimeData.cgi?Scope=System) and post your output for this?

Updated now (had to pause for eating...)

Nachtkapp commented 3 years ago

Hi Mattias

the amps value for current consumption is just a calculated value from the current consumption watts. There it depends on what version of the script you are using and what your settings in the config.json file are. Before v1.2.0 240V and 1 phase was hard coded. Starting with v1.2.0 you can define the number of phases and your default voltage. These are the default values: "defaultVoltage": 240, "numberOfPhases": 1, The obove seems to be your setting: 3214.40W / 240V = 13.39A For me its 230V and three phases. So if you have not 240V from your grid or more than one phase, adjust the settings in the config.json. As far as I clould see, the values gets updated every 10 seconds. Therefore the value should be up to date.

Hope that helps.

I am curios: What does this URL show you as json result (please replace MyIP with your inverter IP): http://MyIP/solar_api/v1/GetPowerFlowRealtimeData.fcgi

Regards Sven

mattiasclaesson commented 3 years ago

Hi Sven!

the amps value for current consumption is just a calculated value from the current consumption watts. There it depends on what version of the script you are using and what your settings in the config.json file are. Before v1.2.0 240V and 1 phase was hard coded. Starting with v1.2.0 you can define the number of phases and your default voltage. These are the default values: "defaultVoltage": 240, "numberOfPhases": 1, The obove seems to be your setting: 3214.40W / 240V = 13.39A For me its 230V and three phases. So if you have not 240V from your grid or more than one phase, adjust the settings in the config.json.

I'm on v1.2.0, but I was under the impression that either the fornius API (or perhaps the TWC would report the voltages in realtime). Atleast the charge voltages is visible in the twcmanager web for me. The wording in the configuration does say its used for older TWCs, not for calculation of the consumption current. "# Newer TWCs can report actual voltages. For older TWCs, you can specify"

As far as I clould see, the values gets updated every 10 seconds. Therefore the value should be up to date.

Hope that helps.

I am curios: What does this URL show you as json result (please replace MyIP with your inverter IP): http://MyIP/solar_api/v1/GetPowerFlowRealtimeData.fcgi

Yeah, I posted it above, but here is a fresh response.

{
    "Body": {
        "Data": {
            "Inverters": {
                "1": {
                    "DT": 114,
                    "E_Day": 4724,
                    "E_Total": 101203.0078125,
                    "E_Year": 25522,
                    "P": 0
                }
            },
            "Site": {
                "E_Day": 4724,
                "E_Total": 101203.0078125,
                "E_Year": 25522,
                "Meter_Location": "grid",
                "Mode": "meter",
                "P_Akku": null,
                "P_Grid": 3050,
                "P_Load": -3050,
                "P_PV": null,
                "rel_Autonomy": 0,
                "rel_SelfConsumption": null
            },
            "Version": "12"
        }
    },
    "Head": {
        "RequestArguments": {},
        "Status": {
            "Code": 0,
            "Reason": "",
            "UserMessage": ""
        },
        "Timestamp": "2021-01-05T22:34:49+01:00"
    }
}

Nachtkapp commented 3 years ago

Hi Mattias

sorry, shomehow I overlooked the previously posted json values.

You wrote: I'm on v1.2.0, but I was under the impression that either the fornius API (or perhaps the TWC would report the voltages in realtime). Atleast the charge voltages is visible in the twcmanager web for me.

This is correct. What I wrote above is correct for the "current consumption" values for power and voltage coming from the Fronius inverter. The charger related values for voltage and power come from the TWC slaves seen by the master. As you mentioned that you are not sure that the amperage value is correct or does not get updated; with be formula provieded above, your values are consistent: 3214.40W / 240V = 13.39A

ngardiner commented 3 years ago

Great, thanks all for the feedback. I am also getting detail from that endpoint

/solar_api/v1/GetPowerFlowRealtimeData.fcgi

{
   "Body" : {
      "Data" : {
         "Inverters" : {
            "1" : {
               "DT" : 76,
               "E_Day" : 7689,
               "E_Total" : 21288800,
               "E_Year" : 130906.203125,
               "P" : 4060
            }
         },
         "Site" : {
            "E_Day" : 7689,
            "E_Total" : 21288800,
            "E_Year" : 130906.203125,
            "Meter_Location" : "grid",
            "Mode" : "meter",
            "P_Akku" : null,
            "P_Grid" : -2945.71,
            "P_Load" : -1114.29,
            "P_PV" : 4060,
            "rel_Autonomy" : 100,
            "rel_SelfConsumption" : 27.445566502463052
         },
         "Version" : "12"
      }
   },
   "Head" : {
      "RequestArguments" : {},
      "Status" : {
         "Code" : 0,
         "Reason" : "",
         "UserMessage" : ""
      },
      "Timestamp" : "2021-01-07T10:47:55+11:00"
   }
}

/solar_api/GetAPIVersion.cgi

{
    "APIVersion" : 1,
    "BaseURL" : "/solar_api/v1/",
    "CompatibilityRange" : "1.5-16"
}

Given the output we see above, it appears that:

Generation: Site > P_PV
Consumption: Site > P_Load (negated)

For the /solar_api/v1/GetPowerFlowRealtimeData.fcgi endpoint.

I'm going to test this in my environment, and if the values match those shown in the inverter, I'll push the changes out for testing.

ngardiner commented 3 years ago

Just pushed an update to v1.2.1 (d4f5fff) using the system context API endpoints to capture generation and consumption - could I please get you to test and let me know what you see? Per the screenshots below, it's aligned with my inverter values.

ngardiner commented 3 years ago

I've just added a few extra checks in commit d602e81 to deal with situations where there's no generation values and / or voltage is not reported by the Inverter. The latest version is stable for me, please let me know how it works for your install too

Nachtkapp commented 3 years ago

Hi, thanks for the quick implementation. I installed it yesterday, but due to the time (11 PM) the values were not updated (as intented). Now I was able to test and it works:

As you can see it's wintertime in Germany and not very much sun outside. But the values fit. From my side you can close this issue. Or do you also want to implement the other URL /solar_api/v1/GetPowerFlowRealtimeData.fcgi? The only value that is missing here is the production voltage. Where is that used by the way?

mikey4321 commented 3 years ago

It works well for me (5kw 3phase inverter and a 3phase meter).

My DeviceIDs are same as yours (1 for the inverter and 0 for the meter).

ngardiner commented 3 years ago

Or do you also want to implement the other URL /solar_api/v1/GetPowerFlowRealtimeData.fcgi?

We're doing that now :) The values are now fetched using that endpoint at the System context

The only value that is missing here is the production voltage. Where is that used by the way?

I'm able to fetch the production voltage from the Inverter API endpoint, so this is retained. At this point it's not used for any calculations, however we do capture this value for all inverters reporting voltage as an option for improving the accuracy of power calculations where it makes sense.

ngardiner commented 3 years ago

Thanks @Nachtkapp and @mikey4321 for the feedback on the revised module, it's much appreciated

mattiasclaesson commented 3 years ago

Tested and works well.

Keep in mind that there is a background task that periodically fetches production and consumption values. That task is default only running as part of the green policy.

What is the best way to configure the task to run 24/7 to report consumption?

ngardiner commented 3 years ago

In the Web UI, go to Settings and change Non-Scheduled power action to "Track Green Energy", and you'll get 24/7 tracking

mattiasclaesson commented 3 years ago

In the Web UI, go to Settings and change Non-Scheduled power action to "Track Green Energy", and you'll get 24/7 tracking

Perfect! Thanks. Did find that setting, but apparently chosed the wrong option.

Nachtkapp commented 3 years ago

I totally forgot to close this issue. Thanks again for the help.

ngardiner / TWCManager

No "Current Consumption" via Fronius EMS #190