search

jesserizzo / envoy_reader

MIT License
37 stars 26 forks source link

Envoy S Production data is not read / read incorrectly when current transformers are not enabled/installed [bug] #37

Closed lnlp closed 3 years ago

lnlp commented 3 years ago

Problem

Envoy S Production data is not read when not using current transformers which renders reported Production data useless.

This is because envoy_reader assumes that when `envoy/production.json' exists that this is the location to read the Production data from. But that is an incorrect assumption because it is only valid when current transformers (CT) are enabled/installed.

Situation

I use Envoy S but without any CT's installed. For a 3-phase installation I would need to buy 4 additional CT's while consumption and production data is already provided by my smart electricity meter (via DSMR P1 port). I therefore did not install any CT's.

When using Envoy S without having CT's enabled then production data in envoy/production.json is NOT updated.

Actual Production that gets updated is available via envoy/api/v1/production however. But envoyreader does not read it. It only reads envoy/production.json. As a result the reported Production values are either 0 or some bogus value. (I have observed bogus values for 'production' and 'consumption' in range between 0 and 2)._

An Envoy S (and probably Envoy IQ) without CT's enabled should be handled as 'endpoint_type' "P" instead of 'endpoint_type' "PC".

Details about my Envoy: Enphase Envoy S 'metered' Multiphase (probably identical to Envoy IQ). Part number: 800-00554-r03, Software Version: R4.10.35 (6ed292) Electrical system: 3-phase 230V.

Solution

Handle Envoy S (and Envoy IQ) without CT's as 'endpoint_type' "P" instead of 'endpoint_type' "PC".

There are 2 options to implement this:

By default envoy_reader should automatically determine whether CT's are enabled and act accordingly. By providing an optional ct_enabled parameter the user can override it for testing purposes.

This is what is shown on the envoy/home page when CT's are not enabled:

Meters Not Enabled

Let me know if you need more information or help.

gtdiehl commented 3 years ago

@lnlp Would it be possible for you to post a copy of your /production.json?

The code currently does a check to see if the attributes of production and consumption exist in the web page. I'm thinking maybe another check of the attribute eim. If these are zero than poll data from the other production page.

Also would need to check this behavior on newer firmware such as D5.0.49 (77afa8) to see what happens when metering is not setup and if the /production.json page returns valid data or not.

rct commented 3 years ago

Currently there are two problems:

  1. Envoy reader makes an assumption depending on the type of Envoy whether CT metering is being used. CT metering can be turned off even on newer Envoys if the CTs aren't installed or are installed incorrectly. So far, I haven't seen a flag from the Envoy API that indicates whether it is using the inverter measurements or the CTs, but maybe I haven't looked deeply enough. It does show in the HTML output, but scraping should be avoided.

  2. Envoy reader doesn't have the flexibility to allow the user to override envoy reader's assumption and specify which data items or sources you'd like to track in Home Assistant.

Current production Watts (wNow) from either inverters, CTs, or both

I believe there should be the option to get either or both production wNow values for the inverters and the eim (CT) structures. The inverters and the CTs don’t report the same values and Envoy reader can make an incorrect assumption about whether CT metering is being used. Ideally, the user should be able to specify and track either or both from Home Assistant.

I'm not sure what would be a good way to name production Watts data elements to allow referencing either or both of them in Home Assistant's configuration. Perhaps for compatibility there should be three available entities for the wNow values in Home Assistant:

Having these three elements would allow the user to override Envoy reader's choice if necessary as well as track both on systems with CTs if the user chooses that.

(*) Note: I don't know that production[0].type is always inverters and production[1].type is always eim.

Obtaining WH totals/accumulation:

From what I have seen on the Envoy IQ it only accumulates WH totals (today and 7 day) in the eim production structure of production.json WHEN CT metering is enabled.

When CT metering is off you have to look at /api/v1/production for the WH accumulation (today, 7 day) totals. This isn't currently accessible via Home Assistant, since at least for my configuration it is polling production.json and using the `eim`` structure which never change, which is the issue being reported here.

In other words, without CTs to use the Envoy's accumulation of WH Today and WH Past 7 days, you'd need to be able to get Envoy Reader to poll /api/v1/production.

An alternative to polling /api/v1/production is to use production.json type == inverter whLifeTime and compute the delta locally, either within envoy reader, or Home Assistant. This would allow access to finer grained energy production data for uses such as graphing 15 minute intervals.

@gtdiehl Here is my current production.json from my IQ D5.0.49 (77afa8) with CT metering turned off. /api/v1/production

{
  "production": [
    {
      "type": "inverters",
      "activeCount": 27,
      "readingTime": 1604240164,
      "wNow": 485,
      "whLifetime": 12877526
    },
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "production",
      "readingTime": 1604240284,
      "wNow": 543.128,
      "whLifetime": 26225.198,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 9500.168,
      "vahLifetime": 34969.369,
      "rmsCurrent": 6.616,
      "rmsVoltage": 251.275,
      "reactPwr": 571.381,
      "apprntPwr": 831.587,
      "pwrFactor": 0.65,
      "whToday": 32.198,
      "whLastSevenDays": 32.198,
      "vahToday": 41.369,
      "varhLeadToday": 0,
      "varhLagToday": 17.168
    }
  ],
  "consumption": [
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "total-consumption",
      "readingTime": 1604240284,
      "wNow": 543.128,
      "whLifetime": 26225.198,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 9500.168,
      "vahLifetime": 0,
      "rmsCurrent": 6.343,
      "rmsVoltage": 251.642,
      "reactPwr": -571.381,
      "apprntPwr": 1596.116,
      "pwrFactor": 0.34,
      "whToday": 26225.198,
      "whLastSevenDays": 26225.198,
      "vahToday": 0,
      "varhLeadToday": 0,
      "varhLagToday": 9500.168
    },
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "net-consumption",
      "readingTime": 1604240284,
      "wNow": -0,
      "whLifetime": 0,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 0,
      "vahLifetime": 0,
      "rmsCurrent": 0.273,
      "rmsVoltage": 252.009,
      "reactPwr": -0,
      "apprntPwr": 34.56,
      "pwrFactor": 0,
      "whToday": 0,
      "whLastSevenDays": 0,
      "vahToday": 0,
      "varhLeadToday": 0,
      "varhLagToday": 0
    }
  ],
  "storage": [
    {
      "type": "acb",
      "activeCount": 0,
      "readingTime": 0,
      "wNow": 0,
      "whNow": 0,
      "state": "idle"
    }
  ]
}

Here is my current http://envoy.local/api/v1/production

{
  "wattHoursToday": 867,
  "wattHoursSevenDays": 75431,
  "wattHoursLifetime": 12877300,
  "wattsNow": 529
}
gtdiehl commented 3 years ago

I have read through everything yet but I want to point out that when polling uses the /api/v1/production page the envoy appears to only update it every 15mins. Some may view this as a draw back. Like I said I haven't looked through all of the comments yet.

@rct If you have code changes could you post it to GitHub for a possible merge?

rct commented 3 years ago

@gtdiehl - I think the only cases where you'd want to use /api/v1/production are:

  1. The envoy doesn't support production.json
  2. The envoy is configured without CT metering and the user wants WH totals for today and 7 days which aren't available in production.json.

I don't have code changes that are worth submitting. I need to understand how the Home Assistant configuration interface interacts with envoy_reader to configure it.

gtdiehl commented 3 years ago

@rct Those are good points. 😃

I'll look through the code and see what it might take to implement. Or atleast get reporting working for all attributes for each hardware platform.

lnlp commented 3 years ago

@lnlp Would it be possible for you to post a copy of your /production.json?

I will make a capture tomorrow when it's light and the system is actualually producing.

I have read through everything yet but I want to point out that when polling uses the /api/v1/production page the envoy appears to only update it every 15mins. Some may view this as a draw back.

This is not my experience. I see information in /api/v1/procuction and /api/v1/production/inverters being updated every 5 minutes. Maybe this is Envoy-S Metered (Multiphase) and or firmware version dependent? (Maybe you are mistaking with the cloud API's update frequency?)

My /api/v1/production/inverters inverter data is already properly read and output by envoy_reader (I can see the values in Home Assistant). I currently check the /api/v1/production data in a browser and refresh manually.

Fyi: There exist 3 different Envoy-S models:

I have the Envoy-S Metered Multiphase (EU) SKU: ENV-S-WM-230.

Envoy-S Models

Envoy-S Metered and Envoy-S Metered Multiphase (IQ Envoy) are both black but the Envoy-S Metered Multiphase has the cover screw on the right side while the Envoy-S Metered has the cover screw on the left side (like Envoy-S Standard).

lnlp commented 3 years ago

@gtdiehl - I think the only cases where you'd want to use /api/v1/production are:

  1. The envoy doesn't support production.json
  2. The envoy is configured without CT metering and the user wants WH totals for today and 7 days which aren't available in production.json.

Regarding bullet 2: No, not only those two values. In Home Assistant ('Enphase Envoy' add-in which uses envoy_reader) I do not get actual values for all 4 values shown in /api/v1/production. Because envoy_reader appears to read them from /production.json where the names are present but their values are 0 and do not get updated.

When I run envoy_reader from a command prompt, the 4 production values shown are 0. When I modify envoy_reader.phy to explicitly set endpoint_type to "P" then the production values are read from /api/v1/production and are displayed properly.

gtdiehl commented 3 years ago

I'm still looking over the code and the different paths taken and the different types of envoy devices with and without features.

Though I wonder if something like this would solve the problem

Add a function called hasMeteringEnabled() which will return True if production[1]eim activeCount is >= 1 else False

Then change part of detect_model() from https://github.com/jesserizzo/envoy_reader/blob/c43704cc5e64d45c8ea9c362e26dd4b9d3197f1c/envoy_reader/envoy_reader.py#L50-L51

To:

        if response.status_code == 200 and self.hasProductionAndConsumption(response.json()) and self.hasMeteringEnabled():
            self.endpoint_type = "PC"

So if the Envoy has Metering enabled values will be retrieved from /production.json and if Metering is not enabled values will be retrieved from /api/v1/production.

I just wonder is there a configuration of the Envoy where a production CT is installed without a consumption CT? EDIT: I actually just found a setup like this. A CT is installed and Metering is only monitoring Production. According to the IQ Envoy manual this is possible if the user wants revenue-grade production metering.

lnlp commented 3 years ago

Here are captures of /api/v1/production, /production.json and /api/v1/production/inverters. All were taken at the same time. Data coming from /api/v1/* is updated every 5 minutes.

/api/v1/production:

{
  "wattHoursToday": 1716,
  "wattHoursSevenDays": 53220,
  "wattHoursLifetime": 92854,
  "wattsNow": 2236
}


/production.json:

{
    "production": [
        {
            "type": "inverters",
            "activeCount": 15,
            "readingTime": 1604478431,
            "wNow": 2235,
            "whLifetime": 92854
        },
        {
            "type": "eim",
            "activeCount": 0,
            "measurementType": "production",
            "readingTime": 1604478474,
            "wNow": 2.599,
            "whLifetime": 0.0,
            "varhLeadLifetime": 0.0,
            "varhLagLifetime": 0.0,
            "vahLifetime": 0.0,
            "rmsCurrent": 0.534,
            "rmsVoltage": 245.583,
            "reactPwr": -0.0,
            "apprntPwr": 60.789,
            "pwrFactor": 0.01,
            "whToday": 0.0,
            "whLastSevenDays": 0.0,
            "vahToday": 0.0,
            "varhLeadToday": 0.0,
            "varhLagToday": 0.0
        }
    ],
    "consumption": [
        {
            "type": "eim",
            "activeCount": 0,
            "measurementType": "total-consumption",
            "readingTime": 1604478474,
            "wNow": -0.035,
            "whLifetime": 0.0,
            "varhLeadLifetime": 0.0,
            "varhLagLifetime": 0.0,
            "vahLifetime": 0.0,
            "rmsCurrent": 0.285,
            "rmsVoltage": 246.27,
            "reactPwr": 0.0,
            "apprntPwr": 70.082,
            "pwrFactor": -0.0,
            "whToday": 0.0,
            "whLastSevenDays": 0.0,
            "vahToday": 0.0,
            "varhLeadToday": 0.0,
            "varhLagToday": 0.0
        },
        {
            "type": "eim",
            "activeCount": 0,
            "measurementType": "net-consumption",
            "readingTime": 1604478474,
            "wNow": -2.634,
            "whLifetime": 0.0,
            "varhLeadLifetime": 0.0,
            "varhLagLifetime": 0.0,
            "vahLifetime": 0.0,
            "rmsCurrent": 0.249,
            "rmsVoltage": 246.958,
            "reactPwr": 0.0,
            "apprntPwr": 59.944,
            "pwrFactor": -0.12,
            "whToday": 0,
            "whLastSevenDays": 0,
            "vahToday": 0,
            "varhLeadToday": 0,
            "varhLagToday": 0
        }
    ],
    "storage": [
        {
            "type": "acb",
            "activeCount": 0,
            "readingTime": 0,
            "wNow": 0,
            "whNow": 0,
            "state": "idle"
        }
    ]
}


/api/v1/production/inverters (serial numbers have been obfuscated):

[
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478424,
      "devType": 1,
      "lastReportWatts": 151,
      "maxReportWatts": 244
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478408,
      "devType": 1,
      "lastReportWatts": 152,
      "maxReportWatts": 252
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478415,
      "devType": 1,
      "lastReportWatts": 148,
      "maxReportWatts": 246
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478420,
      "devType": 1,
      "lastReportWatts": 151,
      "maxReportWatts": 242
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478411,
      "devType": 1,
      "lastReportWatts": 149,
      "maxReportWatts": 247
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478416,
      "devType": 1,
      "lastReportWatts": 147,
      "maxReportWatts": 246
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478421,
      "devType": 1,
      "lastReportWatts": 151,
      "maxReportWatts": 244
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478428,
      "devType": 1,
      "lastReportWatts": 148,
      "maxReportWatts": 240
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478412,
      "devType": 1,
      "lastReportWatts": 151,
      "maxReportWatts": 250
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478416,
      "devType": 1,
      "lastReportWatts": 147,
      "maxReportWatts": 241
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478424,
      "devType": 1,
      "lastReportWatts": 147,
      "maxReportWatts": 244
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478407,
      "devType": 1,
      "lastReportWatts": 150,
      "maxReportWatts": 252
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478413,
      "devType": 1,
      "lastReportWatts": 146,
      "maxReportWatts": 245
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478420,
      "devType": 1,
      "lastReportWatts": 148,
      "maxReportWatts": 241
    },
    {
      "serialNumber": "12203xxxxxxx",
      "lastReportDate": 1604478426,
      "devType": 1,
      "lastReportWatts": 150,
      "maxReportWatts": 239
    }
  ]

Observations:

Questions

lnlp commented 3 years ago

I'm still looking over the code and the different paths taken and the different types of envoy devices with and without features.

Though I wonder if something like this would solve the problem

Add a function called hasMeteringEnabled() which will return True if production[1]eim activeCount is >= 1 else False

Then change part of detect_model() from ...

That should probably do it.

Do you know what the "eim" stands for? (I suspect something like: electrical induction metering)

lnlp commented 3 years ago

Also would need to check this behavior on newer firmware such as D5.0.49 (77afa8) to see what happens when metering is not setup and if the /production.json page returns valid data or not.

Is that new Envoy firmware? Does that get installed automatically?

lnlp commented 3 years ago

I'm not sure what would be a good way to name production Watts data elements to allow referencing either or both of them in Home Assistant's configuration. Perhaps for compatibility there should be three available entities for the wNow values in Home Assistant:

  • production - the current key that references the code's choice of wNow source
  • production_inverters - a new key that references wNow for productiontype == inverters. (* This is production[0].wNow)
  • production_eim - a new key that references wNow for production type == eim (* This is production[1].wNow)

Having these three elements would allow the user to override Envoy reader's choice if necessary as well as track both on systems with CTs if the user chooses that.

(*) Note: I don't know that production[0].type is always inverters and production[1].type is always eim.

What about 3-phase systems? Will there be / shouldn't there be 3 separate "eim" sections, each for one phase? In that case, where is the total power reported in /production.json ? In the "inverters" section?

And when envoy_reader reads its data from the "eim" section shouldn't that be done for each phase separately and then summed to get totals for all 3 phases?

I'm a bit puzzled here because I have a 3-phase system with no CT's installed, and (only) a single "eim" production section shows up where a single rmsVoltage (and rmsCurrent) are specified while each phase has its own different voltage (which my smart electricity meter shows).

gtdiehl commented 3 years ago

I agree a better solution would be to have additional parameters.

Currently the envoy_reader api returns:

production
daily_production
seven_days_production
lifetime_production

consumption
daily_consumption
seven_days_consumption
lifetime_consumption

inverter_production

These come from mainly 3 pages (excluding Envoy running <3.9 firmware)

/api/v1/production
/production.json
/api/v1/production/inverters

I would propose removing the original consumption parameters and adding additional parameters (lets prepend them in this example with metered_*

metered_production
metered_daily_production
metered_seven_days_production
metered_lifetime_production

metered_consumption
metered_daily_consumption
metered_seven_days_consumption
metered_lifetime_consumption

Then change it so any polling of the original production parameters only come from /api/v1/production. And the metered_* parameters come from the appropriate section of /production.json that contains type of eim.

Currently to minimize api calls between the envoy and home assistant one function is called to gather up all the data (even if the device does not support the parameter) and send it across.

Then a question, does the api return the eim values that is got from the envoy even when the activeCount = 0? And possibly add another parameter called activeCTCount to tell the user if the Metering feature is enabled on their Envoy?

Going down this path means renaming 4 monitored_conditions and adding 4 or5 new ones will appear for each new Envoy sensor added to Home Assistant

I'm a bit puzzled here because I have a 3-phase system with no CT's installed, and (only) a single "eim" production section shows up where a single rmsVoltage (and rmsCurrent) are specified while each phase has its own different voltage (which my smart electricity meter shows).

I'm not sure on this one. According to the Envoy-S Metered Multiphase Installation manual there are 3 CTs installed on the Production and Consumption sides. Would have to find a system that has 3Phase with Metering enabled and CTs installed.

@jesserizzo What do you think about moving all of the production polling to /api/v1/production and only use /production.json for metered production and consumption data?

lnlp commented 3 years ago

@gtdiehl I like the suggestion of using /api/v1/production by default and add the additional metered_* parameters for /production.json.

I would also prefer if the timestamp of when the values were updated can be added, e.g. lastReportDate and readTime (there's little consistency in the different names unfortunately).

For /api/v1/production the timestamp can be read from production.json "inverters" section.

And for the nice to have wishlist: It would be nice if other useful parameters e.g. rmsCurrent and rmsVoltage could be reported by envoy_reader as well.

rct commented 3 years ago

@gtdiehl

Question for you or anyone who has CT metering turned on:

I believe production.json - production[0].wNow shows the current power from the inverters regardless of whether CT metering is turned on or not. Unless I'm wrong about that, it would be useful to be able to get wNow (production) for both, so hopefully your naming would also include inverters_production. If the answer to the above question is true, then it would also be useful to have inverter_lifetime_production in addition to metered_lifetime_production.

Thanks for working on this.

rct commented 3 years ago

@lnlp - trying to answer a few of your questions with what I've learned so far.

gtdiehl commented 3 years ago

So I've hacked together some code

Here is the output of envoy_reader from various Envoys. I removed the inverter data and right now the values are zero due to no sunlight, but you should get an idea of what would be sent to Home Assistant once that sensor code is updated.

Envoy-S or IQ running D5.0.49 without Metering Enabled

Reading...
production:                                  0
daily_production:                            42499
seven_days_production:                       297600
lifetime_production:                         23543316
metered_production_activeCount:              0
metered_production:                          0
metered_daily_production:                    0
metered_seven_days_production:               0
metered_lifetime_production:                 0
metered_consunsumption_activeCount:          0
metered_consumption:                         0
metered_daily_consumption:                   0
metered_seven_days_consumption:              0
metered_lifetime_consumption:                0
inverters_production:                        {....}

Envoy-S or IQ running D5.0.49 with Metering Enabled

Reading...
production:                                  0
daily_production:                            33110
seven_days_production:                       215061
lifetime_production:                         41599751
metered_production_activeCount:              1
metered_production:                          0
metered_daily_production:                    33111
metered_seven_days_production:               215062
metered_lifetime_production:                 41599752
metered_consunsumption_activeCount:          1
metered_consumption:                         27
metered_daily_consumption:                   39608
metered_seven_days_consumption:              171145
metered_lifetime_consumption:                -140548
inverters_production:                        {...}

Envoy-S running D5.0.49 without Metering Enabled and the /production.json page does not contain Production or Consumption keys

Reading...
production:                                  0
daily_production:                            131036
seven_days_production:                       1641458
lifetime_production:                         424042360
metered_production_activeCount:              Metered Production data not available for your Envoy device.
metered_production:                          Metered Production data not available for your Envoy device.
metered_daily_production:                    Metered Production data not available for your Envoy device.
metered_seven_days_production:               Metered Production data not available for your Envoy device.
metered_lifetime_production:                 Metered Production data not available for your Envoy device.
metered_consunsumption_activeCount:          Metered Consumption data not available for your Envoy device.
metered_consumption:                         Metered Consumption data not available for your Envoy device.
metered_daily_consumption:                   Metered Consumption data not available for your Envoy device.
metered_seven_days_consumption:              Metered Consumption data not available for your Envoy device.
metered_lifetime_consumption:                Metered Consumption data not available for your Envoy device.
inverters_production:                        {...}

Envoy-C running D3.15.7

Reading...
production:                                  0
daily_production:                            10271
seven_days_production:                       73658
lifetime_production:                         18227440
metered_production_activeCount:              Metered Production data not available for your Envoy device.
metered_production:                          Metered Production data not available for your Envoy device.
metered_daily_production:                    Metered Production data not available for your Envoy device.
metered_seven_days_production:               Metered Production data not available for your Envoy device.
metered_lifetime_production:                 Metered Production data not available for your Envoy device.
metered_consunsumption_activeCount:          Metered Consumption data not available for your Envoy device.
metered_consumption:                         Metered Consumption data not available for your Envoy device.
metered_daily_consumption:                   Metered Consumption data not available for your Envoy device.
metered_seven_days_consumption:              Metered Consumption data not available for your Envoy device.
metered_lifetime_consumption:                Metered Consumption data not available for your Envoy device.
inverters_production:                        {...}
gtdiehl commented 3 years ago

The code is on one of my branches. I would like some input from @jesserizzo and others before opening a PR to merge this change.

Modified envoy_reader.py

lnlp commented 3 years ago

I would like some input from @jesserizzo and others

I tested your modified envoy_reader with my Envoy-S Metered Multiphase with metering disabled (no CT's). It appears to work fine.

Observation: metered values get truncated instead of rounded. Why not use actual values (with decimals, when reported)?

My Envoy reports tiny metered values while metering is disabled. This is caused by my Envoy not by envoy_reader. Could this have been fixed in a later firmware revision? Reported values while metering disabled: metered_production wNow: 2.79 (gets truncated to 2) metered_consumption wNow: 0.836 (gets truncated to 0).

My Envoy homepage shows "Software version R4.10.35 (6ed292)" is that the Envoy's firmware version? Is D5.0.49 newer than R4.10.35? (latter number is lower but 'R' looks higher/newer than 'D')

Can you please add the following parameters for reading timestamps?: production_readingtime (source: production.json production inverters readingTime) metered_production_readingtime metered_consumption_readingtime

lnlp commented 3 years ago

How about simplifying "Metered Production data not available for your Envoy device." and "Metered Production data not available for your Envoy device."

to: "not available"

rct commented 3 years ago

@gtdiehl - Here's my Envoy IQ (metering turned off) output using your change_polling_pages branch:

production:                                  3032
daily_production:                            9615
seven_days_production:                       108838
lifetime_production:                         12948525
metered_production_activeCount:              0
metered_production:                          2323
metered_daily_production:                    32
metered_seven_days_production:               32
metered_lifetime_production:                 26225
metered_consunsumption_activeCount:          0
metered_consumption:                         2323
metered_daily_consumption:                   26225
metered_seven_days_consumption:              26225
metered_lifetime_consumption:                26225
inverters_production: <snip>

/api/v1/production

{
  "wattHoursToday": 9615,
  "wattHoursSevenDays": 108838,
  "wattHoursLifetime": 12948525,
  "wattsNow": 3032
}

production.json

{
  "production": [
    {
      "type": "inverters",
      "activeCount": 27,
      "readingTime": 1604594985,
      "wNow": 3033,
      "whLifetime": 12948525
    },
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "production",
      "readingTime": 1604595051,
      "wNow": 2334.435,
      "whLifetime": 26225.198,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 9500.168,
      "vahLifetime": 34969.369,
      "rmsCurrent": 19.475,
      "rmsVoltage": 252.92,
      "reactPwr": 618.904,
      "apprntPwr": 2460.099,
      "pwrFactor": 0.95,
      "whToday": 32.198,
      "whLastSevenDays": 32.198,
      "vahToday": 41.369,
      "varhLeadToday": 0,
      "varhLagToday": 17.168
    }
  ],
  "consumption": [
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "total-consumption",
      "readingTime": 1604595051,
      "wNow": 2334.435,
      "whLifetime": 26225.198,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 9500.168,
      "vahLifetime": 0,
      "rmsCurrent": 19.204,
      "rmsVoltage": 253.218,
      "reactPwr": -618.904,
      "apprntPwr": 4862.85,
      "pwrFactor": 0.48,
      "whToday": 26225.198,
      "whLastSevenDays": 26225.198,
      "vahToday": 0,
      "varhLeadToday": 0,
      "varhLagToday": 9500.168
    },
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "net-consumption",
      "readingTime": 1604595051,
      "wNow": -0,
      "whLifetime": 0,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 0,
      "vahLifetime": 0,
      "rmsCurrent": 0.27,
      "rmsVoltage": 253.516,
      "reactPwr": -0,
      "apprntPwr": 34.248,
      "pwrFactor": 0,
      "whToday": 0,
      "whLastSevenDays": 0,
      "vahToday": 0,
      "varhLeadToday": 0,
      "varhLagToday": 0
    }
  ],
  "storage": [
    {
      "type": "acb",
      "activeCount": 0,
      "readingTime": 0,
      "wNow": 0,
      "whNow": 0,
      "state": "idle"
    }
  ]
}
gtdiehl commented 3 years ago

Thank you for running the code!

I originally had the metered values not being returned when metering is disabled (activeCount == 0), but then I thought someone might want these values or some other reason. So I just display the values unless the device actually does not support metering.

They can always filter these values out in Home Assistant by specifying which monitored_conditions they want to see.

rct commented 3 years ago

Yes, I do want and think the values for both inverters and eim should be accessible if available. I just wanted to let you know the logic might not be working as intended.

In my comments above, I was arguing for the case to still be able to get the inverter numbers even if metering is enabled.

gtdiehl commented 3 years ago

I see where you are coming from but to keep the code simple I'm retrieving inverter production values from /api/v1/production rather than pulling a value from /production.json and the rest from /api/v1/production

It seems the the values are almost the same and they are both updated every 5mins. Other than reducing http calls to the envoy device, is there another benefit to pull the production data from the two pages rather than one?

gtdiehl commented 3 years ago

Observation: metered values get truncated instead of rounded.

Why not use actual values (with decimals, when reported)?

Good catch! I'll return values with a decimal

rct commented 3 years ago

Other than reducing http calls to the envoy device, is there another benefit to pull the production data from the two pages rather than one?

One case that is not handled, if you want the inverters current production and WH lifetime production, with metering turned on, you can only get that from production.json. In other words, if you explicitly want to access that data for the inverters whether or not metering is on, production[0] is the place to get it.

Maybe this is only of interest to me. When metering gets turned on /api/v1/production will switch over to metered values and will no longer have access to the history in the inverter data.

gtdiehl commented 3 years ago

One case that is not handled, if you want the inverters current production and WH lifetime production, with metering turned on, you can only get that from production.json. In other words, if you explicitly want to access that data for the inverters whether or not metering is on, production[0] is the place to get it.

Maybe this is only of interest to me. When metering gets turned on /api/v1/production will switch over to metered values and will no longer have access to the history in the inverter data.

@rct I get what you're saying. I'll have to look into this one a bit.

gtdiehl commented 3 years ago

I know this may seem like a back step but I'm thinking of changing back to the original output of the envoy_reader api, and have the code get the production values based on the activeCount attribute. The rationale is so that the Home Assistant sensor monitored_conditions does not have to change which ultimately does not create a breaking change or force users to change their environment.

Like I said I'm still thinking about it and weighing the pros and cons. I really like to give the user all of the data and let them figure out what they want to use. But have to think about current users.

rct commented 3 years ago

@gtdiehl - I understand. It's worth thinking about what people really expect and would really like to get from Envoy reader. Thank you for working on envoy_reader.

Being able to access any the data I need motivated me to just try going direct with Home Assistant's REST sensor. So far I'm happy. I have control to access to any of the raw values I want. Currently I poll both /api/v1/production and production.json.

Though HA's utility sensor (and with a little more work, possibly with the statistics integration) I won't need to poll /api/v1/production since I have flexibility to compute the deltas and aggregate into what ever periods I'd like.

I've stopped using envoy_reader since it isn't currently giving me anything I can't easily get with REST. I wasn't able to use the per inverter data because the 54 HTTP requests for 27 inverters eventually caused problems with the envoy. Also via the REST sensor and template sensors, I'm getting the data I need with a single HTTP request.

The downside of using the REST sensor set up is the complexity of editing configuration.yaml.

There are a number of things envoy_reader could be doing that probably wouldn't be too hard using the data from production/inverters:

There could also be an envoy sensor with informational attributes about the Envoy/system, number of inverters, what metering is being used, serial number, software version,

lnlp commented 3 years ago

I know this may seem like a back step but I'm thinking of changing back to the original output of the envoy_reader api, and have the code get the production values based on the activeCount attribute.

@rct For me and others that will mean that envoy_reader and the 'Enphase Envoy' Home Assistant add-in will be useless for reading the standard production counters as activeCount = 0 due to not having installed CT's. That envoy_reader currently doesn't properly handle this is a serious flaw. I understand the importance of backwards compatibility for current users, but not fixing this issue makes envoy_reader unusable for any IQ Envoy and Envoy-S Metered (multi-phase) where CT's are not installed.

I have chosen my Envoy model for compatibility and future expandability (expand with CT's in case I feel need for them) and was unpleasantly surprised that envoy_reader / 'Enphase Envoy' add-in (currently) do not properly handle it.

I hope you can implement a fix for this, which can be done even when changing back to the original output format.

lnlp commented 3 years ago

@rct

Being able to access any the data I need motivated me to just try going direct with Home Assistant's REST sensor. So far I'm happy. I have control to access to any of the raw values I want. Currently I poll both /api/v1/production and production.json.

Though HA's utility sensor (and with a little more work, possibly with the statistics integration) I won't need to poll /api/v1/production since I have flexibility to compute the deltas and aggregate into what ever periods I'd like.

I've stopped using envoy_reader since it isn't currently giving me anything I can't easily get with REST. I wasn't able to use the per inverter data because the 54 HTTP requests for 27 inverters eventually caused problems with the envoy. Also via the REST sensor and template sensors, I'm getting the data I need with a single HTTP request.

The downside of using the REST sensor set up is the complexity of editing configuration.yaml.

Are you willing to share how you have setup the REST sensor for Envoy?

rct commented 3 years ago

I should probably put this in a GIST.

First the simple example. It reads the values from /api/v1/production and creates a sensor for each. It's pretty straightforward since the returned JSON is simple and flat.

# Simple JSON REST exercise Envoy's simple /api/v1/production.
sensor envoy_simple_rest:
  - platform: rest
    name: envoy_production_rest
    resource: http://envoy.local/api/v1/production
    scan_interval: 150
    json_attributes:
      - wattsNow
      - wattHoursToday
      - wattHoursSevenDays
      - wattHoursLifetime
    value_template: '{{ states.sensor.envoy_production_rest.attributes["wattsNow"] }}'
    unit_of_measurement: 'W'
  - platform: template
    sensors:
      envoy_rest_production_today:
        friendly_name: 'Envoy Rest Production Today'
        unit_of_measurement: 'Wh'
        value_template: '{{ states.sensor.envoy_production_rest.attributes["wattHoursToday"] }}'
      envoy_rest_production_7days:
        friendly_name: 'Envoy Rest Production 7 days'
        unit_of_measurement: 'Wh'
        value_template: '{{ states.sensor.envoy_production_rest.attributes["wattHoursSevenDays"] }}'
      envoy_rest_production_lifetime:
        friendly_name: 'Envoy Rest Production Lifetime'
        unit_of_measurement: 'Wh'
        value_template: '{{ states.sensor.envoy_production_rest.attributes["wattHoursLifetime"] }}'

The main sensor has the current wattsNow with the other values available as attributes. It is showing zero because it's night time here.

image

The 3 template sensors break out those attributes into their own entities:

image

Here's the utility based sensor that gives easy access to period based accumulations and the previous period for comparisons.

Using the above sensor, this tracks daily, weekly, monthly WH production totals. Weekly by default resets based on day of the week. It isn't a 7 day rolling counter like the Envoy has. It isn't really needed since /api/v1/production has it's own counters to accumulate in, but showing as an example.

# Solar Envoy production using utility meter integration and Simple REST sensor /v1/production
utility_meter:
  envoy_utility_daily_production:
    source: sensor.envoy_rest_production_lifetime
    cycle: daily
  envoy_utility_weekly_production:
    source: sensor.envoy_rest_production_lifetime
    cycle: weekly
  envoy_utility_monthly_production:
    source: sensor.envoy_rest_production_lifetime
    cycle: monthly

image

This shows that today's production based on wattHoursLifetime is 24,021 WH and yesterday it was 23,333 WH. I'm not sure why this is 24,021 vs. the 24,493 that the envoy accumulated in wattHoursToday.

I can also show how to get the values out of production.json. It was a little trickier because returned JSON is more complex (nested structures).

Hope this helps.

gtdiehl commented 3 years ago

@rct and @lnlp Thank you for testing out my code!

Given that changes are not being implemented/reviewed/merged on the Home Assistant side in a timely manner, as well as weighing the pros and cons as I have previously noted. I have decided to go back to the original format and just fix the issue that @lnlp had originally wrote up. Maybe in the future the additional sensors can be created but for now the easiest thing is to fix the envoy_reader API.

So I think I have fixed the issue. My code detects if the activeCount > 0 and uses the CT (eim) values from /production.json. If not than the production values are retrieved from two different pages; the current production is retrieved from /production.json in the first part of output ["production"][0]["wNow"] the rest of the production values are retrieved from /api/v1/production.

I have also reduced the number of HTTP calls the API makes from about 17 calls to /production.json to 1 call to that page. Of course this does not fix the issue on the Home Assistant side where the sensor is calling the /api/v1/production/inverters page for each inverter. That is resolved in this PR home-assistant/core#42857

EDIT: Of course the link is needed for the code. The updated code is located at envoy_reader.py

@lnlp Does this code fix your problem?

lnlp commented 3 years ago

@gtdiehl

I have tested your latest envoy_reader.py. This is the output: (Inverter serial numbers are obfuscated, linefeeds have been added for clarity)

Reading...
Metering (CT) Status:    False
production:              2329
consumption:             0
daily_production:        5511
daily_consumption:       0
seven_days_production:   52206
seven_days_consumption:  0
lifetime_production:     168910
lifetime_consumption:    0
inverters_production:   {'<obfuscated>': [156, '2020-11-13 11:28:18'],
                         '<obfuscated>': [159, '2020-11-13 11:28:01'], 
                         '<obfuscated>': [154, '2020-11-13 11:28:08'], 
                         '<obfuscated>': [154, '2020-11-13 11:28:14'], 
                         '<obfuscated>': [154, '2020-11-13 11:28:03'], 
                         '<obfuscated>': [156, '2020-11-13 11:28:09'], 
                         '<obfuscated>': [156, '2020-11-13 11:28:14'], 
                         '<obfuscated>': [155, '2020-11-13 11:28:21'], 
                         '<obfuscated>': [157, '2020-11-13 11:28:04'], 
                         '<obfuscated>': [152, '2020-11-13 11:28:10'], 
                         '<obfuscated>': [156, '2020-11-13 11:28:17'], 
                         '<obfuscated>': [159, '2020-11-13 11:28:00'], 
                         '<obfuscated>': [154, '2020-11-13 11:28:05'], 
                         '<obfuscated>': [154, '2020-11-13 11:28:12'], 
                         '<obfuscated>': [154, '2020-11-13 11:28:19']}

Remarks

The code appears to work fine now but I do have the following remarks:

Questions

OllemGit commented 3 years ago

["_Be aware that CT's can be installed for either Production, or Consumption, or both. envoyreader should handle all these (4) situations correctly."]

Exactly: I have only CT's installed for measuring Consumption...

My json output looks like this:

{
   "production":[
      {
         "type":"inverters",
         "activeCount":14,
         "readingTime":1605608206,
         "wNow":546,
         "whLifetime":1456901
      },
      {
         "type":"eim",
         "activeCount":0,
         "measurementType":"production",
         "readingTime":1605608474,
         "wNow":0.0,
         "whLifetime":0.982,
         "varhLeadLifetime":1.333,
         "varhLagLifetime":3.617,
         "vahLifetime":597.594,
         "rmsCurrent":0.316,
         "rmsVoltage":706.091,
         "reactPwr":1.397,
         "apprntPwr":74.398,
         "pwrFactor":0.0,
         "whToday":0.982,
         "whLastSevenDays":0.982,
         "vahToday":3.594,
         "varhLeadToday":0.333,
         "varhLagToday":0.617
      }
   ],
   "consumption":[
      {
         "type":"eim",
         "activeCount":1,
         "measurementType":"total-consumption",
         "readingTime":1605608474,
         "wNow":309.368,
         "whLifetime":2197497.272,
         "varhLeadLifetime":2352896.133,
         "varhLagLifetime":1251.029,
         "vahLifetime":5424211.588,
         "rmsCurrent":7.952,
         "rmsVoltage":706.203,
         "reactPwr":-902.634,
         "apprntPwr":1872.004,
         "pwrFactor":0.17,
         "whToday":6425.272,
         "whLastSevenDays":137122.272,
         "vahToday":15363.588,
         "varhLeadToday":10692.133,
         "varhLagToday":2.029
      },
      {
         "type":"eim",
         "activeCount":1,
         "measurementType":"net-consumption",
         "readingTime":1605608474,
         "wNow":309.368,
         "whLifetime":0.0,
         "varhLeadLifetime":2352894.8,
         "varhLagLifetime":1247.412,
         "vahLifetime":5424211.588,
         "rmsCurrent":7.636,
         "rmsVoltage":706.147,
         "reactPwr":-901.237,
         "apprntPwr":5391.886,
         "pwrFactor":0.06,
         "whToday":0,
         "whLastSevenDays":0,
         "vahToday":0,
         "varhLeadToday":0,
         "varhLagToday":0
      }
   ],
   "storage":[
      {
         "type":"acb",
         "activeCount":0,
         "readingTime":0,
         "wNow":0,
         "whNow":0,
         "state":"idle"
      }
   ]
}
lnlp commented 3 years ago

@OllemGit

Exactly: I have only CT's installed for measuring Consumption...

"CT's" is plural but consumption/eim/activeCount only shows 1. Is it correct that you have only one single CT for consumption?

gtdiehl commented 3 years ago

I have tested your latest envoy_reader.py. This is the output: @lnlp Thank you for running the code!

  • I suggest to use the term 'CT Metering' (or 'CT Metering Enabled') instead of "Metering (CT) Status". 'Status' also gives an impression that its value may change frequently which is not the case. In most cases this value will be fixed for the lifetime of the system.

This status was for me while debugging. It'll be removed from the code.

  • Be aware that CT's can be installed for either Production, or Consumption, or both. envoy_reader should handle all these (4) situations correctly. So in practice it can be possible that only CT('s) for consumption are installed. This means that we need separate 'CT Metering' parameters for production and for consumption. I suggest to add the following two output parameters for this (so they can be read as sensor values in Home Assistant also);

    • ct_metering_production
    • ct_metering_consumption (add '_enabled' to these names if preferred)

Can you open a new issue (enhancement) to track adding new parameters?

  • If ct_metering_consumption is false then the consumption values should be omitted from the output because it is meaningless to report values (and create sensors for them in Home Assistant) that are actually not present / not generated. Parameter ct_metering_consumption should allways be present in the output however and should not be omitted (for Envoys that output production.json).

I'm following the current convention that was previously coded. I would have to look into how to send specific data from the envoy_reader API to Home Assistant. Can you open a new issue (enhancement) to track this one?

  • As previously requested please add the following output parameters: (for recent Envoy models that output production.json)

    • production_readingtime
    • consumption_readingtime _(only when ct_meteringconsumption is True) When ct_metering_production is False readingTime should be taken from the production/inverters section, when True from the production/eim section.
  • In my situation (ct_metering_production is False) the production value is read correctly from the production/inverters section in production.json.

Can you open a new issue (enhancement) to track this one?

Questions

  • When the production version of envoy_reader will be updated, will this get automatically updated in existing Home Assistant (Enphase Envoy add-in) installations?

No Home Assistant will not get fully updated automatically. That's why I'm asking to track the above as separate issues.

When I mean fully is that if something can be fixed on the Api side, such as the original bug of the Production values being zero than I can make a change here and request Home Assistant through a PR to update the version of the API being used. That's one line changed on the Home Assistant side and as long as the reviewer is okay with the API side change log the updated API will be included in the next Home Assistant release.

Now to change how things are displayed in Home Assistant, or adding/removing sensors that means probably both an API side code change as well as the sensor code change on the Home Assistant side. The review process is greater.

I know not a perfect solution but through small changes eventually we will get to a release that meets the needs of Envoy users across all firmwares 😃

So as it stands is your original bug of Production values displaying zero fixed?

OllemGit commented 3 years ago

@OllemGit

Exactly: I have only CT's installed for measuring Consumption...

"CT's" is plural but consumption/eim/activeCount only shows 1. Is it correct that you have only one single CT for consumption?

That's an interesting question... I have a 3 phase power system, so I also have 3 CT's which each measure 1 fase, but in the Ephase app the value is shown as a single Power consumption parameter.

lnlp commented 3 years ago

Very interesting indeed. I would expect to see at least three different sets of values, one for each phase.

Could there there be another (URL) location on the Envoy where differentiated data for each phase is available? Or maybe the Envoy combines the data of all three phases and the differentiation gets lost?

What use are values of e.g. rmsCurrent and rmsVoltage if only single instances reported for these instead of separately per phase?

lnlp commented 3 years ago

So as it stands is your original bug of Production values displaying zero fixed?

Yes the original bug in envoy_reader has been fixed.

I hope we can see it being updated in Home Assistant as well soon.

lnlp commented 3 years ago

@gtdiehl

  • ct_metering_production
  • ct_metering_consumption (add '_enabled' to these names if preferred)

Can you open a new issue (enhancement) to track adding new parameters?

Done, see #43.

  • production_readingtime
  • consumption_readingtime _(only when ct_meteringconsumption is True) When ct_metering_production is False readingTime should be taken from the production/inverters section, when True from the production/eim section.
  • In my situation (ct_metering_production is False) the production value is read correctly from the production/inverters section in production.json.

Can you open a new issue (enhancement) to track this one?

Done, see #42.

gtdiehl commented 3 years ago

So as it stands is your original bug of Production values displaying zero fixed?

Yes the original bug in envoy_reader has been fixed.

I hope we can see it being updated in Home Assistant as well soon.

@lnlp I'll release the new version on pypi this weekend and open a PR on the Home Assistant side as well. Yeah hopefully it'll make it into the 0.119 release!

gtdiehl commented 3 years ago

@gtdiehl

  • ct_metering_production
  • ct_metering_consumption (add '_enabled' to these names if preferred)

Can you open a new issue (enhancement) to track adding new parameters?

Done, see #43.

  • production_readingtime
  • consumption_readingtime _(only when ct_meteringconsumption is True) When ct_metering_production is False readingTime should be taken from the production/inverters section, when True from the production/eim section.
  • In my situation (ct_metering_production is False) the production value is read correctly from the production/inverters section in production.json.

Can you open a new issue (enhancement) to track this one?

Done, see #42.

Thank you for opening those issues. Maybe I should have been a little clearer but Home Assistant is not letting existing integrations to change their monitored_conditions. So to make changes unfortunately we can't add a new sensor and display the new data. I added some more detail in the Issue #41.

lnlp commented 3 years ago

See my response in #41.

gtdiehl commented 3 years ago

Very interesting indeed. I would expect to see at least three different sets of values, one for each phase.

Could there there be another (URL) location on the Envoy where differentiated data for each phase is available?

Or maybe the Envoy combines the data of all three phases and the differentiation gets lost?

What use are values of e.g. rmsCurrent and rmsVoltage if only single instances reported for these instead of separately per phase?

There was a reply in the Home Assistant Community forum where a person took this screenshot. image

Not sure what the actual url for the page is though.

Forum link

lnlp commented 3 years ago

Data from /stream/meter (updates every second) shows the data separately for each phase but unfortunately this requires installer permissions. The data from my Envoy (3-phase, no CT's) looks like this (pretty-printed):

data: {
    "production": {
        "ph-a": {
            "p": 2.69,
            "q": 0.913,
            "s": 58.822,
            "v": 235.022,
            "i": 0.25,
            "pf": 0.13,
            "f": 50.0
        },
        "ph-b": {
            "p": 0.212,
            "q": -0.0,
            "s": 0.714,
            "v": 3.764,
            "i": 0.191,
            "pf": 1.0,
            "f": 50.0
        },
        "ph-c": {
            "p": 0.071,
            "q": 0.0,
            "s": 0.35,
            "v": 3.657,
            "i": 0.097,
            "pf": 0.0,
            "f": 50.0
        }
    },
    "net-consumption": {
        "ph-a": {
            "p": -2.686,
            "q": 0.689,
            "s": 58.584,
            "v": 235.027,
            "i": 0.25,
            "pf": 0.0,
            "f": 50.0
        },
        "ph-b": {
            "p": 0.0,
            "q": 0.0,
            "s": -0.0,
            "v": 3.344,
            "i": -0.0,
            "pf": -1.0,
            "f": 50.0
        },
        "ph-c": {
            "p": -0.0,
            "q": -0.0,
            "s": 0.0,
            "v": 5.602,
            "i": 0.0,
            "pf": -1.0,
            "f": 50.0
        }
    },
    "total-consumption": {
        "ph-a": {
            "p": 0.004,
            "q": -0.224,
            "s": 0.048,
            "v": 235.024,
            "i": 0.0,
            "pf": 0.07,
            "f": 50.0
        },
        "ph-b": {
            "p": 0.212,
            "q": 0.0,
            "s": 0.68,
            "v": 3.554,
            "i": 0.191,
            "pf": 0.31,
            "f": 50.0
        },
        "ph-c": {
            "p": 0.071,
            "q": -0.0,
            "s": 0.448,
            "v": 4.629,
            "i": 0.097,
            "pf": 0.16,
            "f": 50.0
        }
    }
}

From @OllemGit's feedback:

That's an interesting question... I have a 3 phase power system, so I also have 3 CT's which each measure 1 fase, but in the Ephase app the value is shown as a single Power consumption parameter.

Based on above it appears that /production.json is not going to contain data for each phase separately (in 3-phase system). Hopefully that data can be read from elsewhere on the Envoy, without requiring installer permissions.

@OllemGit Are all three phases wired to your Envoy power connector(most-left connector on the Envoy)?

OllemGit commented 3 years ago

Are all three phases wired to your Envoy power connector_(most-left connector on the Envoy)_? => 4 wires enter the most-left connector of the Envoy + 3x2 wires for the CT's I did not check (yet) where the come from and/or lead to

lnlp commented 3 years ago

I did not check (yet) where the come from and/or lead to

Thanks for checking.

That can only mean that all 3 phases are connected, the 4 wires will be: N, L1, L2 and L3.

My guess is that data from /stream/meter on your envoy will report realistric (true) voltages for all three of data/production/ph-a/v, data/production/ph-b/v and data/production/ph-c/v.

@OllemGit Would it be possible for you to verify this?

My Envoy reports a realistic value for data/production/ph-a/v while no CT's are installed. So this voltage is apparently read from L1 on the 'power input' connector. (My solar installer did not connect L2 and L3 on the 'power input' connector so I am currently unable to verify for ph-b and ph-c.)

My Envoy also reports a realistic value for data/xxx-consumption/ph-a/v. It appears that this value is probably taken from the same L1 input, but its value differs few-thousand's of a volt. An explanation for the difference may be rounding errors or different time of measurement.

The CT (current transformers) are used for measuring current. Theoretically one of the two CT wires could also be used for measuring voltage. Whether this is actually implemented (case B') when CT's are installed or that only the power input lines (left connector) are used ('case A') for voltage measurement I'm not sure. In 'case A' one may wonder why the phase voltages are reported separately for production, net-consumption and total-consumption. In 'case B' one may wonder why the 'power connector' on the left exists and allows to wire all 3 phases.

From the 'Envoy-S Metered Multiphase (ENV-S-WM-230) Installation and Operation Manual':

Use a Phase Coupler for a Multiphase Site If you are installing in a multiphase application, the power line communication signal must be “coupled” between the phases to allow the Envoy to communicate with all of the Enphase devices in the system. Install a phase coupler on the load side of the over-current protection device.

A phase coupler for communication with the inverters is not built-in to the Envoy. From this perspective it is not required to connect all 3 phases to the power connector.

  1. Provide a Power Connection The Envoy-S Metered Multiphase (ENV-S-WM-230) uses terminal blocks for power and metering connections. It does not include an AC power cord, and you must hard-wire it.

The manual is not clear about why all 3 phases should be connected to the 'power input' connector. Connecting only a single phase to the 'power input' connector (instead of all 3) already makes the system work (but will cause some values to be reported incorrectly).

For just the powering of the Envoy a single phase would be sufficient. This does not require to connect all 3 phases to the power connector.

From above I derive the conclusion that the reason of the presence of all 3 phases on the 'power input' connector must be to measure the voltage of each phase (as there appears no other plausible reason for wiring all 3 phases on this connector).

pedrocr commented 3 years ago

Instead of /stream/meter has anyone tried /ivp/meters/readings? I've seen that referred online as being able to get the individual readings and maybe it doesn't require the installer password.

rct commented 3 years ago

Interesting, /ivp/meters gives some configuration/status information such as phaseCount and phaseMode that I haven't seen elsewhere:

[
    {
        "eid": 704643xxx,
        "state": "disabled",
        "measurementType": "production",
        "phaseMode": "split",
        "phaseCount": 2,
        "meteringStatus": "normal",
        "statusFlags": []
    },
    {
        "eid": 704643yyy,
        "state": "disabled",
        "measurementType": "net-consumption",
        "phaseMode": "split",
        "phaseCount": 2,
        "meteringStatus": "normal",
        "statusFlags": []
    }
]

http://envoy.local/ivp/meters/readings seems to be very useful. I does give a way of polling the detailed data from each CT for what looks to be all three phases, whether it is installed or not. At first I seemed to get a 404 for but maybe I didn't get the plurals right though http://envoy.local/ivp/meters/readings.

Here's my output. Note: my CTs are currently disabled. There isn't any identifying object info inside the two objects in the top-level list. So it looks like you need to join with /ivp/meters based on eid to know what you are looking at and whether it is enabled.

[
    {
        "eid": 704643xxx,
        "timestamp": 1606326399,
        "actEnergyDlvd": 26225.198,
        "actEnergyRcvd": 1745.899,
        "apparentEnergy": 34969.369,
        "reactEnergyLagg": 9500.168,
        "reactEnergyLead": 0.000,
        "instantaneousDemand": 4741.303,
        "activePower": 4741.303,
        "apparentPower": 4869.138,
        "reactivePower": 672.800,
        "pwrFactor": 0.975,
        "voltage": 252.728,
        "current": 38.569,
        "freq": 60.000,
        "channels": [
            {
                "eid": 1778385yyy,
                "timestamp": 1606326399,
                "actEnergyDlvd": 13116.878,
                "actEnergyRcvd": 872.078,
                "apparentEnergy": 17522.510,
                "reactEnergyLagg": 4758.520,
                "reactEnergyLead": 0.000,
                "instantaneousDemand": 2376.200,
                "activePower": 2376.200,
                "apparentPower": 2439.620,
                "reactivePower": 337.502,
                "pwrFactor": 0.976,
                "voltage": 126.641,
                "current": 19.280,
                "freq": 60.000
            },
            {
                "eid": 1778385170,
                "timestamp": 1606326399,
                "actEnergyDlvd": 13108.320,
                "actEnergyRcvd": 873.821,
                "apparentEnergy": 17446.859,
                "reactEnergyLagg": 4741.648,
                "reactEnergyLead": 0.000,
                "instantaneousDemand": 2365.104,
                "activePower": 2365.104,
                "apparentPower": 2429.518,
                "reactivePower": 335.298,
                "pwrFactor": 0.973,
                "voltage": 126.087,
                "current": 19.289,
                "freq": 60.000
            },
            {
                "eid": 1778385171,
                "timestamp": 1606326399,
                "actEnergyDlvd": 0.000,
                "actEnergyRcvd": 0.000,
                "apparentEnergy": 0.000,
                "reactEnergyLagg": 0.000,
                "reactEnergyLead": 0.000,
                "instantaneousDemand": 0.000,
                "activePower": 0.000,
                "apparentPower": 0.000,
                "reactivePower": 0.000,
                "pwrFactor": 0.000,
                "voltage": 0.000,
                "current": 0.000,
                "freq": 60.000
            }
        ]
    },
    {
        "eid": 704643584,
        "timestamp": 1606326399,
        "actEnergyDlvd": 0.000,
        "actEnergyRcvd": 0.000,
        "apparentEnergy": 0.000,
        "reactEnergyLagg": 0.000,
        "reactEnergyLead": 0.000,
        "instantaneousDemand": -0.000,
        "activePower": -0.000,
        "apparentPower": 34.637,
        "reactivePower": 0.703,
        "pwrFactor": 0.000,
        "voltage": 253.326,
        "current": 0.274,
        "freq": 60.000,
        "channels": [
            {
                "eid": 1778385425,
                "timestamp": 1606326399,
                "actEnergyDlvd": 0.000,
                "actEnergyRcvd": 0.000,
                "apparentEnergy": 0.000,
                "reactEnergyLagg": 0.000,
                "reactEnergyLead": 0.000,
                "instantaneousDemand": 0.000,
                "activePower": 0.000,
                "apparentPower": 16.618,
                "reactivePower": -0.000,
                "pwrFactor": 0.000,
                "voltage": 126.916,
                "current": 0.130,
                "freq": 60.000
            },
            {
                "eid": 1778385426,
                "timestamp": 1606326399,
                "actEnergyDlvd": 0.000,
                "actEnergyRcvd": 0.000,
                "apparentEnergy": 0.000,
                "reactEnergyLagg": 0.000,
                "reactEnergyLead": 0.000,
                "instantaneousDemand": -0.000,
                "activePower": -0.000,
                "apparentPower": 18.019,
                "reactivePower": 0.703,
                "pwrFactor": 0.000,
                "voltage": 126.410,
                "current": 0.144,
                "freq": 60.000
            },
            {
                "eid": 1778385427,
                "timestamp": 1606326399,
                "actEnergyDlvd": 0.000,
                "actEnergyRcvd": 0.000,
                "apparentEnergy": 0.000,
                "reactEnergyLagg": 0.000,
                "reactEnergyLead": 0.000,
                "instantaneousDemand": 0.000,
                "activePower": 0.000,
                "apparentPower": 0.000,
                "reactivePower": 0.000,
                "pwrFactor": 0.000,
                "voltage": 0.000,
                "current": 0.000,
                "freq": 60.000
            }
        ]
    }
]

I see now that there was mention of these URLs in the Envoy-S data scraping blog that I missed previously