Use better detection for PC type

[dennisdegreef]

This PR should improve the model detection of different types.

Currently the type I test with is a P-type, which has the following production.json:

{
   "production" : [
      {
         "activeCount" : 12,
         "readingTime" : 12345678,
         "type" : "inverters",
         "whLifetime" : 12345678,
         "wNow" : 123
      }
   ],
   "storage" : [
      {
         "activeCount" : 0,
         "type" : "acb",
         "readingTime" : 0,
         "state" : "idle",
         "whNow" : 0,
         "wNow" : 0
      }
   ]
}

The device I am testing with is not my own, and thus I don't always have access to it. I am trying to fix homeassistant integration with this, and after using this patch and the previous PR, the values show up in HA.

Would be nice if somebody with a PC-type could test that this patch doesn't break anything.

[Pino7777]

Hi, I am new and want to help. My Envoy-S Metered has the following production.json:

Is this what you are looking for?

regards, Pino (NL)

[dennisdegreef]

Hi @Pino7777 Thanks for helping! :+1:

That looks like the same model that I am trying to integrate.

The model that I want some testing for, is called "PC" in the code. The models that we have, only provide "production" metrics, while the model that I am looking for should also have "consumption" metrics.

The change that I propose tries to be more specific in detecting if the production.json of those models have a "production" and "consumption" key in there.

return "production" in json and "consumption" in json

So that JSON file should look something like this:

{
    "production" : [
        ...
    ],
    "consumption": [
        ...
    ],
    ...
}

In the current code, it's trying to detect if it is a "PC" model, by checking if the production.json exists and if it has content, thus incorrectly detecting our type of models as "PC" type.

With this new check, it will detect our models as "P", as it should. That will improve the handling of our models because of one of my earlier changes (https://github.com/jesserizzo/envoy_reader/pull/15).

The only thing I am worried about, is that this change might break "PC" models, and since I don't have such a model available for testing, I'd like to know for sure that I don't break other people's working setups :)

[jesserizzo]

@dennisdegreef Can confirm that it does work on my Envoy with production and consumption. I wonder if we should change some names. If it is apparently possible to have an endpoint classified as 'PC' that doesn't have consumption.

[dennisdegreef]

@jesserizzo Good to hear that this works on a PC model!

If I understand you correctly, you want to change naming because there can be "PC" type that do not have "consumption"?

That is (at least for me and @Pino7777 it seems) incorrect. The type I have is a "P" type (Envoy S) that is incorrectly detected as a "PC" type. That detection is what this PR is attempting to fix.

With this patch, the library detects my device as a type P instead of PC

As far as I can guess, the type naming in this library are derived from P (production) and C (consumption). And additionally P0 for a legacy model.

[gtdiehl]

@dennisdegreef What firmware is your Envoy running?

[gtdiehl]

I always attributed the physical models to firmware revisions P0: < 3.9 P: >= 3.9 and < 4.0 PC: >= 4.0

The physical appearance of the P0 and P (the models referenced in the envoy_reader code) models are oval, but the P0 is the older IEMU-03 model whereas P is the ENV-120-01.

While the PC (the model referenced in the envoy_reader code) model is the newer Envoy-S which I've seen 3 variants of; Metered version that can monitor Production and Consumption, the Standard version which monitors just Production; and a Commerical version that can monitor Production and Consumption of 3-phase installations. Envoy-S Install Guide [Commerical Envoy](https://enphase.com/en-us/products-and-services/envoy/family

This information is what I have found online while trying to figure out the code when I wanted to make a change. And I don't know if the above will change when going outside the United States. 🤷‍♂

In any case, I did test the code change against my oval Envoy running R3.11.30, and it didn't break anything on my end! 👍

[dennisdegreef]

@gtdiehl It seems that my assumptions were then completely incorrect. Thanks for clarifying this.

I have an Envoy-S with only production values (see my production.json earlier in this PR), so that seems to be the Standard version then. The firmware (or "software version") that it is running is D5.0.34 (6e87e2) the software build is from 26 Jun, 2019 9:36 AM.

With this new insights, my patch will incorrectly detect "PC" types without consumption metrics as "P" then, so while the functionality technically will fix a lot of usecases, a renaming of models would be in place I think.

We have at least two options here if you ask me. 1) Merge this PR and release since it will solve a lot of scenarios for users and create a new issue to discuss a new naming strategy 2) Use this PR to discuss and implement the new naming strategy

I'd like to hear your thoughts on this @gtdiehl @jesserizzo

[gtdiehl]

@dennisdegreef You bring up some good points. I never correlated P/C to Production/Consumption. I always just tied P0, P, and PC to actual hardware, never to the functions

Though from my initial investigations into getting Production data from an Envoy-S running firmware D5.0.34 (6e87e2) that only has production info, I see issues between the /production.json and /api/v1/production web pages. As the proposed change will change the detect_model() to detect an Envoy-S with only Production capabilities from a PC to a P, which means the code will now get Production data from the /api/v1/production web page.

Here is the issue I see /production.json {"production":[{"type":"inverters","activeCount":175,"readingTime":1577666571,"wNow":31,"whLifetime":41262516}],"storage":[{"type":"acb","activeCount":0,"readingTime":0,"wNow":0,"whNow":0,"state":"idle"}]}

/api/v1/production

{
  "wattHoursToday": 28738,
  "wattHoursSevenDays": 225718,
  "wattHoursLifetime": 41262516,
  "wattsNow": 2
}

Notice the wNow and wattsNow are different between the two pages, 31 vs 2. I'll check these values out tomorrow when the sun is shining.

[jesserizzo]

@gtdiehl I believe api/v1/production only updates every 15 minutes, where as production.json updates pretty much continuously. That's probably the reason for that discrepancy.

[gtdiehl]

@jesserizzo Yeah your right. The /production.json page is updating every minute whereas the /api/v1/production page is updated at a much slower rate, about every 15minutes.

I don't have an Envoy-S with Production only, but if I did, I would want to have the 1minute granularity versus 15minutes.

[dennisdegreef]

I'm looking into implementing better model detection.

It would be nice if we have different sets of production.json for different types of PC models. These can aid in automated testing and preventing regressions. I can also write some documentation for these cases if we know them.

I am wondering why some of the production calls reference json["production"][1] instead of json["production"][0]?

The production call tries to read out [1] and falls back on [0]. (see https://github.com/jesserizzo/envoy_reader/blob/master/envoy_reader/envoy_reader.py#L119).

But other calls (like daily_production) don't seem to (see https://github.com/jesserizzo/envoy_reader/blob/master/envoy_reader/envoy_reader.py#L177).

Since my model only has [0], it breaks on this. And with my previous assumptions it would "work" as a P-model. However, that is indeed only 15-minute refresh instead of 1-minute.

For the seven_days_production, it tries to read out raw_json["production"][1]["whLastSevenDays"], however, it is not present in my production.json. But it is for some models? If it's not, while it is a PC model, do we want to fallback to /api/v1/production in those cases?

I will push a new commit, and let's figure out the best way of making all models getting the best metrics.

[rct]

I can supply data from the Envoy IQ running D5.0.34.

As indicated by type, on my Envoy:

• production[0].wNow is supplied by the inverters.
• production[1].wNow labeled eim is from the CTs.

I used to have a case that I think needs to be accounted for. There could be cases were the "eim" CT wNow value is present but isn't correct. (I couldn't use the value from the CTs because they were wired incorrectly. The CTs could be not installed, broken, etc. but the microinverters may still be given a useful value.)

Additionally in some cases it might be of interest to track both the inverter and CT supplied wNow. (Personally I'd like to store both if it isn't too expensive.)

Note: I have no consumption CTs because my Envoy is located away from the panel.

If there is any data I can supply to incorporate as part of the test cases, please let me know.

http://envoy.local/production.json

{
  "production": [
    {
      "type": "inverters",
      "activeCount": 27,
      "readingTime": 1577811765,
      "wNow": 1037,
      "whLifetime": 3537381
    },
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "production",
      "readingTime": 1577811974,
      "wNow": 1126.17,
      "whLifetime": 26225.198,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 9500.168,
      "vahLifetime": 34969.369,
      "rmsCurrent": 10.275,
      "rmsVoltage": 248.275,
      "reactPwr": 576.409,
      "apprntPwr": 1275.399,
      "pwrFactor": 0.88,
      "whToday": 32.198,
      "whLastSevenDays": 32.198,
      "vahToday": 41.369,
      "varhLeadToday": 0,
      "varhLagToday": 17.168
    }
  ],
  "consumption": [
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "total-consumption",
      "readingTime": 1577811974,
      "wNow": 1126.17,
      "whLifetime": 26225.198,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 9500.168,
      "vahLifetime": 0,
      "rmsCurrent": 10.546,
      "rmsVoltage": 248.591,
      "reactPwr": -576.409,
      "apprntPwr": 2621.669,
      "pwrFactor": 0.43,
      "whToday": 26225.198,
      "whLastSevenDays": 26225.198,
      "vahToday": 0,
      "varhLeadToday": 0,
      "varhLagToday": 9500.168
    },
    {
      "type": "eim",
      "activeCount": 0,
      "measurementType": "net-consumption",
      "readingTime": 1577811974,
      "wNow": 0,
      "whLifetime": 0,
      "varhLeadLifetime": 0,
      "varhLagLifetime": 0,
      "vahLifetime": 0,
      "rmsCurrent": 0.271,
      "rmsVoltage": 248.906,
      "reactPwr": 0,
      "apprntPwr": 33.875,
      "pwrFactor": 0,
      "whToday": 0,
      "whLastSevenDays": 0,
      "vahToday": 0,
      "varhLeadToday": 0,
      "varhLagToday": 0
    }
  ],
  "storage": [
    {
      "type": "acb",
      "activeCount": 0,
      "readingTime": 0,
      "wNow": 0,
      "whNow": 0,
      "state": "idle"
    }
  ]
}

http://envoy.local/api/v1/production

{
  "wattHoursToday": 5507,
  "wattHoursSevenDays": 74904,
  "wattHoursLifetime": 3538411,
  "wattsNow": 820
}

[dennisdegreef]

Additional information: With the Envoy-S with Production metrics that I have, I made a loop that polled the production.json each minute, and afterwards I parsed the results.

It seems that the {"production": { "type": "inverters" }} which I have, updates every 15 minutes. I haven't checked the /api/v1/production metrics though.

[rct]

Additional information: With the Envoy-S with Production metrics that I have, I made a loop that polled the production.json each minute, and afterwards I parsed the results.

It seems that the {"production": { "type": "inverters" }} which I have, updates every 15 minutes. I haven't checked the /api/v1/production metrics though.

@dennisdegreef - on my Envoy-IQ, the production.json[0].readingTime is usually < 300. In other words, I'm under the impression It gets updated every 5 minutes.

I believe this is a setting you can can control on the Envoy. There is a "bandwidth" setting that controls whether the Envoy reports every 5 or 15 minutes. However, I think you may have to use the Installer Toolkit app to change the setting.

You are probably aware, but on the off chance you haven't seen it this blog article, Envoy S Data Scraping has a lot of info. Unfortunately, you've got to wade through a lot of comments.

[gtdiehl]

@rct There must be some configuration in the Envoy-S. I have looked at the readingTime of three. One updates whenever the production.json pages is polled (that's why I thought it was 1 minute in my previous post) and another two update every 15 minutes.

The Bandwidth setting in the Installer login is how often the information is sent to Enphase. In Standard mode the Envoy sends out data every 5minutes. In Low-bandwidth mode the Envoy sends out 15minute interval data 4 times a day.

Here's a screenshot:

Atleast for the 2 Envoy-S that updated the readingTime and wNow attributes of the production section in /production.json every 15minutes, they were both set to Standard mode.

As a sidenote, I noticed that the readingTime attribute resets to 0 when the inverters stop producing energy

[gtdiehl]

I was thinking and wanted to circle back to the original problem. Given that there are various Enphase Envoy models with various capabilities why don't we expand the number of monitored conditions from 9 to 13?

Currently, these are monitored conditions as they appear in Home Assistant

We can expand the Inverter and CT Production info to produce these monitored conditions

@dennisdegreef You mentioned the whLastSevenDays. I just noticed this while creating the spreadsheet. The whToday and whLastSevenDays Production values appear in the eim (CT) section of /production.json. In my spreadsheet, the Inverter Daily and 7 Day values would have to come from /api/v1/production

[jesserizzo]

I'm going to merge this in, we can discuss name changes and adding more monitored conditions elsewhere.

[dennisdegreef]

Thanks for merging this! Could this perhaps also get a new release tagged?

[jesserizzo]

@dennisdegreef Sorry about the delay, totally forgot. It's 0.13.0

[tdn131]

@dennisdegreef Sorry about the delay, totally forgot. It's 0.13.0

Does this mean this PR going to integrated in Home Assistant? I don't see it yet in the new release-106. https://www.home-assistant.io/blog/2020/02/26/release-106/ Thanks!

[damienclaveau]

@gtdiehl @jesserizzo Hello gentlemen, in the past months, you did such dependency upgrades in HA. Do you think you could handle such bump to 0.13.0 again ?

[Pino7777]

@gtdiehl @jesserizzo What is the status of this? When to expect it in an release? What can I do to help?