search

briancmpbll / home_assistant_custom_envoy

177 stars 77 forks source link

Feature Request: add "Number of Microinverters Online" #170

Closed kchiem closed 7 months ago

kchiem commented 11 months ago

Hello,

I've been dealing with some long-standing communications issues and would find it useful if the reported number of microinverters the envoy is connected to could be tracked. This number varies throughout the day for me.

catsmanac commented 11 months ago

Hi @kchiem, not sure where the get the information from at this moment. Some options come to mind.

There is a activecount in the production report (/production) on the Envoy, but have no idea if that number would change if a microinverter is not tracked. To validate if that number is useful and changing, download a diagnostic report afbeelding and open it. Look for Endpoint-production and see what the active count is at a good and bad moment in the day. If that number changes there's something to work with

 "{\"production\":[{\"type\":\"inverters\",\"activeCount\":24,\"readingTime\":1698217399,

What is the inverter data doing if one is not tracked? Is there a number available? Is timestamp of the value changing?

In the same report Endpoint-production_inverters is the list of inverters reported in last data collection. You could see if all are include there or not.

The timestamp for the inverter values comes from the Envoy, so if that freezes it's could be an indication for an outage, if it freezes in this case. You could use this in HA template to like calculated frozen panels if last update is > 5 min or so.

kchiem commented 11 months ago

My envoy is the oldest -R version. I'm not seeing an activeCountin Endpoint-production:

        "Endpoint-production": "    <!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\">\n    <html lang=\"en\">\n    <!-- START HEAD CONTENT -->\n    <head>\n      <title>production</title>\n      <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n      <!-- include style sheet -->\n      <link rel=\"styleSheet\" type=\"text/css\" href=\"/include/css/style.css\" >\n      <style type=\"text/css\">.nav { width: 190px; font-size: 10px; padding: 10px; margin: 10px;float: left; background: #D8DBD4; color: #213d30; border: 2px solid #333; }\n\t.navholder { width: auto; height: auto; float: left; margin: 10px 0px 0 0 }\n\t.tbl_cap { font-size:8pt; background-color:#DD4F05; color:white; border-style:solid; border-width:1px; border-color:white; }\n\t.tbl_bod { font-family:arial; font-size:8pt; color:#213d30; background-color:#D8DBD4; border-style:solid; border-color:white; border-width:1px; }\n\t.tbl_bod_right { font-family:arial; font-size:8pt; color:#213d30; background-color:#D8DBD4; border-style:solid; border-color:white; border-width:1px; text-align:right }\n\t.tbl_hdr { font-size:8pt; background-color:#CCCCCC; border-style:solid; border-width:1px; border-color:white; color:#213d30; }\n\t.clearer { clear: both; font-size: 1px; }\n\t.outer { width: auto; height: auto; padding: 0px; font-size: 10px }\n\t.table1 { width: 100%; margin-left: auto; margin-right: auto;border: 1px solid #CCCCCC }\n\t.table2 { width: 100%;text-align: center;margin-left: auto; margin-right: auto;border: 1px solid #CCCCCC }\n\t.table3 { width: 720px;margin-left: auto; margin-right: auto;border: 1px solid #CCCCCC }\n\t.div_params { height: 190px; width: 380px; overflow: auto;margin-left: auto; margin-right: auto; }\n\t.div_edit_params { height: 230px; width: 470px; overflow: auto;margin-left: auto; margin-right: auto; }\n\t.div_outer_box { width: 720px;margin-left: auto; margin-right: auto;border: 1px solid #FFFFFF }\n\t.txt_expl_center { font-size: 12px; text-align:center;height: auto; border: 1px solid #CCCCCC;margin-top: 5px; margin-left: 5px; margin-right: auto; }\n\t.label_text_right { text-align: right; border: 1px solid #CCCCCC;font-size: 10px; }\n\t.label_text_left { text-align: left; border: 1px solid #CCCCCC;font-size: 10px; }\n\t.label_text_left_alert { text-align: left; color: #7d3020; border: 1px solid #CCCCCC;font-size: 12px; }\n\t.link_cell_right { font-size: 10px;height: auto; border: 1px solid #CCCCCC;text-align: right;margin-top: 5px; margin-left: auto; margin-right: 25px; }\n\t.link_cell_left { font-size: 10px;height: auto; border: 1px solid #CCCCCC;text-align: left;margin-top: 5px; margin-left: 25px; margin-right: auto; }\n\t.link_cell_right_out { font-size: 12px; padding: 15px;height: auto; border: 1px solid #CCCCCC;text-align: right;margin-top: 5px; margin-left: auto; margin-right: 25px; }\n\t.link_cell_left_out { font-size: 12px;height: auto; border: 1px solid #CCCCCC;text-align: left; padding: 15px;margin-top: 5px; margin-left: 25px; margin-right: auto; }\n\t.ns_fldset { height: auto; width: 720px; border: 1px solid #7d3020; background: #CCCCCC; padding: 15px; }\n\t.ns_fldset_inner { border: 1px solid #7d3020; background: #CCCCCC; }\n\t.enp_legend { text-align: left; border: 1px solid #7d3020; background: #FFD401; }\n\t.input_left { border: 1px solid #CCCCCC; text-align: left; }\n\t.grid_config { float: left; margin-left: 50px; width: 830px; text-align: center; }\n\t.swimgnav { width: 160px; height: auto; font-size: 10px; padding: 10px; margin: 10px;float: left; background: #D8DBD4; color: #213d30;border: 2px solid #333; }\n\t.frames { clear: right; width: auto; padding: 15px; height: auto; margin-left: 200px; margin-right: auto; }\n\t.tbl_cap_left { font-size:8pt; background-color:#DD4F05; color:white; border-style:solid; border-width:1px; border-color:white; text-align:left; }\n\t.tbl_cap_center { font-size:8pt; background-color:#DD4F05; color:white; border-style:solid; border-width:1px; border-color:white; text-align:center; }\n\t.tbl_bod_left { font-family:arial; font-size:8pt; color:#213d30; background-color:#D8DBD4; border-style:solid; border-color:white; border-width:1px; text-align:left; }\n\t.tablePaged { float: left;width: 100%;text-align: center;margin-left: auto; margin-right: auto;border: 1px solid #CCCCCC }\n\t.table.dataTable { margin: 0 auto; clear: both; width: 100%; }\n\t.table.dataTable thead th { padding: 3px 18px 3px 10px;border-bottom: 1px solid black;font-weight: bold;cursor: pointer;*cursor: hand; }\n\t.table.dataTable tfoot th { padding: 3px 18px 3px 10px;border-top: 1px solid black;font-weight: bold; }\n\t.table.dataTable td { padding: 3px 10px; }\n\t.table.dataTable td.center, table.dataTable td.dataTables_empty { text-align: center; }\n\t.table.dataTable tr.odd { background-color: #E2E4FF; }\n\t.table.dataTable tr.even { background-color: white; }\n\t.table.dataTable tr.odd td.sorting_1 { background-color: #D3D6FF; }\n\t.table.dataTable tr.odd td.sorting_2 { background-color: #DADCFF; }\n\t.table.dataTable tr.odd td.sorting_3 { background-color: #E0E2FF; }\n\t.table.dataTable tr.even td.sorting_1 { background-color: #EAEBFF; }\n\t.table.dataTable tr.even td.sorting_2 { background-color: #F2F3FF; }\n\t.table.dataTable tr.even td.sorting_3 { background-color: #F9F9FF; }\n\t.dataTables_wrapper { position: relative; clear: both; *zoom: 1; }\n\t.dataTables_length { float: left; }\n\t.dataTables_filter { margin: 0px 0px 2px 0px; float: right; text-align: right; }\n\t.DTTT_button_text { margin: 0px 0px 0px 3px; float: right; text-align: right; }\n\t.dataTables_info { clear: both; float: left; }\n\t.dataTables_paginate { float: right; text-align: right; }\n\t.paging_full_numbers { height: 22px; line-height: 22px; }\n\t.paging_full_numbers a:active { outline: none }\n\t.paging_full_numbers a:hover { text-decoration: none; }\n\t.paging_full_numbers a.paginate_button,.paging_full_numbers a.paginate_active { border: 1px solid #aaa;-webkit-border-radius: 5px;-moz-border-radius: 5px;border-radius: 5px;padding: 2px 5px;margin: 0 3px;cursor: pointer;*cursor: hand;color: #333 !important; }\n\t.paging_full_numbers a.paginate_button { background-color: #ddd; }\n\t.paging_full_numbers a.paginate_button:hover { background-color: #ccc;text-decoration: none !important; }\n\t.paging_full_numbers a.paginate_active { background-color: #99B3FF; }\n\t.paging_full_numbers a.paginate_button_disabled { border: 1px solid #aaa;-webkit-border-radius: 5px;-moz-border-radius: 5px;border-radius: 5px;padding: 2px 5px;margin: 0 3px;cursor: default;opacity: 0.6;color: gray; }\n\t.dataTables_processing { position: absolute;top: 50%;left: 50%;width: 250px;height: 30px;margin-left: -125px;margin-top: -15px;padding: 14px 0 2px 0;border: 1px solid #ddd;text-align: center;color: #999;font-size: 14px;background-color: white; }\n\t.sorting { padding: 2px 15px;background-clip: padding-box;background-origin: border-box;background-image: url('/images/sort_both.png');background-repeat: no-repeat;background-position: right center;background-color: #DD4F05; }\n\t.sorting_asc { padding: 2px 15px;background-clip: padding-box;background-origin: border-box;background-image: url('/images/sort_asc.png');background-repeat: no-repeat;background-position: right center;background-color: #DD4F05; }\n\t.sorting_desc { padding: 2px 15px;background-clip: padding-box;background-origin: border-box;background-image: url('/images/sort_desc.png');background-repeat: no-repeat;background-position: right center;background-color: #DD4F05; }\n\t.sorting_asc_disabled { padding: 2px 15px;background-clip: padding-box;background-origin: border-box;background-image: url('/images/sort_asc_disabled.png');background-repeat: no-repeat;background-position: right center;background-color: #DD4F05; }\n\t.sorting_desc_disabled { padding: 2px 15px;background-clip: padding-box;background-origin: border-box;background-image: url('/images/sort_desc_disabled.png');background-repeat: no-repeat;background-position: right center;background-color: #DD4F05; }\n\t.table.dataTable th:active { outline: none; }\n\t.p.t_fixed_header_scroll_inner { width: 100%;height: 200px; }\n\t.div.t_fixed_header_scroll_outer { top: 0; left: 0; visibility: hidden; width: 200px; height: 150px; overflow: hidden; }\n\t.div.t_fixed_header { position: relative;margin: 0;width: 100%; }\n\t.div.t_fixed_header * { margin: 0;padding: 0; }\n\t.div.t_fixed_header table.head { position: relative; }\n\t.div.t_fixed_header table { width: 100%;table-layout: fixed;empty-cells: show;border-collapse: collapse;border-spacing: 0; }\n\t.div.t_fixed_header table.headtable td.fillScrollbar { font-size: 0px;-o-text-overflow: inherit;text-overflow: inherit; }\n\t.div.t_fixed_header tr th, div.t_fixed_header tr td { overflow: hidden;white-space: nowrap;-o-text-overflow: ellipsis;text-overflow: ellipsis; }\n\t.div.t_fixed_header.t_fixed_header_wrap tr th, div.t_fixed_header.t_fixed_header_wrap tr td { white-space: normal; }\n\t.div.t_fixed_header div.body { overflow-x: auto;padding: 0;width: 100%; }\n\t.div.t_fixed_header_main_wrapper { position: relative;overflow: hidden; }\n\t.div.t_fixed_header_main_wrapper_child { position: relative;width: 100% }\n\t.div.t_fixed_header_main_wrapper.ui { border-width: 1px 1px 1px 1px;border-style: solid;font-weight: normal;font-size: 1.1em; }\n\t.div.t_fixed_header_main_wrapper.ui  div.t_fixed_header_caption { text-align: center;font-weight: bold;padding: 0 5px;height: 30px;line-height: 30px;border-width: 0 0 1px 0;border-style: solid; }\n\t.div.t_fixed_header_main_wrapper.ui  div.t_fixed_header_caption.toggle { border-width: 0; }\n\t.div.t_fixed_header div.headtable { border: 0;overflow-x: hidden;overflow-y: auto; }\n\t.div.t_fixed_header.ui .body { border-width: 1px 0 0 0;border-style: solid;background-image: none; }\n\t.div.t_fixed_header.ui .headtable th { padding: 5px;text-align: center;border-width: 0 1px 0 0;border-style: solid; }\n\t.div.t_fixed_header.ui .headtable .hover { cursor: pointer;height: 16px; }\n\t.div.t_fixed_header.ui .headtable th span.ui-resize { background: transparent;cursor: col-resize;display: inline;float: right;height: 20px;margin: -3px -5px -5px 0;width: 5px;z-index: 10000; }\n\t.div.t_fixed_header.ui div.ui-resize-ghost { display: none;position: absolute;top: 0;width: 2px;border: none; }\n\t.div.t_fixed_header.ui .body tr { border: 0px; }\n\t.div.t_fixed_header.ui .body td { padding: 5px;text-align: left;line-height: 15px; }\n\t.div.t_fixed_header.ui .body tr td { border-width: 0 1px 1px 0;border-style: solid;background: none; }\n\t.div.t_fixed_header.ui .body tr:last-child td { border-bottom-width: 0px; }\n\t.div.t_fixed_header.ui.default .headtable th:last-child, div.t_fixed_header.ui.default .body td:last-child { border-right-width: 0px; }\n\t.div.t_fixed_header.ui .body td.last_td, div.t_fixed_header.ui .headtable th.last_td { border-right-width: 0px; }\n\t.div.t_fixed_header_main_wrapper.ui div.pager { border-width: 0 0 0 0;padding: 4px 2px 2px 3px;overflow: hidden;background-image: none; }\n\t.div.t_fixed_header_main_wrapper.ui div.pager .button { float: right;padding: 4px 0;cursor: pointer; }\n\t.div.t_fixed_header_main_wrapper.ui div.pager .button.noborder { border-right-width : 0px; }\n\t.div.t_fixed_header_main_wrapper.ui div.pager .button span { margin: 0 4px; }\n\t.div.t_fixed_header_main_wrapper.ui div.pager div.page_infos { height: 27px;line-height: 27px;float: right;margin-right: 10px;font-weight: normal; }\n\t</style>\n      <!-- include JS libraries -->\n      <script type=\"text/JavaScript\">\n      <!--\n      function refreshPage(period) {\n        setTimeout(\"document.forms['tpm_status'].submit()\", period);\n      }\n      //   -->\n      </script>\n      \n    </head>\n    <!-- END HEAD CONTENT -->\n      <body>\n    <!-- START TOP NAV CONTENT -->\n      <form action=\"/production\" method=\"post\">\n      <div class=\"nav_header\">\n      <input type=\"hidden\" name=\"fname\" value=\"nav_header\" >\n      <input type=\"hidden\" name=\"profile\" value=\"0\" >\n      <input type=\"hidden\" name=\"regSpec\" value=\"0\" >\n      <input type=\"hidden\" name=\"country\" value=\"0\" >\n      <input type=\"hidden\" name=\"regSpecName\" value=\"\" >\n      <input type=\"hidden\" name=\"profileName\" value=\"\" >\n      <table width=\"100%\" style=\"border-style:none;\">\n        <tr>\n          <td class=\"hdr_line\" valign=\"middle\">\n              <a href=\"http://enphase.com/en/\" title=\"Visit the Enphase Energy website\">\n              <img src=\"/images/enphase-logo.png\"\n              style=\"background:white; text-align:left; height:50px;\"\n              alt=\"Enphase Energy, Inc.\">\n              </a>\n              </td>\n          <td class=\"hdr_line\" valign=\"middle\" colspan=\"4\">Envoy Serial Number: 121121741109</td>\n          <td class=\"hdr_line\" valign=\"middle\">\n              <a href=\"http://enlighten.enphaseenergy.com?locale=en\" title=\"Visit the Enlighten website\">\n              <img src=\"/images/enlighten_logo.png\"\n              style=\"background:white; text-align:right; height:40px;\"\n              alt=\"Enlighten website\">\n              </a>\n              </td>\n        </tr>\n        <tr>\n          <td class=\"hdr_line\"><a href=\"/home?locale=en\">Home</a></td>\n          <td class=\"hdr_line\"><a href=\"/event?locale=en\">Events</a></td>\n          <td class=\"hdr_line\"><a id=\"nav_header_production\" href=\"/production?locale=en\">Production</a></td>           \n          <td class=\"hdr_line\"><a id=\"nav_header_inventory\" href=\"/inventory?locale=en\">Inventory</a></td>\n          <td class=\"hdr_line\"><a id=\"nav_header_admin\" href=\"/admin/home?locale=en\">Administration</a></td>\n          <td class=\"hdr_line\"></td>\n          <td class=\"hdr_line\" align=\"left\">Language\n            <select id=\"locale\" name=\"locale\" onchange=\"this.form.submit()\">               <option value=\"it\">Italiano</option><option value=\"de\">Deutsch</option><option value=\"es\">Espa\u00f1ol</option><option value=\"en\" selected=\"selected\" >English</option><option value=\"nl\">Nederlands</option><option value=\"fr\">Fran\u00e7ais</option>\n            </select>\n          </td>\n        </tr>\n      </table>\n      </div>\n      </form>\n      <HR>\n    <!-- END TOP NAV CONTENT -->\n    <!-- START MAIN PAGE CONTENT -->\n      <h1>System Energy Production</h1>\n      <div style=\"margin-right: auto; margin-left: auto;\"><table>\n      <tr><td colspan=\"3\">System has been live since\n                             <div class=good>Tue Nov 08, 2011 01:59 PM PST</div></td></tr>\n      <tr><td>Currently</td>    <td> 3.14 kW</td></tr><tr><td>Today</td>     <td> 3.45 kWh</td></tr><tr><td>Past Week</td>    <td>  240 kWh</td></tr><tr><td>Since Installation</td>    <td>  165 MWh</td></tr>\n      </table><br></div>\n    <!-- END MAIN PAGE CONTENT -->\n    <!-- START BOTTOM NAV CONTENT -->\n    <HR>\n    <div style=\"margin-left: auto; margin-right:auto; width: 100%; text-align: center; \">\n    &copy; 2007-2020, [e] Enphase Energy, Inc. All rights reserved. |\n    <a href=\"http://www.enphaseenergy.com/licenses\">Licenses</a>\n    <!-- END BOTTOM NAV CONTENT -->\n    <br><br>\n    </div>\n    </body>\n    </html>\n",

My /production page looks like this, and I think it's being html scraped:

ScreenClip

The online microinverter count is available in /home, which looks like this:

ScreenClip

The relevant html is:

            <tr><td>Number of Microinverters Online</td>
                <td>35</td></tr>
catsmanac commented 10 months ago

Ah, I see. Technically that can be done. I would need the HTML content of the /home page for my envoy simulator to be able to test such a change.

There is another alternative that will be quicker to implement and tailor to what you need. In HACS there is a custom integration multiscrape. That will allow you to get that data from the web page directly and is not too hard to get going.

afbeelding

After installing it using HACS, you need to add a scrape definition to configuration.yaml in the /config folder and restart HA.

Below an example to read currently kW from the production page

multiscrape:
  - name: Envoy Production Scraper
    resource: http://envoy/home
    scan_interval: 60
    sensor:
      - unique_id: Envoy_current_production
        name: Envoy Current production
        select: "body > div > table > tr:nth-child(2) > td:nth-child(2)"
        value_template: '{{ (value.split("k")[0]) }}'
        unit_of_measurement: "kW"

The select: defines the search pattern in the webpage. I can assist with that if you provide the html content of the home page that you share a screenshot of above.

For starters one could consider:


multiscrape:
  - name: Envoy Production Scraper
    resource: http://envoy/production
    scan_interval: 60
    sensor:
      - unique_id: Envoy_inverters_configured
        name: Envoy Inverters Configured
        select: "body > div > table > tr:nth-child(4) > td:nth-child(2)" <-- needs change
        value_template: '{{ (value) }}' <-- needs change
      - unique_id: Envoy_inverters_online
        name: Envoy Inverter Active Count
        select: "body > div > table > tr:nth-child(5) > td:nth-child(2)" <-- needs change
        value_template: '{{ (value) }}' <-- needs change
kchiem commented 10 months ago

Here's the full html: envoy_home.zip

Thanks for the pointer to multiscrape. I'll check it out.

catsmanac commented 10 months ago

Had a quick look at the scrape selection for the home page and it worked with below configuration. The force_update is optional, with it set True it will always write a value to HA instead of only writing when the value or status changed.

multiscrape:
  - name: Envoy Scraper
    resource: http://envoy/home
    scan_interval: 60
    sensor:
      - unique_id: Envoy_inverters_configured
        name: Envoy Inverters Configured
        select: "body > table > tr > td:nth-child(2) > table > tr:nth-child(4) > td:nth-child(2)"
        value_template: "{{ (value) }}"
        state_class: measurement
        force_update: True
      - unique_id: Envoy_inverters_online
        name: Envoy Inverters Online
        select: "body > table > tr > td:nth-child(2) > table > tr:nth-child(5) > td:nth-child(2)"
        value_template: "{{ (value) }}"
        state_class: measurement
        force_update: True

afbeelding

If you want more items to display, add more -unique_id sections and change the tr:nth-child(4) to another line number in the table.

catsmanac commented 10 months ago

Hi @kchiem, did the multiscrape work and are you ok now?

I used your data to update my Envoy simulator and added some code to my test version to grab the active count, just for legacy envoy types. That worked so far. I've you're still into this, next step would be for you to test in with your system.

If you are fine with the multiscrape, that's fine too, my envoy simulator needed a long overdue cleanup anyway and this was as good a moment as others.

Screenshot of my local dev system using the legacy simulator. afbeelding

kchiem commented 10 months ago

Hello, sorry about the late response. I got busy over the weekend.

I had gotten multiscrape working before that based on your initial suggestion. Although I have to say that enabling debug logging and log_response for multiscrape wasn't very helpful. Instead I installed beautifulsoup4 and was able to come up with a working select statement. On this, is the value_template: "{{ (value) }}" necessary? It seems to return the numbers without that line.

If you have the integration pulling the online inverter count now, I'd love to test it. But how?

catsmanac commented 10 months ago

Had the same experience finding the right pattern for multiscrape, just copying from the browser as described didn't work and I ended up adding debug statement to the integration with various test patterns.

Don't think the value template is needed, leftover from the example.

The test version of the integration can be downloaded in HACS by adding custom library https://github.com/catsmanac/home_assistant_custom_envoy. It will show as DEV-TEST. When you have both DEV and DEV-TEST in hacs you can switch between these by re-downloading from the repository you want. Make sure to enable the show beta test option switch on DEV-TEST.

catsmanac commented 7 months ago

Released in v0.0.20.