API disabled?

[SoulGit]

Hi Jeff -

Is this API still available for you? I've been using it for about two years and just yesterday I am now getting access denied and with a new hostID field being returned in the the JSON. Coincidentally I see you pushed an update to the repo 13 hours ago - so wondering if you are seeing the same thing that I am. The website also no longer returns any REST info . . . at least for me. Thanks!

[jeffkowalski]

Yes, @SoulGit, I see the same thing! In fact I was just looking at the problem. The last time the current API worked for me was 2019-08-13 05:00:00 Pacific Time. Today, I get 403 Forbidden When I logged into their monitoring site https://monitor.us.sunpower.com/#!/dashboard yesterday, I recall seeing a message about an upgrade in progress.
My guess is that they changed the monitoring website and backend substantially. This will require a bit more sniffing of the XHRs to decipher.

[jeffkowalski]

Btw, the push from yesterday should have been very benign. I bumped the required version of yard to accommodate a security update. Locally, I hadn't changed it - just in github, and my local sunpower script is failing anyway. So I don't think the yard update is to blame.

[SoulGit]

Apologies my message wasn't clear. I'm sure your commit had nothing to do with the issue - in fact - I just used the URLs from your repo as a launch point to create my own python plugin for Indigo Domotics about two years ago. I starting receiving errors last night (8/12/19 ~7:30pm Pacific) when my prior auth was invalidated (and I had plenty of time left on the session).

The main portal does mention a maintenance window from 6 to 8am Pacific today, but that seems to have come and gone. From what I can see, they moved the service over to AWS Cloudfront and the endpoint is no longer configured to accept POST requests necessary for auth.

Time to get out my Saint Bernard and figure out what is going on . . .

[SoulGit]

Looks to be a new auth URL: https://elhapi.edp.sunpower.com/v1/elh/authenticate I am once again able to get a token. Now need to figure out the rest of the endpoints.

[SoulGit]

Looking good Jeff. I also spent time looking through the XHRs and have most of my plugin working again. Where did you find the /components endpoint? I couldn't seem to find that one and what I am actually looking for is the replacement for the old getACPVModuleInfo. That endpoint listed moduleID, tilt, XY position and other useful info. That's the last piece I need to get working again . . . Let me know if you've come across anything like that in your research. Thanks!

[jeffkowalski]

@SoulGit, I found the /components endpoint in app.min.js, injected by the dashboard at https://monitor.us.sunpower.com/#!/dashboard

I have been really disappointed in two things in the new API:

1. The /power endpoint's response includes only CurrentConsumption and CurrentProduction, but no timestamp, so it's not discernable what time the measurement was taken.
2. Related to #1, the reporting aggregation seems to be terribly delayed. I'm seeing ~4 hour delays in the power I know I generate to what is eventually reported as "CurrentProduction". Do you see the same delay? Really, I think this issue can't be closed without resolution to these points. I've made some progress in transferring to the new API, but I'm disappointed in the result. I chatted with sunpower on this point (case 01702972), but they said to wait for the results to appear, so I don't think they're understand the underlying issue in the reporting.

As to the fields from getACPVModuleInfo - I don't see them in /components. I think those data may be supplied by a different API still accessible from the "partner" Flash-based site at https://www.sunpowermonitor.com/partner/partner.aspx# (use your usual username/password). If you find more here, please let me know! ;) I take pull requests, you know ;)

Thanks for your interest and help.

[SoulGit]

Ah - I missed looking through app.min. Wish there was more in there, but seems ACPV is no longer available. I was reading around and saw a post that the individual panel data leads to additional support calls as folks tend to believe something is awry when all panels are not producing similarly. However, I never saw individual panel data in the old portal, and the average user is not consuming from the API, so I don't see why it needed to be removed or obfuscated in the new API.

I agree with you Jeff that the approach and the new API is unacceptable. There are folks who simply like to collect data on a system they invested in. I have no interest to contact SunPower to let them know panel 8 is producing four watts less than panel 10. I can however use this data to analyze trends over time and know which trees need trimming on my property.

I just got my plugin working again yesterday and was so pleased that I didn't actually analyze any data. Sure enough, you are correct, the reported values are way off. The sun has been shining brightly over here for two hours and I still see 0 for CurrentProduction. Unacceptable and useless as there is no way to tie this data to other metrics (such as a rainforest consumption unit), especially considering there is no timestamp :(

In your case to SunPower, do you mentioned that you are using the API directly or rather not seeing data in their portal? I was under the impression that direct use of the API was not supported, but if that is not the case then I think we can definitely run this up the chain. I'd be happy to use the partner API if that provides the necessary data, although from a quick review there isn't much in the way of XHR requests as it's all happening from a VERY SLOW flash applet.

BTW - my plugin enabled the use of the old Logout functionality. I have adapted this call to the new endpoint and didn't see it in your updates. If you send a DELETE request to /session with your token in the SP-CUSTOM header, it will log you out.

[SoulGit]

Ugh - they also touted the new portal to be faster and more reliable than the old. I am now seeing extensive delays in response with many timeouts . . . { "message": "Endpoint request timed out" }

[jeffkowalski]

@SoulGit, thank you for your continued interest. Thank you too for informing me about logout method. I'll add that to my notes. In reporting the case to Sunpower, I did not mention I was making direct use of the API, since the API is used by the https://monitor.us.sunpower.com/#!/dashboard and the "Now Producing" panel thereupon shows the same extreme data latency. Regrettably, one could argue that if the data is "mostly right" and uniformly delayed, you could still use it for seeing how trees affect the panels season/season. I do deeply miss the older API, two revs back from this one, where the production numbers were indeed real-time, and I could see clouds pass over the panels on a breezy day.
I wonder if you dug into https://www.sunpowermonitor.com/partner/partner.aspx# at all to see it's data exchange. There I see my panels diagrammed on my roof, so the data should have the position information you seek. Could there be a different power API used there as well?

[sashkab]

Looks like recent change:

https://monitor.us.sunpower.com/v09272019065330/envcfg.js

(function (w) {
    w.__env = {
        SP_ELH_VERSION: 'SP_ELH_VERSION_2', // SP_ELH_VERSION must match SP_ELH_VERSION lambda variable in ElhApi cloudformation
        ELH_API_USER: w.__envapidom + 'v1/elh/user/',
        ELH_API_ADDRESS: w.__envapidom + 'v1/elh/address/',
        ELH_API_ASYNC_RESULT: w.__envapidom + 'v1/elh/asyncResult',
        ELH_API_PASSWORD_RESET: w.__envapidom + 'v1/elh/passwordreset/',
        ELH_API_AUTHENTICATE: w.__envapidom + 'v1/elh/authenticate',
        ELH_API_SESSION: w.__envapidom + 'v1/elh/session',
        ELH_API_AUTHENTICATEKIOSK: w.__envapidom + 'v1/elh/authenticatekiosk',
        ELH_API_FEEDBACK: w.__envapidom + 'v1/elh/processFeedback',
        ELH_API_CHECK_EMAIL: w.__envapidom + 'v1/elh/register/checkEmail',
        ELP_API_VALIDATE_SN: w.__envapidom + 'v1/elh/register/validateSerial/',
        ELH_API_CREATE_ACCOUNT: w.__envapidom + 'v1/elh/register/createAccount',
        ELH_API_SEND_ACCOUNT_NOTIFICATION: w.__envapidom + 'v1/elh/register/sendAccountNotification',
        ELH_API_VALIDATE_EMAIL: w.__envapidom + 'v1/elh/register/validateEmail',
        ELH_API_COLLECTOR: w.__envapidom + 'v1/elh/wifi/collector',
        ELH_API_DEVICE_STATUS: w.__envapidom + 'v1/elh/wifi/status',
        ELH_API_WIFI_NETWORKS: w.__envapidom + 'v1/elh/wifi/networks',
        ELH_API_SET_WIFI_NETWORK: w.__envapidom + 'v1/elh/wifi/setNetwork',
        ELH_API_SET_WIFI_NETWORK_STATUS: w.__envapidom + 'v1/elh/wifi/setNetwork/status',
        ELH_API_TOU: w.__envapidom + 'v1/elh/tou',
        ELH_API_SET_TOU: w.__envapidom + 'v1/elh/setTou',
        ELH_API_REFERRAL: w.__envapidom + 'v1/elh/referral'
    };
}(this));

SP_ELH_VERSION changed from SP_ELH_VERSION_1 into SP_ELH_VERSION_2

[sashkab]

As to the fields from getACPVModuleInfo - I don't see them in /components.

Did you inspect /address/[addressID]? Use addressId you receive after authentication.

[SoulGit]

Thanks @sashkab - I missed your first response a week ago. Yes, it does appear the API has changed. Under v1 I did find the ACPV components (https://elhapi.edp.sunpower.com/v1/elh/address/{{ addressId }}/components?async=false), which created a JSON device discovery report with all of the components of the system (however the serial numbers for individual panels were not provided, nor was the power output.) Now, when I run the same query, the response only contains a single entry for the PV Supervisor." Do you have a working URL that provides ACPV info?

[sashkab]

My records show that I saw it using /address/id/components endpoint, but no longer can see it there.

[SoulGit]

Yes, looks like they have removed it.

[sashkab]

I guess, we should try to utilize information from the comment 84 without querying API, but instead connecting to PVS directly.

[jeffkowalski]

The truly disappointing part is that the PVS sends useful, timestamped data to Sunpower, where it is garbled and rendered meaningless, yet accessible by the version 2 api. I mean the result is not useful to us - how could it possibly by useful to Sunpower? It just feels like a total failure. Sorry to vent...

On Tue, Oct 8, 2019 at 11:58 AM Aleks Bunin notifications@github.com wrote:

I guess, we should try to utilize information from the comment 84 https://www.solarpaneltalk.com/forum/solar-panels-for-home/solar-panel-system-equipment/19587-mirroring-intercepting-sunpower-monitoring-traffic/page6 without querying API, but instead connecting to PVS directly.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jeffkowalski/sunpower/issues/4?email_source=notifications&email_token=AAAN4MBS7SEITRR4WYGWLEDQNTJ7BA5CNFSM4ILYRPJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAVHSXQ#issuecomment-539654494, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAN4MFF7HP6OTKT6OEQVXTQNTJ7BANCNFSM4ILYRPJA .

[sashkab]

PVS sends data to them; they store and aggregate and provide it via non-public API which is all they want to share with their customers. Another thing I found, that they grab and store all wifi access points information available nearby. What else they capture?

[sashkab]

Btw, does anyone understands what ELH_API_TOU endpoint does? I wasn't able to to cause web-app to ping it.

[sashkab]

New endpoint:

ELH_API_ADDRESS_V2: w.__envapidom + 'v2/elh/address/',

[jsdanie1]

Is there a way to test this in Postman? Trying to follow this thread a little bit. and a little confused on which URL to follow.

https://monitor.us.sunpower.com/v2/elh/authenticate using this url i at least get a response back saying access denied.

using the url everybody talking about on this thread: https://elhapi.edp.sunpower.com/v1/elh/authenticate i'm getting "Endpoint has not been found, Code EDP4041"

Since the first url seems to get me further, what is the authentication parameters or correct way to do the authorization via Postman if anybody has tried that?

[sashkab]

i'm getting "Endpoint has not been found, Code EDP4041"

Because you're sending GET instead of POST.

[jsdanie1]

i'm getting "Endpoint has not been found, Code EDP4041"

Because you're sending GET instead of POST.

Which url were you referring to? Iv tried both POST AND GET. The POST is whats giving me the "Endpoing has not been found" error. Technically i get the error on GET as well. Which is why i thought maybe i'm doing the params wrong or using the wrong API.

[sashkab]

idk. My code is like this and works. I think you need to say that you send data as "'application/json" and accept "'application/json".

import requests 

headers = {'Content-Type': 'application/json',
           'accept': 'application/json',
           'User-Agent': 'Mozilla/1.22 (compatible; MSIE 2.0; Windows 3.1)'}

data = {"username": "user@email.com",
         "password": "Passw0rd",
         "isPersistent": False}

r = requests.post("https://elhapi.edp.sunpower.com/v1/elh/authenticate", headers=headers, json=data)
r.raise_for_status()
j = r.json()
print(j)

[sashkab]

Out of curiosity, how many of you have "Communication Issue" message in the us.monitor.sunpower.com?

[jeffkowalski]

Right now, I see the banner "Possible error in your displayed data. Learn more." Clicking the link for more information, I get " Temporary error in your displayed data

Your data may not be displaying properly right now. Don't worry, your system performance is unaffected. We are working on getting this fixed as soon as we can.

"

That said, when I poll the API, I am getting reasonable results.

On Tue, Jan 21, 2020 at 10:20 AM Aleks Bunin notifications@github.com wrote:

Out of curiosity, how many of you have "Communication Issue" message in the us.monitor.sunpower.com?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jeffkowalski/sunpower/issues/4?email_source=notifications&email_token=AAAN4MCQ2POXXO6BXBSFVK3Q644FPA5CNFSM4ILYRPJKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQXKNA#issuecomment-576812340, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN4MGYCZOAISSYJY6BJMTQ644FPANCNFSM4ILYRPJA .

[SoulGit]

I am seeing the same as Jeff. I have all but given up on their monitoring systems and system as a whole. I don't even receive their automated "Your Latest SunPower Monitoring Monthly Report" anymore. Last one that came out was end of October. Very disappointed where things have gone with Sunpower. I am going to be expanding my capacity shortly and I'll be looking to replace their system entirely as I have no idea how it is performing day in and day out.

[sashkab]

I don't even receive their automated "Your Latest SunPower Monitoring Monthly Report" anymore.

I don't either. But... it seems that checkbox was checked off when I logged into the mine account over the weekend. I checked it on and will see if I'll get it in early February. On other hand "partner" site which uses flash, can generate these reports for any specific date ranges.

The temporary error been there for a month or so, but it never was causing data not showing up. Now it's "communication error" and PVS rebooting every 20-25 minutes. Which means something is wrong either with PVS, or with their online services. If later, it's easy. But what I've seen it's issue with PVS and/or software on PVS.

[SoulGit]

Sure enough mine was turned off as well. Nice that they made that change without notifying end users . . .

[sashkab]

Mine got fixed sometime today. Seems like firmware upgrade on PVS happened around noon.

[shellykv]

Hi Jeff,

Any of their api gives a panel level data?

[jeffkowalski]

The API does not, as far as I know, give access to panel-level data. In fact, it appears to be a digested summary of data, sourced from a database at sunpower's service. It is not actually communicating directly with your system. There is another communication channel from your supervisor hardware to the sunpower service. Panel status is available there, but I do not know the API.

On Mon, Apr 13, 2020 at 11:30 AM shelly varghese notifications@github.com wrote:

Hi Jeff,

Any of their api gives a panel level data?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jeffkowalski/sunpower/issues/4#issuecomment-613029646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN4MB76PJCQI6JLZRHQLDRMNK33ANCNFSM4ILYRPJA .

[sashkab]

I doubt there is an API -- PVS pushes data at schedule, about once every 5 minutes or so, but channel is encrypted so unappeasable.

I wrote a small script which parses PVS status page into several csv files using Raspberry Pi connected to the "service" ethernet port of the PVS.

[anthonysparrow]

I doubt there is an API -- PVS pushes data at schedule, about once every 5 minutes or so, but channel is encrypted so unappeasable.

I wrote a small script which parses PVS status page into several csv files using Raspberry Pi connected to the "service" ethernet port of the PVS.

Would you be willing to share your Pi image that does so, and perhaps a writeup on necessary changes needed to configure this at another location?

[jeffkowalski]

@anthonysparrow I created a different repo to document the hardware I use (a raspberry pi zero w and a ethernet hat, which fit into the Supervisor box itself). I also wrote a small Ruby app that grabs the data and puts it into Influx for graphing with Graphana. Hope this helps.

[sashkab]

Would you be willing to share your Pi image that does so, and perhaps a writeup on necessary changes needed to configure this at another location?

I've used standard raspbian and tiny python script. You need python 3.6 or newer for this to run, and requests package.

import requests
from datetime import datetime
import time
import json
from pathlib import Path
import csv

PVS5_URL = "http://172.27.153.1/cgi-bin/dl_cgi?Command=DeviceList"

FIELD_NAMES = {
    "pvs": [
        "DETAIL",
        "STATE",
        "STATEDESCR",
        "SERIAL",
        "MODEL",
        "HWVER",
        "SWVER",
        "DEVICE_TYPE",
        "DATATIME",
        "dl_err_count",
        "dl_comm_err",
        "dl_skipped_scans",
        "dl_scan_time",
        "dl_untransmitted",
        "dl_uptime",
        "dl_cpu_load",
        "dl_mem_used",
        "dl_flash_avail",
        "panid",
        "CURTIME",
    ],
    "inverter": [
        "ISDETAIL",
        "SERIAL",
        "TYPE",
        "STATE",
        "STATEDESCR",
        "MODEL",
        "DESCR",
        "DEVICE_TYPE",
        "SWVER",
        "PORT",
        "MOD_SN",
        "NMPLT_SKU",
        "DATATIME",
        "ltea_3phsum_kwh",
        "p_3phsum_kw",
        "vln_3phavg_v",
        "i_3phsum_a",
        "p_mpptsum_kw",
        "v_mppt1_v",
        "i_mppt1_a",
        "t_htsnk_degc",
        "freq_hz",
        "stat_ind",
        "origin",
        "OPERATION",
        "CURTIME",
    ],
    "powermeter": [
        "ISDETAIL",
        "SERIAL",
        "TYPE",
        "STATE",
        "STATEDESCR",
        "MODEL",
        "DESCR",
        "DEVICE_TYPE",
        "SWVER",
        "PORT",
        "DATATIME",
        "ct_scl_fctr",
        "net_ltea_3phsum_kwh",
        "p_3phsum_kw",
        "q_3phsum_kvar",
        "s_3phsum_kva",
        "tot_pf_rto",
        "freq_hz",
        "CAL0",
        "origin",
        "OPERATION",
        "CURTIME",
    ],
}

def get_usage():
    """Get Information from PVS5"""

    try:
        r = requests.get(PVS5_URL, timeout=10)
    except requests.exceptions.ConnectionError as err:
        print(f"{datetime.now():%Y.%m.%d %H:%M:%S} {err}")
        return

    js = r.json()
    dte = datetime.now()
    data = {}
    if js.get("result") == "succeed":
        data = {}
        for device in js.get("devices"):
            device_type = "".join(device.get("DEVICE_TYPE").split()).lower()
            data.setdefault(device_type, []).append(device)
    else:
        print(f"{dte:%Y.%m.%d %H:%M:%S} result={js['result']}")
        return

    for key in data.keys():
        dump_file = Path(f"{dte:%Y%m%d}_{key}.csv")
        write_header = not dump_file.exists()
        with dump_file.open("a+", newline="") as csv_file:
            writer = csv.DictWriter(csv_file, fieldnames=FIELD_NAMES.get(key))
            if write_header:
                writer.writeheader()
            for v in data.get(key):
                writer.writerow(v)

if __name__ == "__main__":
    while True:
        try:
            get_usage()
        except Exception as err:
            print(f"Unhandled Exception {err}")
        print(f"{datetime.now()}", end="\r")
        time.sleep(60)

[arcreative]

@sashkab Looks nice! I saw another post where they just set up HAProxy (although any would do) to proxy the LAN API via WiFi. Seems pretty straightforward and allows you to do the API logic on a more capable machine that's easier to access.

[jeffkowalski]

@arcreative That proxy is the approach I used in my other repo: https://github.com/jeffkowalski/pvs

[arcreative]

@jeffkowalski Nice! Instead of going with Node-RED, I did something a bit more simple that I can run along with my other docker-compose services. It's a ruby script that you can build into a Docker image for easy running without having to install ruby on the host:

https://gist.github.com/arcreative/baf7174d7c5574a515c5cf4bb5877bbb

[sashkab]

@arcreative your script looks simple enough, but you're ignoring error state on inverters, I believe. When inverters doesn't produce electricity (or after PVS restart and before first reported value), they will report STATE=error, but value in the p_3phsum_kw will be last value they produced. You probably want to zero that out, because it will introduce a small discrepancy overnight. So line 54 should look something like this:

inverters = data['devices'].filter { |i| i['DEVICE_TYPE'] == 'Inverter' }.filter { |i| i['STATUS'] != 'error' }

I'm not sure (yet) if power meter can have an error state... so you might want to look out for that too.

An yes, HAProxy is simple enough to configure so you could access PVE data from another host.

[arcreative]

@sashkab Thanks! I didn't really dig into it too too much as you seem to have noticed from my complete lack of error handling :-). That's good info, I'll make it more robust when I get a little time. I'm still having trouble on the Home Assistant side figuring out exactly what statistics to use and how to do them, so there's still a bit of work ahead.

[alexandreshah]

@jeffkowalski Looks like the endpoint https://elhapi.edp.sunpower.com/v1/elh/authenticate no longer exists and responds with a 401. Any ideas on if this script will still work with some minor tweaks?