New provider: National Grid

[disforw]

Let's gooooo! I'd love to jump onboard! This library looks good... There's a lot of articles talking about national Grid and Opower relations but I can't seem to find a login page. Can you direct me in the right direction?

[tronikos]

Do you see requests to opower.com when going to your utility website? If yes, it's up to you or the community to add some code in this library to convert the login credentials to an opower access token. Existing utility implementations might help you.

[danieljkemp]

Some network calls I saw when logging into national grid:

scope openid https://login.nationalgridus.com/opower-uwp/opower profile offline_access redirect_uri https://myaccount.nationalgrid.com/s/opower-widget

  https://login.nationalgridus.com/loginnationalgridus.onmicrosoft.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/authorize

[disforw]

Nice find!! Let me see if I can line this up with any of the existing providers…

[danieljkemp]

Page source also has these urls:

https://ngbk.opower.com https://ngny.opower.com https://ngma.opower.com https://ngri.opower.com https://ngli.opower.com https://ngny-gas.opower.com

[iamkirkbater]

I won't realistically be able to get to this for a few days but I can take a stab at this potentially some time within the next week if it's not already in progress. Just happened to come across this library trying to figure out the same problem, so this directly relates to my interests haha

[disforw]

This seems like it would be very similar to ConEd but maybe we should split it up like Exelon because of the different subdomains.

[disforw]

Here’s what I found for PSEG LI GAS:

initial login endpoint (questionable?) https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/oauth2/v2.0/authorize?client_id=88d004b4-3d39-4599-b410-093849907ee5&p=B2C_1A_UWP_NationalGrid_convert_merge_signin

initial login form payload: signInName: xxxxx@email.com password: xxxxxxxxxxxxxxxxx Signin-forgotPassword: FORGOT_PASSWORD_FALSE rememberUserName: true request_type: RESPONSE

Get opower access_token: https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token

the response will include the access_token

the token seems to require a code, I see it in the payload of the request, just not sure where it comes from. There also looks to be many opower subdomains as indicated above, that is only relevant once we have the access_token

[yigitgungor]

This is also one of the calls they make to create the "Green Button Report" https://ngli.opower.com/ei/edge/apis/DataBrowser-v1/cws/utilities/ngli/customers//usage_export/download?format=csv it starts a download of a csv with all account data, however even though the browser is logged in it gives:

{ "error": { "httpStatus": 401, "serviceErrorCode": "UNAUTHORIZED", "details": "Expected logged in customer but received NOT_LOGGED_IN_WEB_USER" } }

so perhaps the call should explicitly pass the user information too.

[disforw]

I need help here. To get the access_token needed for this integration seems to be a simple call to the following URL with 3 fields in the body of the request. I just cant seem to find where to locate the code or the code_verifier. They change every time I login, so I know it’s being generated upon login, I just cant find them. https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token grant_type: authorization_code code: xxxxxxxx code_verifier: xxxxxx

[disforw]

Looks like a user on another project built a login to an Azure app exactly like we need to do here! https://github.com/LeighCurran/AuroraPlus/issues/2#issuecomment-1817435429

also, adding the following blob for reference https://github.com/blue-army/zync/blob/03f652bbac4d43be92ed5969d1c22df60f78fcc5/jibe/src/web/assets/docs/token.txt#L18

[X-sam]

Ngrid uses a standard Azure AD B2C provider MS provides a python library implementation. Shouldn't need to reinvent the wheel here.
Probably all the info you need to instantiate a valid PublicClientApplication:

accountNumber: "" //account-specific
authority: "https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/B2C_1A_NationalGrid_convert_merge_signin"
clientId: "36488660-e86a-4a0d-8316-3df49af8d06d"
coreJsUrl:"https://ngny.opower.com/ei/x/embedded-api/core.js?auth-mode=oauth&locale=en_US"
customerType: "Residential"
envurl: "https://myaccount.nationalgrid.com/s/opower-widget"
fuelType: "ELEC"
isOpowerSessionForCurrentUser: false
region: "UNY" //region-specific. UNY is 'upstate new york'
scope: "https://login.nationalgridus.com/opower-uwp/opower"
servicePostalCode: "" //account-specific
source: "CSS"
tenant: "loginnationalgridus"

edit: looks like they use two different client ids for authentication (i.e. login page) (88d004b4-3d39-4599-b410-093849907ee5) and authorization (i.e. getting the token for opower) (see above json data)

[disforw]

from msal import PublicClientApplication

authority = "https://login.nationalgrid.com/loginnationalgridus.onmicrosoft.com/B2C_1A_NationalGrid_convert_merge_signin"
app = PublicClientApplication("88d004b4-3d39-4599-b410-093849907ee5", authority=authority)

result = app.acquire_token_by_username_password("36488660-e86a-4a0d-8316-3df49af8d06d", username="{username}", password="{password}")

if "acess_token" in result:
    print("Login successful!")
    access_token = result["access_token"]
    # Use the access token for further operations
else:
    print("Login failed. Check your credentials.")

[X-sam]

Several notes, most of which can be summed up by "read up on the links above, as well as authentication & authorization flows with OIDC"-

• you'll need to login with the login client, then use the received JWT to obtain an authorization against the authorization client.
• the login client doesn't appear to support authentication non-interactively

  result['error_description']
  'AADB2C90224: Resource owner flow has not been enabled for the application.'

  you'll either have to solve that, provide a pop-up to the user, or build a request-based flow to authenticate, whereupon you can pass the response to a MSAL client on the authorization side. (You can follow along with how national grid instantiates the web version of MSAL, it's not minified so it's easy to find with debug tools in the post-login browser session)

[1ockwood]

Hey, all. After recently getting a National Grid "smart meter" installed, I've been going down the path of trying to integrate the data into Home Assistant, eventually arriving here. I've been digging in a bit myself to see if I can get anything working, but think I may need some help. Based on the discussion so far, and from my experience, it seems like the National Grid auth flow is somewhat troublesome to get working correctly with this project. However, one potential solution that occurred to me would be to instead try to leverage the token endpoint itself.

It would be more roundabout to setup initially, but theoretically, if a user were to log into the site using their browser and manually grab the refresh_token, we could then use that to get an access_token for opower. I'm not that well versed in Python, so I'm a bit out of my depth in adjusting the code to get something like this to work, but here's what I'm thinking if anyone has any feedback or thoughts:

• Log in to your account at https://myaccount.nationalgrid.com/ using your browser.
• Once logged in, you should see some POST requests to https://login.nationalgrid.com/login.nationalgridus.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token in your Network panel. In the response, you'll see the access_token as well as a refresh_token (among other things).
• If you grab that refresh_token, you can now make a GET request to https://login.nationalgrid.com/login.nationalgridus.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token with the following:
  • grant_type=refresh_token
  • refresh_token=abc... (this would be the refresh token you pulled from the site)
• That will return your access_token as well as all of the other data, so that I think the refresh token could potentially be stored since it will expire, as noted by the refresh_token_expires_in key in the response.

I'm curious to hear if this is feasible in the context of this project. I have successfully modified the code to roughly work like this, but the main bumps I'm hitting are:

• I'm not sure how to implement support for passing in the refresh_token instead of username and password.
• I'm not sure how to handle updating the refresh token when it expires.
• I'm not sure how to handle the various subdomains, as noted above (such as ngny, ngri, ngli, etc.).

Below is my nationalgrid.py, for reference. Mind you this is a rough proof-of-concept, but using this you can successfully run the demo command, such as:

python src/demo.py --utility nationalgrid --username="anything" password="abcdef" --start_date 2024-04-01 --end_date 2024-04-07 --aggregate_type day

where password is actually the refresh_token you grabbed.

You'll also need to modify the subdomain to match your region.

"""National Grid"""

from typing import Optional

import aiohttp

from ..exceptions import InvalidAuth
from .base import UtilityBase

import logging

class NationalGrid(UtilityBase):
    """National Grid utility class."""

    @staticmethod
    def name() -> str:
        return "National Grid"

    @staticmethod
    def subdomain() -> str:
        return "ngny"

    @staticmethod
    def timezone() -> str:
        return "America/New_York"

    @staticmethod
    def uses_refresh_token() -> bool:
        """Check if utility uses refresh token for authorization."""
        return True

    @staticmethod
    async def async_login(
        session: aiohttp.ClientSession,
        username: str,
        password: str,
        optional_mfa_secret: Optional[str],
    ) -> Optional[str]:
        #
        logging.debug("LOGIN");
        token_url = "https://login.nationalgrid.com/login.nationalgridus.com/b2c_1a_nationalgrid_convert_merge_signin/oauth2/v2.0/token"
        async with session.post(
            token_url,
            data={
                "grant_type": "refresh_token",
                "refresh_token": password
            }
        ) as response:
            if response.status == 200:
                data = await response.json()
                access_token = data.get("access_token")
                return access_token
            else:
                # If login fails, raise an exception
                raise InvalidAuth("Invalid username or password")

[X-sam]

Sorry, I haven't had time to put everything together into a PR for opower. But it's not hard to get through the initial user OAuth, here's how I did it for my region.

import re
import requests
import json

def authSession():
    username=""
    password=""
    session = requests.Session()
    session.verify = False
    session.cookies.set("USRW","r=nyupstate&ct=home",domain=".nationalgridus.com")
    session.headers['User-Agent']='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36'
    req = session.get("https://myaccount.nationalgrid.com/services/auth/sso/NGP_SignIn_NY_Upstate_Home")
    sets = re.search("var SETTINGS = ({.*?});",req.text,re.DOTALL)
    settings = json.loads(sets.group(1))
    # var f = r.hosts.tenant + "/" + r.api + "?tx=" + r.transId + "&p=" + r.hosts.policy
    url = f"https://login.nationalgrid.com/{settings['hosts']['tenant']}/{settings['api']}?tx={settings['transId']}&p={settings['hosts']['policy']}"
    headers = {'X-CSRF-TOKEN':settings['csrf']}
    authed=session.post(url,headers=headers,data=[("signInName",username),("password",password),("Signin-forgotPassword","FORGOT_PASSWORD_FALSE"),("request_type","RESPONSE")])

    completed = f"https://login.nationalgrid.com/{settings['hosts']['tenant']}/api/{settings['api']}/confirmed?tx={settings['transId']}&p={settings['hosts']['policy']}&csrf_token={settings['csrf']}"
    res = session.get(completed)
    breakpoint()
    r = res.text
    session.close()

    return r

[1ockwood]

That's great to hear @X-sam! Let me know if I can help in any way. Like I said, I'm a little out of my area of expertise here, but happy to contribute if I'm able.

[disforw]

Don't forget about me, what can I do to help?

[mag2352]

Are there any updates on this? I'm going to take a look this week, but am interested if any further developments have been made here.

[disforw]

Nothing really, I believe we have all the components here but couldn't seem to put the pieces together.