[Source Request]: Waste Management (WM)

[Rthrall]

Municipality / Region

United States

Collection Calendar Webpage

https://www.wm.com/us/en/mywm/my-services/view-pickup-eta

Example Address

Address: 39440 10th St W Palmdale, CA 93551 (Nearby store--not my address)

Customer ID: 000000123456789

Collection Data Format

As html on a webpage

Additional Information

Waste Management (they are going through a rebranding right now and will be called WM soon). The operate throughout North America, in particular suburbs.

The link provided for Collection Calendar Webpage has a "Continue as Guest" options where on can enter either a Service Address in the US or Canadian address format or a 15 digit Customer ID

[jamesshannon]

I found this because I was surprised that WM wasn't already integrated. According to their website, they "serve more than 20 million residential, commercial, industrial and municipal customers ... [in the US and maybe Canada]".

I did some poking around on their website with CDT.

You can start at https://www.wm.com/us/en/mywm/locate?redirect=/us/en/mywm/my-services/view-pickup-eta and enter an address. You then get redirected to https://www.wm.com/us/en/mywm/my-services/view-pickup-eta which (eventually) gets the schedule data with a pretty clean JSON request to https://rest-api.wm.com/account/XXXXX/services/1/pickupinfo?lang=en_US . Example of the return JSON:

{
    "statusCode": 200,
    "status": "SUCCESS",
    "requestTrackingId": "XXXX",
    "errorMsg": {
        "msg": "no data available"
    },
    "data": {
        "wasteStreams": {
            "MSW": {
                "services": [
                    {
                        "customerId": "XXXXXX",
                        "serviceId": "1",
                        "inquiryDate": "2024-08-31T00:00:00.000Z",
                        "pickupScheduleInfo": {
                            "schedule": "Every Monday",
                            "message": null,
                            "pickupDates": [
                                "09-02-2024",
                                "09-09-2024",
                                "09-16-2024",
                                "09-23-2024",
                                "09-30-2024",
                                "10-07-2024",
                                "10-14-2024",
                                "10-21-2024",
                                "10-28-2024"
                            ],
                            "date": null,
                            "messageDesc": null,
                            "frequency": "1",
                            "occurs": "Per Week"
                        },
                        "pickupDayInfo": {
                            "date": "09-02-2024",
                            "message": null,
                            "messageDesc": null
                        },
                        "accountId": "XXXXXX",
                        "eta": [],
                        "category": "FRAN/MUNI",
                        "wasteStream": "MSW",
                        "wasteStreamGroupCode": "MSW",
                        "ticketType": "",
                        "isXPU": true,
                        "mpuMessageDetail": "",
                        "mpuNotEligibleReason": "Service Day was more than 72hrs ago and next service day is in future",
                        "mpuMessage": "Your pickup was completed today.",
                        "serviceDate": "2024-08-26 XXXXXX PDT",
                        "messageHead": [
                            "Pickup Completed"
                        ],
                        "containers": "1",
                        "containersPicked": 1,
                        "containersMissed": 0,
                        "containerType": "1 - 35 Gallon Toter",
                        "lineOfBusiness": "RESIDENTIAL"
                    }
                ]
            }
        },
        "errors": {
            "serverErrors": [],
            "customerNotFound": []
        }
    },
    "loggedInStatus": "Guest"
}

The website UI suggests that you'll see the ETA of the truck on the arrival day, which is pretty neat.

There's also a request to https://rest-api.wm.com/holidays?lang=en_US&street=STREET_ADDRESS&city=CITY&state=STATE&postalCode=ZIP&country=US&type=upcoming, but there are currently no holidays in the next month so it's not clear what that would return.

Getting to the pickupinfo URL is a bit weird, though. The XXXX in the URL represents my account number (consistent with the /account/ in the REST URL structure). Or, rather, it represents the account number of whichever address you may have entered. It may not be the account number that the customer is familiar with as it's quite long. In the response to another request it's labeled the ezpayId.

The prior request is https://rest-api.wm.com/account/search?street=ADDRESS&city=CITY&state=STATE&country=US&address=FULL_ADDRESS&postalCode=ZIP&lang=en_US&googlePlaceId=XXXXXX

The return JSON of that request is various bits of account info, including the ezPayId, the name on the account, the phone number, etc. Note that I am not logged in, so this PII is accessible to the public with just an address. ¯_(ツ)_/¯

There are no nonces or anything. Some responses include a token header, but I don't see that being re-used on future requests. There is an apikey value in the HTTP request header which is required. That value is "hardcoded" into the main.js file so it's likely a static ID (maybe required by their usage of AWS?), but it is possible that it's dynamically generated or frequently rotated?

[evopix]

I was able to grab an example of a holiday response:

{
    "reqId": "9a912c48-cf26-4a7f-8fdd-accf54163111",
    "facilityId": "S04213",
    "facility_name": "WM of ***",
    "holidayMessage": {
        "code": 4612,
        "message": "Regularly scheduled service may shift during a holiday week. Any changes to your schedule will be available at least a week prior to the holiday, but can also change as the holiday approaches. You can receive notifications about schedule changes by text, email or phone – whichever you prefer! Please visit our <a href=https://www.wm.com/us/en/mywm/user/preferences/search>Communication Preferences Page</a> to enroll."
    },
    "holidayData": [
        {
            "additionalHolidayDesc": "Labor Day",
            "holidayName": "Labor Day",
            "holidayHours": "Due to the Labor Day holiday, your scheduled service between Monday, September 2nd, and Friday, September 6th, will not be delayed.",
            "holidayDate": "2024-09-02",
            "holidayId": "61"
        }
    ]
}

It also adds a message to the pickupDayInfo key in the /pickupinfo response:

"pickupDayInfo": {
  "date": "09-03-2024",
  "message": "REFER TO HOLIDAY SCHEDULE",
  "messageDesc": "Your regular scheduled pickup may be affected due to a holiday. Please check your holiday schedule for any changes regarding your pickup."
}

[5ila5]

But how does it look when the collection day is moved?

I have a working prototype without holday schedule, and I have not tested it with other addresses yet:

import requests
import random
import re

START_URL = "https://webapi.wm.com/v1/get-me-started"
PICKUP_URL = "https://rest-api.wm.com/account/{customer_id}/services/{service_id}/pickupinfo?lang=en_US"
SERVICES_URL = "https://rest-api.wm.com/account/{customer_id}/services?lang=en_US&serviceChangeEligibility=Y"
JAVASCRIPT_URL = "https://www.wm.com/static/js/main.8a3fa63d.chunk.js"

params = {
    "street": "3741 Vista Point Way",
    "city": "Palmdale",
    "state": "CA",
}

random_id = random.randint(10000, 99999)
random_profile = "".join(random.sample("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789", 20))

HEADERS = {
    "ProfileId": random_profile,
    "Request-Tracking-Id": str(random_id),
    "Origin": "https://www.wm.com",
}

r = requests.get(START_URL, params=params, headers=HEADERS)
r.raise_for_status()

customer_id = r.json()["customers"][0]["id"]

r = requests.get(JAVASCRIPT_URL)
r.raise_for_status()

#GENESYS: "https://wmchat.wm.com/webapi/api/v2",
GENKEY_REGEX = re.compile(r'GENESYS\s*:\s*"https://wmchat.wm.com/webapi/api/v2"')
API_KEY_REGEX = re.compile(r'CUSTOMER_SERVICES\s*:\s*"(.*?)",')

genkeys = GENKEY_REGEX.search(r.text).group(0)
api_key_search_area = r.text.split(genkeys)[1]
api_key_search_area .split("okta:")[0]

api_key = API_KEY_REGEX.search(api_key_search_area).group(1)

HEADERS = {
    "apiKey": api_key
}

r = requests.get(SERVICES_URL.format(customer_id=customer_id), headers=HEADERS)
r.raise_for_status()
services_reuls = r.json()

for service in services_reuls["services"]:
    service_id = service["serviceId"]
    bin_type = service["serviceDescription"]

    r = requests.get(PICKUP_URL.format(customer_id=customer_id, service_id=service_id), headers=HEADERS)
    r.raise_for_status()
    schedule_data = r.json()

    waset_streams = schedule_data["data"]["wasteStreams"]
    waste_stream = waset_streams[list(waset_streams.keys())[0]]

    for schedule in waste_stream["services"]:
        if not "pickupScheduleInfo" in schedule or "pickupDates" not in schedule["pickupScheduleInfo"]:
            print(bin_type, "no pickup dates", schedule)
            continue
        for date_str in schedule["pickupScheduleInfo"]["pickupDates"]:
            print(bin_type, date_str, )

[evopix]

But how does it look when the collection day is moved?

I'm not sure as my collection day didn't actually move.

[jamesshannon]

JAVASCRIPT_URL = "https://www.wm.com/static/js/main.8a3fa63d.chunk.js"

That ID is almost certainly a version number and likely to change. Best case, you'd always receive the same file, frozen in time. Worst case, they clean those files up after some period of time. In fact, I'm getting a 404 for that URL now.

In theory, you could make a request to the HTML page, get the current-versioned main.js file URL, then request that, then regexp out the value.

To get around that, I was going to suggest that the API key is probably permanent and we could hardcode it. However, I just checked (https://cdn.wm.com/etc/clientlibs/wm/react-app-scripts/static/js/main.20177c7c.chunk.js) and it appears that the CUSTOMER_SERVICES key has already been rotated (60BE3...).

[zelch]

It looks like their API is documented, and while the process to request an API key is pretty opaque, it does exist.

https://api.wm.com/authentication/index.html

It might well be worth contacting them and asking if there is a preferred way to do such an integration.

In an ideal world, they would support OAUTH, but that seems somewhat unlikely based on their documentation.

However I wouldn't be completely surprised if they had some reasonable way to do this.

[jamesshannon]

Hah. And here we trying to reverse-engineer something that is publicly documented.

I'll send them an email and see what they say.

[abillits]

Did they respond?

Also, is there any value in this project?: https://github.com/dcmeglio/homeassistant-waste_management

[jamesshannon]

No. I emailed on the 16th and followed up on the 21st. I kept it vague-ish, in that I didn't mention Home Assistant specifically. No response.

Nice fine. I just looked at the source. Didn't try to install it. It hasn't been touched in over a year, but that's probably fine, assuming they didn't hardcode any keys which have been rotated since then (e.g., these keys)

Seems like having things centralized into something like this project makes more sense. OTOH, most people don't have more than one waste collection provider, so it's not like the centralization really makes it easy for the end user. It's possible that hacs_waste_collection project takes a more thoughtful and useful approach to creating HA entities.

While you provide you credentials directly to the integration (which I don't like doing), rather than via an oauth remote login, they do use oauth via the (undocumented) /user/authenticate URL. The same author (@dcmeglio) also wrote the project which does all the request magic. Seems like @dcmeglio might have done a lot of reverse-engineering too.

IMO, assuming it works it seems perfectly suitable in the short term. But if authentication isn't needed for WM.com APIs, then I don't like the idea of unnecessarily providing my credentials. IMO, one path forward if we're comfortable with the credential thing might to be include the lower-level wm library in this project, and then you get all the request-making for free.

[Commod0re]

OTOH, most people don't have more than one waste collection provider, so it's not like the centralization really makes it easy for the end user.

If we don't have to figure out which specific waste collection integration we need that is still at least a little easier ;)

[abillits]

Nice fine. I just looked at the source. Didn't try to install it. It hasn't been touched in over a year, but that's probably fine, assuming they didn't hardcode any keys which have been rotated since then (e.g., these keys)

It seems they're using the API at rest-api.wm.com to power their customer dashboard. I just logged into my account and wireshark lit up with requests. 99% were using the API key 6277FF29BB19666078AC (same as hardcoded in that project).

Unless they recode their dashboard, it should be very simple to get an update the API key if needed. Obviously this isn't the correct approach but I feel like you already asked nicely.