[NextBus] `sfmta-cis` throws Invalid error, despite working on nextbus site

[MReschenberg]

The problem

I'm trying to use a sensor config corresponding to the URL: https://retro.umoiq.com/#!/sfmta-cis/K/K_0_var2/6991

however when I supply "sfmta-cis" as the agency param in my yaml sensor config like this:

  - platform: nextbus
    name: "Outbound K"
    agency: sfmta-cis
    route: K
    stop: "6991"

I get the following error in logs:

Invalid agency tag `sfmta-cis`. Please use one of the following: APL: jhu-apl, Buses Bío Bío: ATM-BRU-BIOBIO, Cape Cod Regional Transit Authority: ccrta, Chapel Hill Transit: chapel-hill, Chile Sandbox: chile-sandbox, City of Oxford: oxford-ms, City of West Hollywood: west-hollywood, Compañía JAC Transportes Limitada: ATM-BRU-JAC, Downtown Connection: da, Dumbarton Express: dumbarton, EZRide - Charles River TMA: charles-river, Empresa de Transporte Línea N° 8 Padre Las Casas S.A. (Línea N°8): ATM-BUR-L08-TE, Empresa de Transportes Altamira S.A. (Línea N°10): ATM-BUR-L10-TE, Empresa de Transportes de Pasajeros Línea Nueve S.A. (Línea N°9): ATM-BUR-L09-TE, Empresa de los Ferrocarriles del Estado (EFE): ATM-TRR-26-30, Escalon eTrans: escalon, FAST: fast, Glendale Beeline: glendale, Glendale Beeline GTFS: glendale-beeline, Glendale Beeline GTFS - Test: glendale-beeline-test, I. Municipalidad Padre Las Casas: ATM-VAN-19-19, Indianapolis International Airport: indianapolis-air, JFK International Airport: jfk, Jacksonville Transportation Authority: jtafla, LaGuardia Airport Shuttle: lga, MIT Lincoln Lab Shuttle: lincolnlab, Manteca Transit: manteca-transit, Massachusetts Institute of Technology: mit, Merida Transit: merida, Newark Liberty International Airport: ewr, Nova Southeastern University: nova-se, Omnitrans: omnitrans, Palos Verdes Transit: pvpta, Pensacola Beach (SRIA): sria, Portland Streetcar: portland-sc, RTC RIDE, Reno: reno, Roosevelt Island: roosevelt, San Francisco Muni Sandbox: sfmuni-sandbox, Seattle Streetcar: seattle-sc, Sindicato de Trabajadores Independientes Línea N°21: ATM-TCU-L021-TE, Sindicato de Trabajadores Independientes de Taxis Colectivos Caupolicán N°24: ATM-TCU-L024-TE, Sociedad Inmobiliaria e Inversiones El Carmen Cajón S.A. (Línea N°7): ATM-BUR-L07-TE, Sociedad Transportes Av. Alemania – P. Nuevo S.A. (Línea N°1): ATM-BUR-L01-TE, Sociedad de Transportes Línea N°5 S.A. (Línea N°5): ATM-BUR-L05-TE, Sociedad de Transportes SOTRACOL S.A. (Línea N°111 TXC): ATM-TCU-L111-TE, Societe de transport de Laval: stl, SolanoExpress: solano-express, Sonoma County Transit: sct, Spokane International Airport: geg, TEmpresa de Transportes Ñielol Ltda. (Línea N°4): ATM-BUR-L04-TE, Tahoe Truckee Area Regional Transit: tahoe, Taxibuses Número Seis S.A. (Línea N°6): ATM-BUR-L06-TE, Taxibuses Número Seis S.A. (Línea N°66): ATM-BUR-L66-TE, Toronto Transit Commission: ttc, Transportes Jiménez Martinez y Toro (Línea N°3): ATM-BUR-L03-VI, Transportes Jorge Alarcón Jara Empresa Individual (Línea N°4): ATM-BUR-L04-VI, Transportes Línea Número Dos Ltda. (Línea N°2): ATM-BUR-L02-TE, Transportes P. de Valdivia – P. Las Casas Ltda. (Línea N°3): ATM-BUR-L03-TE, UCLA: ucla, Unitrans ASUCD/City of Davis: unitrans, University of California Berkeley: ucb, Vacaville City Coach: vacaville, Western Kentucky University: wku, Winston-Salem: winston-salem

What version of Home Assistant Core has the issue?

core-2023.9.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

NextBus

Link to integration documentation on our website

https://www.home-assistant.io/integrations/nextbus/

Diagnostics information

No response

Example YAML snippet

- platform: nextbus
    name: "Outbound K"
    agency: sfmta-cis
    route: K
    stop: "6991"

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @vividboarder, mind taking a look at this issue as it has been labeled with an integration (nextbus) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `nextbus` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign nextbus` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

nextbus documentation nextbus source _{^{(message by IssueLinks)}}

[MReschenberg]

FWIW, I used to use the recommended sfmuni-sandbox but that stopped working and now supplies no predictions even when there are trains out and about (I think the agency has migrated)

[ViViDboarder]

The sfmuni-sandbox does appear to occasionally return no predictions. It usually comes back eventually. Unfortunately, it seems like the API being used is considered “legacy” and they are adding new features to a new, authenticated internal API.

I emailed them a few years ago to ask about getting an API key, but never got a response.

[ViViDboarder]

I dug a bit more.Still protected by an API key. I could possibly enhance the py_nextbus client to spoof a browser request to get the cookie and API key and use that with the new APIs, or possibly use the ADA site, which is server side rendered, and scrape values.

Alternatively, while it appears that SFMuni has outsourced their predictions to Umo (formerly by NextBus), the Muni website has a predictions API that is not authenticated. This could probably be supported as a new integration separate from the NextBus/Umo one.

[MReschenberg]

Where can I find the Muni predictions API? I'm happy to take a stab at using that for now / writing something new

[ViViDboarder]

You can go here, pick any stops to get a prediction, then inspect the network requests. You should be able to see them there: https://www.sfmta.com/getting-around/muni/routes-stops

[MReschenberg]

Thanks :)

I did some poking around and found there's a separate (mixed?) API that gets accessed via queries to the ADA version of the website here. The queries seem to use the same abbreviated syntax as the nextbus api but they support the new agency tag.

Unfortunately, the request returns an HTML doc instead of a nicely parsable JSON or XML file, but the HTML is semantic so 🤷‍♀️ maybe something is possible there. There's no key required for this API.

This URL is the equiv for the request I was originally trying to make:

https://retro.umoiq.com/predictor/adaPrediction.jsp?a=sfmta-cis&r=K&d=K_0_var2&s=6991

There's also a hardcoded agency list in the js on the predictions index.html source (view-source:https://retro.umoiq.com/predictor/)

      agencies = ["jhu-apl","appcart","tl-bccf","ccrta","chapel-hill","oxford-ms","west-hollywood","da","dumbarton","charles-river","elon","escalon","fast","glendale","great","tl-hbl","indianapolis-air","jfk","kean","lga","tl-l","lincolnlab","maine-med","manteca-transit","mit","merida","montclair","ewr","nova-se","omnitrans","pvpta","sria","pct","portland-sc","reno","roosevelt","salisbury","bawt","sfmta-cis","links","seattle-sc","selkirk","slippery-rock","stl","sct","geg","tahoe","ttc","ucla","unitrans","ucb","unh","vacaville","vmc","wcu","wku","winston-salem",0];

[ViViDboarder]

While technically one can consider that an API that returns HTML, but it would typically just be considered a webpage. Using that as an API source is not possible as a part of homeassistant/core since it would qualify as webscraping: https://github.com/home-assistant/architecture/blob/b85484512dd8f27338c6ae2ff28e9a7cf78abb2f/adr/0004-webscraping.md

[issue-triage-workflows[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.