search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.22k stars 30.58k forks source link

Rejseplanen integration stopped working due to API changes #70222

Closed KennethLavrsen closed 2 years ago

KennethLavrsen commented 2 years ago

The problem

Danish Rejseplanen changed their API from end March 2022. Nothing works anymore

https://help.rejseplanen.dk/hc/da/articles/115002200509-Information-for-previously-existing-users

What version of Home Assistant Core has the issue?

core-2022.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Rejseplanen

Link to integration documentation on our website

https://www.home-assistant.io/integrations/rejseplanen/#stop_id

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

rejseplanen documentation rejseplanen source (message by IssueLinks)

probot-home-assistant[bot] commented 2 years ago

Hey there @darkfox, mind taking a look at this issue as it has been labeled with an integration (rejseplanen) you are listed as a code owner for? Thanks! (message by CodeOwnersMention)

DarkFox commented 2 years ago

I'll look into it, but it works with my config currently, so if you could share a config that doesn't work, that would help a lot. :)

man 18. apr 2022 klokken 10:39 skrev Kenneth Lavrsen:

The problem

Danish Rejseplanen changed their API from end March 2022. Nothing works anymore

https://help.rejseplanen.dk/hc/da/articles/115002200509-Information-for-previously-existing-users

What version of Home Assistant Core has the issue?

core-2022.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Rejseplanen

Link to integration documentation on our website

https://www.home-assistant.io/integrations/rejseplanen/#stop_id

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

— Reply to this email directly, view it on GitHub https://github.com/home-assistant/core/issues/70222, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAG4MVGWA6ESAIYTR7QI2DVFUNVBANCNFSM5TVKH4RA. You are receiving this because you were mentioned.Message ID: @.***>

KennethLavrsen commented 2 years ago

Hi.

It works differently than this morning. This morning nothing worked and I assumed it was the API change that the page I linked talks about.

Now 3 of 4 sensors work.

These are my 4 sensors

- platform: rejseplanen
  name: "Vanløse 9A"
  stop_id: '008600703'
  route:
    - 'Bus 9A'
  direction: 'Glostrup St.'

- platform: rejseplanen
  stop_id: '000003798'
  name: "Byparken"

- platform: rejseplanen
  stop_id: '38'
  name: "Glostrup Busses"
  departure_type: 'BUS'

- platform: rejseplanen
  name: "Glostrup S Train"
  stop_id: '008600622'
  departure_type: 'S'

Here is the error message in the log. The one that failes matching the errors below is the S trains from Glostrup Station

2022-04-18 17:10:14 ERROR (MainThread) [homeassistant.components.sensor] rejseplanen: Error on device update!
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
await entity.async_device_update(warning=False)
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 735, in async_device_update
await task
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
2022-04-18 17:10:16 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [281473110915488] Received invalid command: mobile_app/push_notification_channel
2022-04-18 17:10:16 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [281473110914240] Received invalid command: mobile_app/push_notification_channel
2022-04-18 17:10:16 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [281473110914240] Received invalid command: mobile_app/push_notification_channel
2022-04-18 17:10:16 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [281473110914240] Received invalid command: mobile_app/push_notification_channel
KennethLavrsen commented 2 years ago

Update

Tried to rearrange the 4 so it starts with the S-train and now I get

2022-04-18 17:20:52 ERROR (MainThread) [homeassistant.components.sensor] rejseplanen: Error on device update!
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
await entity.async_device_update(warning=False)
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 735, in async_device_update
await task
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
2022-04-18 17:20:57 WARNING (MainThread) [homeassistant.components.light] Setup of light platform mqtt is taking over 10 seconds.
2022-04-18 17:20:57 WARNING (MainThread) [homeassistant.components.cover] Setup of cover platform mqtt is taking over 10 seconds.
2022-04-18 17:20:59 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform mqtt is taking over 10 seconds.
2022-04-18 17:20:59 WARNING (MainThread) [homeassistant.components.switch] Setup of switch platform mqtt is taking over 10 seconds.
2022-04-18 17:21:02 WARNING (MainThread) [homeassistant.components.camera] Setup of camera platform amcrest is taking over 10 seconds.
2022-04-18 17:21:02 WARNING (MainThread) [homeassistant.components.camera] Setup of camera platform amcrest is taking over 10 seconds.
2022-04-18 17:21:52 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.glostrup_s_train fails
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 514, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 742, in async_device_update
raise exc
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
2022-04-18 17:21:52 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.vanlose_9a fails
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 514, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 742, in async_device_update
raise exc
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
2022-04-18 17:21:52 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.glostrup_busses fails
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 514, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 742, in async_device_update
raise exc
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0

which seems like they all fail. But they actually work in the UI afterwards - all 4. This is really odd. Like something times out too fast and then recovers.

KennethLavrsen commented 2 years ago

One more update

I tried to reduce the number of sensors to just 1. The result is that the sensor appears to work in the UI but it floods the log with errors.

It seems that most requests fails and then eventually succeeds. I ran with the 4 sensors for 6 hours and I had a huge log of errors. The integration is not stable any longer. Something must have changed on the server side. Maybe they trottle the requests.

Here is the first log entries with just one sensor defined

2022-04-18 20:46:51 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.byparken fails
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 514, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 742, in async_device_update
raise exc
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
2022-04-18 20:47:51 ERROR (MainThread) [homeassistant.helpers.entity] Update for sensor.byparken fails
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 910, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/local/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/usr/local/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 514, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 742, in async_device_update
raise exc
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 155, in update
self.data.update()
File "/usr/src/homeassistant/homeassistant/components/rejseplanen/sensor.py", line 191, in update
results = rjpl.departureBoard(
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 258, in departureBoard
response = _request('departureBoard', params, timeout)
File "/usr/local/lib/python3.9/site-packages/rjpl/methods.py", line 23, in _request
return response.json()
File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0
jochke commented 2 years ago

FYI (not sure if it is related)

I also experienced one of the sensors not working anymore (Entity not found error):

- platform: rejseplanen
  name: "Departure to Denmark"
  stop_id: 7401585
  departure_type: "REG"
  direction:
    - "Østerport St."

The following though still work. 

- platform: rejseplanen
  name: "Departure to Lund C."
  stop_id: 7401585
  direction:
    - "Helsingborg C"
    - "Karlskrona C og Kristiansstad C"
    - "Karlskrona C"
    - "Kristiansstad C"
    - "Hässleholm C"
    - "Växjö C"

After a restart it suddenly worked again.

DarkFox commented 2 years ago

If I were to guess, I'd say they've started throttling the public API endpoint.

I've requested access to the new API base URL. I worry they'll require every user to request individual access, but we'll see.

KennethLavrsen commented 2 years ago

While we wait for a fix that may come (we are at the mercy of Rejseplanen - do not blame Martin)

I have made small work around meanwhile which is not too bad.

Rejseplanen has a feature where you can put a Live Departure Ticker (afgangstavle) as an iFrame in your own website. It is no good in a Home Assistant context for automations etc but if you just use it to quickly see departures from your local busstop or train station then it may fill the gap.

Go to http://www.rejseplanen.dk/bin/help.exe/mn?L=vs_newdesign&tpl=liveticker_start Follow the guide to create the HTML code to put in your web site. Copy that code.

In Home Assistant HACS has a custom card called Lovelace HTML Card. Install this and put such a card where you want it to be. And paste in the code you got from Rejseplanen.

In raw yaml it will look like this

type: custom:html-card
title: Busses Byparken
content: >
  <iframe frameborder="0" style="border:0px;height:300px;"
  src="https://webapp.rejseplanen.dk/bin/stboard.exe/mn?L=vs_liveticker&input=Byparken
  (Hovedvejen)!&start=yes&time=now&outputMode=tickerMode&tools=no&LTcss=by&productsFilter=1111111111110000&"></iframe>

The result looks like this

image

Now this has nothing to do with fixing the bug. I just put the info here so people finding the bug report can use the solution. Please do not discuss this here. Just let this be a single post of info. Thanks

DarkFox commented 2 years ago

I've now gotten access to the documentation, but as far as I can see, neither the endpoint, nor the API has changed. The change described here, is only related to their user login, to get the documentation.

jochke commented 2 years ago

Hi @DarkFox

Thank you for your effort!

Mine still work after the restart. So maybe it was a non-bug. At least I'm happy now. :)

KennethLavrsen commented 2 years ago

I reanabled all 4 sensors and they have worked without errors in log for more than 24 hours. So it must have been a problem with Rejseplanen. I am closing the bug report