search

spacemanspiff2007 / HABApp

Easy home automation with MQTT and/or openHAB
Apache License 2.0
54 stars 23 forks source link

set_persistence_data : Attempt to decode JSON with unexpected mimetype #357

Closed ntzagkarakis closed 1 year ago

ntzagkarakis commented 1 year ago

Hi, first of all very nice work, HABapp simplifies a lot of things if someone works with python and it is really fast. I have an issue related with the set_persistence_data function. I know the PUT method of the persistence endpoint is buggy but it seems it works for jdbc persistence service and postgres database, as mentioned here. When I tried to use it from HABapp i got an ERROR: Attempt to decode JSON with unexpected mimetype. I know the endpoint works because i tested directly from their API.

The problem is that async_set_persistence_data tries to load a json response, while the response is empty. Moreover the "itemname" parameter is not necessary as the item name is specified in the URL but this is not related to the error.

spacemanspiff2007 commented 1 year ago

Could you post the example and the returned value? Also the error traceback from the log.

ntzagkarakis commented 1 year ago

Thanks for the quick response. I am using the set_persistence_data function:

from HABApp.openhab.connection_handler.func_sync import set_persistence_data and then inside my rule I call:

set_persistence_data(
                item_name=item.name, # my item name, in this case Obu0_Oxygen0
                persistence=settings.OH_PERSISTENCE_SERVICE, # my persistence service, in this case jdbc
                state=state, # my state, in this case 11
                time=timestamp_d, # my datetime in this case datetime.datetime(2015, 1, 26, 11, 27, 20)
            )

The item variable in the above example is a NamedTuple containing the data i want to send to the OpenHab. Bellow you can find the the logs from the HABapp/log/HABapp.log. 

[2023-01-26 11:34:34,559] [                   HABApp]     INFO | HABApp Version 1.0.6
[2023-01-26 11:34:34,559] [     HABApp.files.watcher]    DEBUG | Adding watcher for /app/HABApp with <FileEndingFilter ending: .yml>
[2023-01-26 11:34:34,559] [                   HABApp]    DEBUG | Added event listener for "HABApp.Files" (filter=EventFilter(type=RequestFileUnloadEvent))
[2023-01-26 11:34:34,559] [                   HABApp]    DEBUG | Added event listener for "HABApp.Files" (filter=EventFilter(type=RequestFileLoadEvent))
[2023-01-26 11:34:34,560] [    HABApp.openhab.plugin]    DEBUG | Starting setup
[2023-01-26 11:34:34,560] [    HABApp.openhab.plugin]    DEBUG | Setup LoadAllOpenhabItems complete
[2023-01-26 11:34:34,560] [      HABApp.openhab.ping]    DEBUG | Ping stopped
[2023-01-26 11:34:34,560] [    HABApp.openhab.plugin]    DEBUG | Setup PingOpenhab complete
[2023-01-26 11:34:34,560] [    HABApp.openhab.plugin]    DEBUG | Setup ThingOverview complete
[2023-01-26 11:34:34,560] [    HABApp.openhab.plugin]    DEBUG | Setup ManualThingConfig complete
[2023-01-26 11:34:34,560] [     HABApp.files.watcher]    DEBUG | Adding recursive watcher for /app/HABApp/rules with <FileEndingFilter ending: .py>
[2023-01-26 11:34:34,561] [   HABApp.mqtt.connection]     INFO | Connecting to 192.168.2.16:1883
[2023-01-26 11:34:34,563] [   HABApp.mqtt.connection]     INFO | Connection Accepted.
[2023-01-26 11:34:34,563] [   HABApp.mqtt.connection]    DEBUG | Subscribing to:
[2023-01-26 11:34:34,563] [   HABApp.mqtt.connection]    DEBUG |  - "testing/#" (QoS 0)
[2023-01-26 11:34:34,563] [HABApp.openhab.connection]    DEBUG | Trying to connect to OpenHAB ...
[2023-01-26 11:34:34,577] [HABApp.openhab.connection]     INFO | Connected to OpenHAB version 3.4.1 (Release Build)
[2023-01-26 11:34:34,588] [             HABApp.Items]    DEBUG | Added Obu0_Oxygen0 (NumberItem)
[2023-01-26 11:34:34,589] [             HABApp.Items]    DEBUG | Added OBU0 (GroupItem)
[2023-01-26 11:34:34,589] [             HABApp.Items]    DEBUG | Added Obu0_Temperature0 (NumberItem)
[2023-01-26 11:34:34,589] [     HABApp.openhab.items]     INFO | Updated 3 Items
[2023-01-26 11:34:34,592] [             HABApp.Items]    DEBUG | Added mqtt:topic:OBU0_b1 (Thing)
[2023-01-26 11:34:34,593] [             HABApp.Items]    DEBUG | Added mqtt:broker:broker1 (Thing)
[2023-01-26 11:34:34,593] [     HABApp.openhab.items]     INFO | Updated 2 Things
[2023-01-26 11:34:39,782] [             HABApp.files]    DEBUG | rules/update_states.py added
[2023-01-26 11:34:40,085] [             HABApp.files]    DEBUG | rules/update_states.py changed to DEPENDENCIES_OK
[2023-01-26 11:34:40,085] [             HABApp.Rules]    DEBUG | Loading file: rules/update_states.py
[2023-01-26 11:34:40,109] [             HABApp.Items]    DEBUG | Added testing/fc-obu0/state (MqttItem)
[2023-01-26 11:34:40,109] [                   HABApp]    DEBUG | Added event listener for "testing/fc-obu0/state" (filter=ValueChangeEventFilter())
[2023-01-26 11:34:40,109] [             HABApp.Rules]     INFO | Added rule "CustomStateUpdate" from rules/update_states.py
[2023-01-26 11:34:40,110] [             HABApp.Rules]    DEBUG | File rules/update_states.py successfully loaded!
[2023-01-26 11:34:40,110] [             HABApp.files]    DEBUG | rules/update_states.py changed to LOADED
[2023-01-26 11:34:55,522] [             HABApp.files]    DEBUG | Worker done!
[2023-01-26 11:34:57,267] [              HABApp.Rule]  WARNING | /usr/local/lib/python3.10/site-packages/HABApp/openhab/connection_handler/func_async.py:128: ResourceWarning:async_set_persistence_data calls a part of the openHAB API which is buggy!
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR | Error in CustomStateUpdate.update_state: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('http://192.168.2.16:8075/rest/persistence/items/Obu0_Oxygen0?itemname=Obu0_Oxygen0&time=2015-01-26T11:27:20.000000%2B0200&state=11&serviceId=jdbc')
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR | File "/app/HABApp/rules/update_states.py", line 82 in update_state
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR | --------------------------------------------------------------------------------
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR |      48 | def update_state(self, event: ValueChangeEvent):
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR |       (...)
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR |      75 |             continue
[2023-01-26 11:34:58,036] [            HABApp.Worker]    ERROR |      77 |         # The timestamp is considered historical or feature: provide the custom timestamp to OH
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      78 |         logger.warning(
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      79 |             f"Received data with timestamp not close to now ({timestamp_d}) from topic: "
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      80 |             f"{event.name!r} for item: {item.name!r}"
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      81 |         )
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR | -->  82 |         set_persistence_data(
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      83 |             item_name=item.name,
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      84 |             persistence=settings.OH_PERSISTENCE_SERVICE,
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      85 |             state=state,
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      86 |             time=timestamp_d,
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      87 |         )
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |    ------------------------------------------------------------
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      event = <MqttValueChangeEvent name: testing/fc-obu0/state, value: {'timestamp': 1422264440, 'oxyguardSensors': [{'oxygen': 11, 'temperature': 12}]}, old_value: None>
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      event.name = 'testing/fc-obu0/state'
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      event.value = {'timestamp': 1422264440, 'oxyguardSensors': [{'oxygen': 11, 'temperature': 12}]}
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      self = <CustomStateUpdate>
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      self.items_config = <habapp_custom.items.config.ItemsConfig object at 0x7f02cc2ea920>
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      self.oh_items = {'Obu0_Oxygen0': <NumberItem name: Obu0_Oxygen0, value: 1, last_change: 2023-01-26T11:34:34.588940, last_update: 2023-01-26T11:34:34.588940>, 'Obu0_Temperature0': <NumberItem name: Obu0_Temperature0, value: 10, last_change: 2023-01-26T11:34:34.589134, last_update: 2023-01-26T11:34:34.589134>}
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      settings = Settings(LOG_LEVEL='DEBUG', SQLALCHEMY_DATABASE_URI=PostgresDsn('postgresql+asyncpg://some_user:some_password@192.168.2.245:8082/OHtest2', ), OH_ITEMS_SCHEMA='fishcare_oh', CHARIOT_SCHEMA='fishcare_chariot', OH_TABLE_ID_DIGIT_COUNT=4, OH_TABLE_NAME_PREFIX='item', OH_PERSISTENCE_SERVICE='jdbc', OH_ITEMS_FILE=PosixPath('/app/mqtt_config.yml'), HISTORICAL_MAX_DIFF=3.0)
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      settings.HISTORICAL_MAX_DIFF = 3.0
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      settings.OH_PERSISTENCE_SERVICE = 'jdbc'
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      current = False
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      item = ItemConfig(name='Obu0_Oxygen0', mqtt_topic='testing/fc-obu0/state', state_path='$.oxyguardSensors[0].oxygen', timestamp_path='$.timestamp')
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      items_config = [ItemConfig(name='Obu0_Oxygen0', mqtt_topic='testing/fc-obu0/state', state_path='$.oxyguardSensors[0].oxygen', timestamp_path='$.timestamp'), ItemConfig(name='Obu0_Temperature0', mqtt_topic='testing/fc-obu0/state', state_path='$.oxyguardSensors[0].temperature', timestamp_path='$.timestamp')]
[2023-01-26 11:34:58,037] [            HABApp.Worker]    ERROR |      logger = <Logger update_state (DEBUG)>
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      state = 11
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      timestamp = 1422264440
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      timestamp_d = datetime.datetime(2015, 1, 26, 11, 27, 20)
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |    ------------------------------------------------------------
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR | File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458 in result
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR | --------------------------------------------------------------------------------
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      430 | def result(self, timeout=None):
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |        (...)
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      455 |             if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      456 |                 raise CancelledError()
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      457 |             elif self._state == FINISHED:
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR | -->  458 |                 return self.__get_result()
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      459 |             else:
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |    ------------------------------------------------------------
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      [CANCELLED, CANCELLED_AND_NOTIFIED] = ['CANCELLED', 'CANCELLED_AND_NOTIFIED']
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      CANCELLED = 'CANCELLED'
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      CANCELLED_AND_NOTIFIED = 'CANCELLED_AND_NOTIFIED'
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      FINISHED = 'FINISHED'
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      self = None
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |      timeout = None
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |    ------------------------------------------------------------
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR |
[2023-01-26 11:34:58,038] [            HABApp.Worker]    ERROR | File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403 in __get_result
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR | --------------------------------------------------------------------------------
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |      400 | def __get_result(self):
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |      401 |     if self._exception:
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |      402 |         try:
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR | -->  403 |             raise self._exception
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |      404 |         finally:
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |      405 |             # Break a reference cycle with the exception in self._exception
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |    ------------------------------------------------------------
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |      self = None
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |    ------------------------------------------------------------
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR | --------------------------------------------------------------------------------
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR | Traceback (most recent call last):
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/site-packages/HABApp/core/internals/wrapped_function/wrapped_thread.py", line 79, in run_sync
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |     self.func(*args, **kwargs)
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/app/HABApp/rules/update_states.py", line 82, in update_state
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |     set_persistence_data(
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/site-packages/HABApp/openhab/connection_handler/func_sync.py", line 248, in set_persistence_data
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |     return run_coro_from_thread(
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/site-packages/HABApp/core/asyncio.py", line 40, in run_coro_from_thread
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |     return fut.result()
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |     return self.__get_result()
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |     raise self._exception
[2023-01-26 11:34:58,039] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/site-packages/HABApp/openhab/connection_handler/func_async.py", line 143, in async_set_persistence_data
[2023-01-26 11:34:58,040] [            HABApp.Worker]    ERROR |     return await ret.json(loads=load_json, encoding='utf-8')
[2023-01-26 11:34:58,040] [            HABApp.Worker]    ERROR |   File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1104, in json
[2023-01-26 11:34:58,040] [            HABApp.Worker]    ERROR |     raise ContentTypeError(
[2023-01-26 11:34:58,040] [            HABApp.Worker]    ERROR | aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('http://192.168.2.16:8075/rest/persistence/items/Obu0_Oxygen0?itemname=Obu0_Oxygen0&time=2015-01-26T11:27:20.000000%2B0200&state=11&serviceId=jdbc')
spacemanspiff2007 commented 1 year ago

I'm specifically interested in the response from openHAB. You write you have tried it manually. Can you post the request/response?

ntzagkarakis commented 1 year ago 
import requests

url = 'http://localhost:8075/rest/persistence/items/Obu0_Oxygen0'
params= {'serviceId': 'jdbc', 'state': 11, 'time': '2015-01-26T11:27:20'}
headers = {'Authorization': 'Bearer your token'}

response = requests.put(params=params, url=url, headers=headers)

print(response.status_code) # = 200
print(response.content) # = b''

Τhe response is empty

ntzagkarakis commented 1 year ago

For your information, I am using OpenHab 3.4.1 The fix related to the jdbc and postgres was merged at that version. So if you are testing the Openhab API at a version <3.4 it won't work.

spacemanspiff2007 commented 1 year ago

I've pushed a fix to dev. Would you be willing to try and see if it now works as expected?

ntzagkarakis commented 1 year ago

Yes, it works like a charm! Great work!