thingsboard / thingsboard-gateway

Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with ThingsBoard IoT Platform using Modbus, CAN bus, BACnet, BLE, OPC-UA, MQTT, ODBC and REST protocols
https://thingsboard.io/docs/iot-gateway/what-is-iot-gateway/
Apache License 2.0
1.72k stars 829 forks source link

[BUG]AttributeError: 'str' object has no attribute 'get' #442

Closed alexvermeersch closed 3 years ago

alexvermeersch commented 3 years ago

""2020-12-17 12:52:30" - ERROR - [json_rest_uplink_converter.py] - json_rest_uplink_converter - 62 - 'str' object has no attribute 'get'", Traceback (most recent call last):, File "/usr/local/lib/python3.8/site-packages/thingsboard_gateway-2.5.5.2-py3.8.egg/thingsboard_gateway/connectors/rest/json_rest_uplink_converter.py", line 56, in convert, if datatype == 'timeseries' and (data.get("ts") is not None or data.get("timestamp") is not None):, AttributeError: 'str' object has no attribute 'get' Connector name (If bug in the some connector): gateway using rest.json connector

Thingsboard and gateway installed under docker


2020-12-17 12:52:30" - ERROR - [json_rest_uplink_converter.py] - json_rest_uplink_converter - 61 - Error in converter, for config: ,
{"type": "json", "deviceNameExpression": "Device ${DevEUI}", "deviceTypeExpression": "default", "attributes": [{"type": "string", "key": "Company", "value": "${companyName}"}, {"type": "string", "key": "DeviceId", "value": "${DevEUI}"}, {"type": "string", "key": "LocationName", "value": "${locationFriendlyName1}"}], "timeseries": [{"type": "string", "key": "${container}", "value": "${value}"}]},
 and message: ,
{"companyName": "Vandeputte Marc bvba","DevEUI": "244E7B000100057A","container": "tank fill level","locationFriendlyName1": "ambachtenstraat 7, Eeklo","value": "59","timestamp": 1608209472000},
"

**Versions (please complete the following information):**
 - OS: Debian 10

tb-gateway | running |   | - | thingsboard/tb-gateway:latest | 2020-12-15 22:22:14 | 5100:5100 | administrators
-- | -- | -- | -- | -- | -- | -- | --
portainer | running |   | - | portainer/portainer-ce | 2020-12-15 20:51:36 | 8000:8000                                     9000:9000 | administrators
alex_mytb_1 | running |   | alex | thingsboard/tb-postgres | 2020-12-15 12:21:51 | 1883:1883                                     5683:5683                                     8080:9090 | administrators
data | running |   | - | postgres | 2020-12-15 01:23:50 | -
alexvermeersch commented 3 years ago

I hope zbeacon or Ilya Barkov is not ill...

i solved the problem by adding code in json_rest_uplink_converter.py

insert on line 30 y = loads(data) tijd = y["timestamp"] then modify line 63 to if datatype == 'timeseries' : dict_result[datatypes[datatype]].append({"ts": tijd, 'values': {full_key: full_value}})

et voila, problem solved!

Maybe better to insert in line 30 :+1: tijd = 0 try: y = loads(data) tijd = y["timestamp"] except Exception as e: log.error("The data string does not have te element timestamp.") log.exception(e) (have not tried it out...)

Working json_rest_uplink_converter.py:

Copyright 2020. ThingsBoard

#

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#

http://www.apache.org/licenses/LICENSE-2.0

#

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

from time import time from simplejson import dumps, loads from thingsboard_gateway.connectors.rest.rest_converter import RESTConverter, log from thingsboard_gateway.tb_utility.tb_utility import TBUtility

class JsonRESTUplinkConverter(RESTConverter):

def __init__(self, config):
    self.__config = config

def convert(self, config, data):
    datatypes = {"attributes": "attributes",
                 "timeseries": "telemetry"}
    dict_result = {"deviceName": None, "deviceType": None, "attributes": [], "telemetry": []}
    y = loads(data)
    tijd = y["timestamp"]        
    try:
        if self.__config.get("deviceNameExpression") is not None:
            dict_result["deviceName"] = TBUtility.get_value(self.__config.get("deviceNameExpression"), data, expression_instead_none=True)
        else:
            log.error("The expression for looking \"deviceName\" not found in config %s", dumps(self.__config))
        if self.__config.get("deviceTypeExpression") is not None:
            dict_result["deviceType"] = TBUtility.get_value(self.__config.get("deviceTypeExpression"), data, expression_instead_none=True)
        else:
            log.error("The expression for looking \"deviceType\" not found in config %s", dumps(self.__config))
    except Exception as e:
        log.error('Error in converter, for config: \n%s\n and message: \n%s\n', dumps(self.__config), data)
        log.exception(e)
    try:
        for datatype in datatypes:
            dict_result[datatypes[datatype]] = []
            for datatype_config in self.__config.get(datatype, []):
                value = TBUtility.get_value(datatype_config["value"], data, datatype_config["type"], expression_instead_none=True)
                value_tag = TBUtility.get_value(datatype_config["value"], data, datatype_config["type"], get_tag=True)
                key = TBUtility.get_value(datatype_config["key"], data, datatype_config["type"], expression_instead_none=True)
                key_tag = TBUtility.get_value(datatype_config["key"], data, get_tag=True)
                if ("${" not in str(value) and "}" not in str(value)) \
                   and ("${" not in str(key) and "}" not in str(key)):
                    is_valid_key = isinstance(key, str) and "${" in datatype_config["key"] and "}" in datatype_config["key"]
                    is_valid_value = isinstance(value, str) and "${" in datatype_config["value"] and "}" in datatype_config["value"]
                    full_key = datatype_config["key"].replace('${' + key_tag + '}', key) if is_valid_key else key
                    full_value = datatype_config["value"].replace('${' + value_tag + '}', value) if is_valid_value else value
                    if datatype == 'timeseries' :
                        dict_result[datatypes[datatype]].append({"ts": tijd, 'values': {full_key: full_value}})
                    else:
                        dict_result[datatypes[datatype]].append({full_key: full_value})
    except Exception as e:
        log.error('Error in converter, for config: \n%s\n and message: \n%s\n', dumps(self.__config), str(data))
        log.exception(e)
    return dict_result
imbeacon commented 3 years ago

Hi @alexvermeersch ,

Thank you for your investigations and for hopes. I am glad to inform that I'm okay and I just was on vacation for two weeks. I hope you are well too. Please create a pull request with your fixes and I will merge it into the repository.

alexvermeersch commented 3 years ago

Hi Ilya,

releaved to hear you're OK, hope you enjoyed your holliday...

I have solved the issue in 2 steps: First inserted 2 lines at pos 30: y = loads(data) tijd = y["timestamp"] then modified line 58 and 59 to: if datatype == 'timeseries' : dict_result[datatypes[datatype]].append({"ts": tijd, 'values': {full_key: full_value}})

That worked well!

Then i added some safety : tijd = 0 timeformat = '%d/%m/%Y %H:%M:%S' try: y = loads(data) if "timestamp" in data: my_date = y["timestamp"] if "ts" in data: my_date = y["ts"] if not (isinstance(my_date, int)): tijd = (int(datetime.datetime.strptime(my_date, timeformat).strftime("%s")))*1000 else: tijd = my_date except Exception as e: log.error("The data string does not have the element timestamp or ts.") log.exception(e)

If no "timestamp" or "ts" in the data string tijd remains 0 (1 jan 1970) if there is either one and it is an int (unix timestamp) that is copied into tijd, if it is a string timeformat it is converted into unix timestamp (*1000 to get milliseconds)

that works also So the attached json_.....py is the one in my system and it does what it has to do...

kind regards,

Alex.

PS i am not familiar with github so could you tell me how to create a pull?

Van: "Illia Barkov" notifications@github.com Aan: "thingsboard/thingsboard-gateway" thingsboard-gateway@noreply.github.com Cc: "alex vermeersch" alex.vermeersch@telenet.be, "Mention" mention@noreply.github.com Verzonden: Dinsdag 22 december 2020 09:58:26 Onderwerp: Re: [thingsboard/thingsboard-gateway] [BUG]AttributeError: 'str' object has no attribute 'get' (#442)

Hi [ https://github.com/alexvermeersch | @alexvermeersch ] ,

Thank you for your investigations and for hopes. I am glad to inform that I'm okay and I just was on vacation for two weeks. I hope you are well too. Please create a pull request with your fixes and I will merge it into the repository.

— You are receiving this because you were mentioned. Reply to this email directly, [ https://github.com/thingsboard/thingsboard-gateway/issues/442#issuecomment-749429260 | view it on GitHub ] , or [ https://github.com/notifications/unsubscribe-auth/ASFFGVCUFMGK7IRBY32ABJLSWBNTFANCNFSM4U74OOEQ | unsubscribe ] .