TTN: SenseCap Tracker data parsed correctly but sensor in HA not useable

northalpha commented 1 month ago

The problem

I can see data beeing processed via TTN Integration for my SenseCAP T1000 Tracker device:

Message for device eui-xxxxxxxxxxxx ignored (type <class 'list'>): [{'measurementId': '4200', 'measurementValue': [], 'motionId': 0, 'timestamp': 1728317683000, 'type': 'Event Status'}, {'measurementId': '4197', 'measurementValue': 8.xxxxx, 'motionId': 0, 'timestamp': 1728317683000, 'type': 'Longitude'}, {'measurementId': '4198', 'measurementValue': 51.xxxxx, 'motionId': 0, 'timestamp': 1728317683000, 'type': 'Latitude'}, {'measurementId': '4097', 'measurementValue': 24, 'motionId': 0, 'timestamp': 1728317683000, 'type': 'Air Temperature'}, {'measurementId': '4199', 'measurementValue': 0, 'motionId': 0, 'timestamp': 1728317683000, 'type': 'Light'}, {'measurementId': '3000', 'measurementValue': 100, 'motionId': 0, 'timestamp': 1728317683000, 'type': 'Battery'}]

but the corresponding sensor is displaying:

Screenshot 2024-10-07 at 6 36 12 PM

i see a new release of the underlying python lib here referencing tracker information but i looks like this version 1.2.1 is not merged into HA until now. would this be fixed merging the new version or how can i help to troubleshoot this issue any further?

What version of Home Assistant Core has the issue?

core-2024.9.3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

thethingsnetwork

Link to integration documentation on our website

https://www.home-assistant.io/integrations/thethingsnetwork/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 month ago

Hey there @angelnu, mind taking a look at this issue as it has been labeled with an integration (thethingsnetwork) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `thethingsnetwork` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign thethingsnetwork` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

thethingsnetwork documentation thethingsnetwork source _{^{(message by IssueLinks)}}

northalpha commented 1 month ago

I have changed the decoder to the following one from: https://www.aeq-web.com/seeed-sensecap-t1000-lorawan-gps-tracker-ttn-payload-decoder/

/*
 * Javascript Decoder for Seeed SenseCAP T1000 LoRaWAN GPS Tracker Version 1.1 (@aeqweb)
 * More Information at: https://www.aeq-web.com/seeed-sensecap-t1000-lorawan-gps-tracker-ttn-payload-decoder/
 */
function decodeUplink(input) {
    const bytes = input['bytes']
    const fport = parseInt(input['fPort'])

    const packetID = bytes[0];
    var decoded = {};

    if (packetID == 1) {
        const batt = bytes[1];
        const swv = bytes[2] + "." + bytes[3];
        const hwv = bytes[4] + "." + bytes[5];
        const wm = getWorkMode(bytes[6]);
        const ps = getPosStrategy(bytes[7]);
        const hiv = bytes[8] << 8 | bytes[9];
        const uiv = bytes[10] << 8 | bytes[11];
        const eiv = bytes[12] << 8 | bytes[13];
        const so = bytes[14];
        const sm = bytes[15];
        const me = Boolean(bytes[16]);
        const mt = bytes[17] << 8 | bytes[18];
        const ms = bytes[19] << 8 | bytes[20];
        const ml = Boolean(bytes[21]);
        const mo = bytes[22] << 8 | bytes[23];
        const se = Boolean(bytes[24]);
        const st = bytes[25] << 8 | bytes[26];
        const te = Boolean(bytes[27]);
        const tu = bytes[28] << 8 | bytes[29];
        const ts = bytes[30] << 8 | bytes[31];
        const th = (bytes[32] << 8 | bytes[33]) / 10;
        const tl = (bytes[34] << 8 | bytes[35]) / 10;
        const tr = bytes[36];
        const le = Boolean(bytes[37]);
        const lu = bytes[38] << 8 | bytes[39];
        const ls = bytes[40] << 8 | bytes[41];
        const lh = bytes[42] << 8 | bytes[43];
        const ll = bytes[44] << 8 | bytes[45];
        const lw = bytes[46];

        var sens = "unkown"
        var sosm = "unkown"

        if (so) {
            sens = "Temp-Light Sen. on"
        } else if (sm === 0) {
            sens = "Temp-Light Sen. off"
        }

        if (sm) {
            sosm = "Continuous Mode"
        } else if (sm === 0) {
            sosm = "Single Mode"
        }

        decoded = {
            packet: "Heartbeat",
            battery: batt,
            softwareV: swv,
            hardwareV: hwv,
            workMode: wm,
            posStrategy: ps,
            heartbeatInterval: hiv,
            uplinkInterval: uiv,
            EventInterval: uiv,
            sensors: sens,
            sosMode: sosm,
            motionEventMode: me,
            motionThreshold: mt,
            motionStartInterval: ms,
            motionlessEvent: ml,
            motionlessTimeout: mo,
            shockEvent: se,
            shockThreshold: st,
            tempEvent: te,
            tempInterval: tu,
            tempSample: ts,
            tempThresholdMax: th,
            tempThresholdMin: tl,
            tempThresholdRule: tr,
            lightEvent: le,
            lightInterval: lu,
            lightSample: ls,
            lightThresholdMax: lh,
            lightThresholdMin: ll,
            lightWarningType: lw
        }
    } else if (packetID == 2) {
        const batt = bytes[1];
        const swv = bytes[2] + "." + bytes[3];
        const hwv = bytes[4] + "." + bytes[5];
        const wm = getWorkMode(bytes[6]);
        const ps = getPosStrategy(bytes[7]);
        const hiv = bytes[8] << 8 | bytes[9];
        const uiv = bytes[10] << 8 | bytes[11];
        const eiv = bytes[12] << 8 | bytes[13];
        const so = bytes[14];
        const sm = bytes[15];
        var sens = "unkown"
        var sosm = "unkown"

        if (so) {
            sens = "Temp-Light Sen. on"
        } else if (sm === 0) {
            sens = "Temp-Light Sen. off"
        }

        if (sm) {
            sosm = "Continuous Mode"
        } else if (sm === 0) {
            sosm = "Single Mode"
        }

        decoded = {
            packet: "Heartbeat",
            battery: batt,
            softwareV: swv,
            hardwareV: hwv,
            workMode: wm,
            posStrategy: ps,
            heartbeatInterval: hiv,
            uplinkInterval: uiv,
            EventInterval: uiv,
            sensors: sens,
            sosMode: sosm
        }
    } else if (packetID == 5) {
        const batt = bytes[1];
        const wm = getWorkMode(bytes[2]);
        const ps = getPosStrategy(bytes[3]);
        const sm = bytes[4];
        var sosm = "unkown"

        if (sm) {
            sosm = "Continuous Mode"
        } else if (sm === 0) {
            sosm = "Single Mode"
        }

        decoded = {
            packet: "Heartbeat",
            battery: batt,
            workMode: wm,
            posStrategy: ps,
            sosMode: sosm
        }
    } else if (packetID == 6) {
        const sta = (bytes[1] << 16 | bytes[2] << 8 | bytes[3]);
        const mcnt = bytes[4];
        const utc = unixToDateTime(bytes[5] << 24 | bytes[6] << 16 | bytes[7] << 8 | bytes[8]);
        const lon = (bytes[9] << 24 | bytes[10] << 16 | bytes[11] << 8 | bytes[12]) / 1000000;
        const lat = (bytes[13] << 24 | bytes[14] << 16 | bytes[15] << 8 | bytes[16]) / 1000000;
        const tmp = ((bytes[17] & 0x80 ? 0xFFFF << 16 : 0) | bytes[17] << 8 | bytes[18]) / 10;
        const lgt = (bytes[19] << 8 | bytes[20]);
        const batt = bytes[21];

        decoded = {
            packet: "GNSS Location & Sensor Data",
            eventStatus: sta,
            motionCount: mcnt,
            utcTime: utc,
            longitude: lon,
            latitude: lat,
            temperature: tmp,
            light: lgt,
            battery: batt
        }
    } else if (packetID == 9) {
        const sta = (bytes[1] << 16 | bytes[2] << 8 | bytes[3]);
        const mcnt = bytes[4];
        const utc = unixToDateTime(bytes[5] << 24 | bytes[6] << 16 | bytes[7] << 8 | bytes[8]);
        const lon = (bytes[9] << 24 | bytes[10] << 16 | bytes[11] << 8 | bytes[12]) / 1000000;
        const lat = (bytes[13] << 24 | bytes[14] << 16 | bytes[15] << 8 | bytes[16]) / 1000000;
        const batt = bytes[17];

        decoded = {
            packetType: "GNSS Location",
            eventStatus: sta,
            motionCount: mcnt,
            utcTime: utc,
            longitude: lon,
            latitude: lat,
            battery: batt
        }
    } else if (packetID == 0x11) {

        const utc = unixToDateTime(bytes[5] << 24 | bytes[6] << 16 | bytes[7] << 8 | bytes[8]);
        const batt = bytes[13];

        if (bytes[11] != 0x80 && bytes[9] != 0x80) {
            const tmp = ((bytes[9] & 0x80 ? 0xFFFF << 16 : 0) | bytes[9] << 8 | bytes[10]) / 10;
            const lgt = (bytes[11] << 8 | bytes[12]);

            decoded = {
                packet: "Positing status",
                utcTime: utc,
                temperature: tmp,
                light: lgt,
                battery: batt
            }
        } else {
            decoded = {
                packet: "Positing status",
                utcTime: utc,
                battery: batt
            }
        }

    } else if (packetID == 0x0D) {
        const ec = (bytes[1] << 24 | bytes[2] << 16 | bytes[3] << 8 | bytes[4]);

        decoded = {
            packet: "Positioning Timeout",
            longitude: 0,
            latitude: 0,
            errorCode: ec
        }

    } else {
        decoded = {
            packetType: "unkown",
            packetID: packetID
        }
    }

    return {
        data: decoded
    }

}

function convertToDec(b) {
    var return_value = parseInt(b, 16);
    return return_value;
}

function unixToDateTime(unixtime) {
    var date = new Date(unixtime * 1000);
    var months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec'];
    var hours = date.getHours();
    var minutes = "0" + date.getMinutes();
    var seconds = "0" + date.getSeconds();
    var year = date.getFullYear();
    var month = months[date.getMonth()];
    var day = date.getDate();
    var DateTime = day + '-' + month + '-' + year + ' ' + hours + ':' + minutes.substr(-2) + ':' + seconds.substr(-2);
    return DateTime;
}

function getPosStrategy(ps) {
    var psText = "";
    switch (ps) {
        case 0:
            psText = "Only GNSS";
            break;
        case 1:
            psText = "Only WiFi";
            break;
        case 2:
            psText = "WiFi + GNSS";
            break;
        case 3:
            psText = "GNSS + WiFi";
            break;
        case 4:
            psText = "Only Bluetooth";
            break;
        case 5:
            psText = "Bluetooth + WiFi";
            break;
        case 6:
            psText = "Bluetooth + GNSS";
            break;
        case 7:
            psText = "Bluetooth + WiFi + GNSS";
            break;
        default:
            psText = "unkown";
    }
    return psText;
}

function getWorkMode(wm) {
    var wmText = "";
    switch (wm) {
        case 0:
            wmText = "Standby Mode";
            break;
        case 1:
            wmText = "Periodic Mode";
            break;
        case 2:
            wmText = "Event Mode";
            break;
        default:
            wmText = "unkown";
    }
    return wmText;
}

this will get me a fine decoded payload:

Screenshot 2024-10-08 at 11 41 01 AM

and now i get this:

Logger: ttn_client.parsers.sensecap
Quelle: components/thethingsnetwork/coordinator.py:51
Erstmals aufgetreten: 09:21:03 (3310 Vorkommnisse)
Zuletzt protokolliert: 10:55:50

Ignoring message without valid=true for device eui-2cf7f1c05410039a: {'battery': 90, 'eventStatus': 0, 'latitude': 51.8xxxx 'light': 100, 'longitude': 8.8xxxxx, 'motionCount': 0, 'packet': 'GNSS Location & Sensor Data', 'temperature': 23.3, 'utcTime': '8-Oct-2024 8:21:18'}
Ignoring message without valid=true for device eui-2cf7f1c05410039a: {'battery': 90, 'eventStatus': 0, 'latitude': 51.8xxxx 'light': 100, 'longitude': 8.8xxxxx, 'motionCount': 0, 'packet': 'GNSS Location & Sensor Data', 'temperature': 23.4, 'utcTime': '8-Oct-2024 8:48:25'}
Ignoring message without valid=true for device eui-2cf7f1c05410039a: {'battery': 90, 'eventStatus': 0, 'latitude': 51.8xxxx, 'light': 100, 'longitude': 8.8xxxxx, 'motionCount': 0, 'packet': 'GNSS Location & Sensor Data', 'temperature': 23.4, 'utcTime': '8-Oct-2024 8:49:29'}
Ignoring message without valid=true for device eui-2cf7f1c05410039a: {'battery': 90, 'eventStatus': 0, 'latitude': 51.8xxxx, 'light': 100, 'longitude': 8.8xxxxx, 'motionCount': 0, 'packet': 'GNSS Location & Sensor Data', 'temperature': 23.4, 'utcTime': '8-Oct-2024 8:50:30'}
Ignoring message without valid=true for device eui-2cf7f1c05410039a: {'battery': 90, 'eventStatus': 0, 'latitude': 51.8xxxx, 'light': 100, 'longitude': 8.8xxxxx'motionCount': 0, 'packet': 'GNSS Location & Sensor Data', 'temperature': 23.4, 'utcTime': '8-Oct-2024 8:53:31'}

but also if i want to add a new sensor, i get this now:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 354, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/thethingsnetwork/coordinator.py", line 51, in _async_update_data
    measurements = await self._client.fetch_data()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ttn_client/client.py", line 57, in fetch_data
    return await self.__storage_api_call(f"?last={fetch_last}&order=received_at")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ttn_client/client.py", line 104, in __storage_api_call
    ttn_values[device_id] = ttn_parse(application_up)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ttn_client/parsers/__init__.py", line 23, in ttn_parse
    return parser(uplink_data)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/ttn_client/parsers/sensecap.py", line 35, in sensecap_parser
    uplink_data, field, decoded_payload[field]
                        ~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'err'

so i guess,my decoder is missing a field with valid=true am i correct?

home-assistant / core