External Converters not loading with 1.36.1 and Home Assistant 2024.4.0

MrEcosse commented 5 months ago

What happened?

None of my external converters to support new devices are working with this combination of Z2M and HA. These were working with the previous versions. Sorry, I don't know whether this happened as a result of the Z2M upgrade or the new April version of HA released today. Z2M01 Z2M02 Z2M03

What did you expect to happen?

To use the external converters specified in the config file as it did before.

How to reproduce it (minimal and precise)

Add an unsupported device to the network Create an external converter with correct fingerprint Add the path to the .js file in the configuration.yaml Restart HA Note that the device is listed as unsupported still

Zigbee2MQTT version

1.36.1

Adapter firmware version

20230707

Adapter

zStack3x0

Setup

Add-on in Home Assistant on Intel NUC

Debug log

No response

MrEcosse commented 5 months ago

Here is the error in the log:

Zigbee2MQTT:error 2024-04-04 14:49:38: Failed to load external converter file 'ZSSQYGLUENGas.js' (Cannot find module 'zigbee-herdsman-converters/lib/extend' Require stack:

/app/dist/util/utils.js
/app/dist/util/settings.js
/app/index.js) Zigbee2MQTT:error 2024-04-04 14:49:38: Probably there is a syntax error in the file or the external converter is not compatible with the current Zigbee2MQTT version Zigbee2MQTT:error 2024-04-04 14:49:38: Note that external converters are not meant for long term usage, it's meant for local testing after which a pull request should be created to add out-of-the-box support for the device

MrEcosse commented 5 months ago

Have added the line const {} = require('zigbee-herdsman-converters/lib/modernExtend'); To my external converter file but still seeing the error:

error 2024-04-04 15:05:28: Failed to load external converter file 'DINRelay.js' (Cannot find module 'zigbee-herdsman-converters/lib/extend' Require stack:

/app/dist/util/utils.js
/app/dist/util/settings.js
/app/index.js)
Note this tried on a different device that I really need to get working again!

MrEcosse commented 5 months ago

I think that the documentation may be incorrect.

It appears that the line

const extend = require('zigbee-herdsman-converters/lib/extend');

should be removed from the external definitions and the line

const {} = require('zigbee-herdsman-converters/lib/modernExtend');

added in its place.

If so the example from the documentation at https://www.zigbee2mqtt.io/advanced/support-new-devices/01_support_new_devices.html#_2-1-extending-the-external-definition is incorrect.

Great-Chart commented 5 months ago

I don't have any external converters myself but I did note others have had such issues. You might find some clues in #22076 with a similar error message and OP's solution(s) seemed to resolve things.

MrEcosse commented 5 months ago

Thanks for the reference - I will try this out. However if this solves the issue the documentation is definitely incorrect.

aurelmarius commented 5 months ago

Not sure is the same problem but, running latest dev build 1.36.1-dev commit: 3e91349. I had 2 IKEA Symfonisk Gen2 remotes paired and noticed play/pause button not publishing state to mqtt I tried to re-pair but clusters doesn't bind to coordinator anymore Failed to configure '0x540f57fffe449664', attempt 3 (Error: Device 0x540f57fffe449664 has no output cluster genPollCtrl at getEndpointsWithCluster (/app/node_modules/zigbee-herdsman-converters/src/lib/modernExtend.ts:28:15) at configure (/app/node_modules/zigbee-herdsman-converters/src/lib/modernExtend.ts:126:31) at Object.configure (/app/node_modules/zigbee-herdsman-converters/src/index.ts:164:27) at Configure.configure (/app/lib/extension/configure.ts:122:37) at /app/lib/extension/configure.ts:87:62 at EventEmitter.wrappedCallback (/app/lib/eventBus.ts:174:23) at EventEmitter.emit (node:events:529:35) at EventBus.emitLastSeenChanged (/app/lib/eventBus.ts:51:22) at Controller. (/app/lib/zigbee.ts:74:27) at Controller.emit (node:events:517:28))

mrskycriper commented 5 months ago

Thanks for the reference - I will try this out. However if this solves the issue the documentation is definitely incorrect.

Yeah, it wasn't updated in time. Legacy extend was removed in zhc 19.0

mrskycriper commented 5 months ago

Not sure is the same problem but, running latest dev build 1.36.1-dev commit: 3e91349. I had 2 IKEA Symfonisk Gen2 remotes paired and noticed play/pause button not publishing state to mqtt I tried to re-pair but clusters doesn't bind to coordinator anymore Failed to configure '0x540f57fffe449664', attempt 3 (Error: Device 0x540f57fffe449664 has no output cluster genPollCtrl at getEndpointsWithCluster (/app/node_modules/zigbee-herdsman-converters/src/lib/modernExtend.ts:28:15) at configure (/app/node_modules/zigbee-herdsman-converters/src/lib/modernExtend.ts:126:31) at Object.configure (/app/node_modules/zigbee-herdsman-converters/src/index.ts:164:27) at Configure.configure (/app/lib/extension/configure.ts:122:37) at /app/lib/extension/configure.ts:87:62 at EventEmitter.wrappedCallback (/app/lib/eventBus.ts:174:23) at EventEmitter.emit (node:events:529:35) at EventBus.emitLastSeenChanged (/app/lib/eventBus.ts:51:22) at Controller. (/app/lib/zigbee.ts:74:27) at Controller.emit (node:events:517:28))

I'll investigate that, please create an issue. It might be connected to the recent IKEA refactoring in the dev branch

Murada99 commented 5 months ago

Same problem, seems to be caused by upgrading Z2M as I didn't upgrade HA (2024.3.3)

kloodhu commented 5 months ago

same problem here.

elmaswebon commented 5 months ago

Yep, same here since the last two updates (1.36.1-0 and 1.36.1-1) with 2 different external files. Nothing to do with IKEA.

Edit: THIS solved the problem, thank you very much @MrEcosse

I think that the documentation may be incorrect.

It appears that the line

const extend = require('zigbee-herdsman-converters/lib/extend');

should be removed from the external definitions and the line

const {} = require('zigbee-herdsman-converters/lib/modernExtend');

added in its place.

If so the example from the documentation at https://www.zigbee2mqtt.io/advanced/support-new-devices/01_support_new_devices.html#_2-1-extending-the-external-definition is incorrect.

Chartreusito commented 5 months ago

Isn't that what the latest release Notes are about ? https://github.com/Koenkk/zigbee2mqtt/releases/tag/1.36.1

MrEcosse commented 5 months ago

I think this has morphed into the necessity to update the documentation which does not match the release notes

Jacksoldano commented 4 months ago

Legend thanks for this was tearing my hair out trying to get an External converter working last night stumbled on this post and this resolved my .js issue!

ayybe commented 1 week ago

Config path with Proxmox Debian LXC: /opt/zigbee2mqtt/data/configuration.yaml

Alpine: nano /etc/zigbee2mqtt/configuration.yaml

add this to z2m configuration.yaml (include empty line)

external_converters:

_TZE200_libht6ua.js

touch _TZE200_libht6ua.js then nano _TZE200_libht6ua.js in the same dir as configuration.yaml with this code:

`const {} = require('zigbee-herdsman-converters/lib/modernExtend');

const zigbeeHerdsmanConverters = require('zigbee-herdsman-converters'); const zigbeeHerdsmanUtils = require('zigbee-herdsman-converters/lib/utils');

const fz = require('zigbee-herdsman-converters/converters/fromZigbee'); const tz = require('zigbee-herdsman-converters/converters/toZigbee'); const exposes = require('zigbee-herdsman-converters/lib/exposes'); const reporting = require('zigbee-herdsman-converters/lib/reporting'); const utils = require('zigbee-herdsman-converters/lib/utils'); const globalStore = require('zigbee-herdsman-converters/lib/store'); const e = exposes.presets; const ea = exposes.access; const tuya = require('zigbee-herdsman-converters/lib/tuya');

const definition = { // Since a lot of TuYa devices use the same modelID, but use different datapoints // it's necessary to provide a fingerprint instead of a zigbeeModel fingerprint: [ { // The model ID from: Device with modelID 'TS0601' is not supported // You may need to add \u0000 at the end of the name in some cases modelID: 'TS0601', // The manufacturer name from: Device with modelID 'TS0601' is not supported. manufacturerName: '_TZE200_libht6ua', }, ], model: 'TS0601_cover', vendor: 'TuYa', description: 'Cover motor', fromZigbee: [tuya.fz.datapoints], toZigbee: [tuya.tz.datapoints], onEvent: tuya.onEventSetTime, // Add this if you are getting no converter for 'commandMcuSyncTime' configure: tuya.configureMagicPacket, exposes: [ e.cover_position().setAccess('position', ea.STATE_SET),e.battery(),e.binary('motor_fault', ea.STATE, true, false), e.enum('motor_direction', ea.STATE_SET, ['left', 'right']).withDescription('Motor side'), e.enum('set_upper_limit', ea.STATE_SET, ['start', 'stop']).withDescription('Learning') ], // Here you should put all functionality that your device exposes

meta: {
    // All datapoints go in here
    tuyaDatapoints: [
        [1, 'state', tuya.valueConverterBasic.lookup({'OPEN': tuya.enum(2), 'STOP': tuya.enum(1), 'CLOSE': tuya.enum(0)})],
        [2, 'position', tuya.valueConverter.coverPosition],
        [3, 'position', tuya.valueConverter.raw],
        [12, 'motor_fault', tuya.valueConverter.trueFalse1],
        [13, 'battery', tuya.valueConverter.raw],
        [101, 'motor_direction', tuya.valueConverterBasic.lookup({ 'left': tuya.enum(0), 'right': tuya.enum(1) })],
        [102, 'set_upper_limit', tuya.valueConverterBasic.lookup({ 'start': tuya.enum(0), 'stop': tuya.enum(1) })],

         //The datapoints below expose values but I can't detect why, I don't find them useful.
        [104], //when opening or closing and charging or not, expose value: 10
        [105], //when opening or closing and charging or not, expose value: 50 
        [7], //when opening or closing and charging or not, expose value: 2,
        [103], //maybe temperature, expose value: 22, 23, 28, 30, 50.. 
        [106], //when opening or closing and charging or not, expose value: 100,
        [108], //when opening or closing and charging or not, not expose value: "nothing",
    ],
},

};

module.exports = definition;`

Koenkk / zigbee2mqtt