Possible Memory Leak

[Fiftiz]

Hello, think there a memory leak with tuya-mqtt with 3.3 protocol. tested on 2 raspberry with openhab (one zero and one 3B+)

In 8 hours tuya-mqtt.js needed 186MB of memory and it continues to increase

openhab+ 359 0.2 18.7 291472 186996 ? Ssl Jul25 2:55 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt

During my test with DEBUG. I saw that, randomly, the socket wasn't disconnected but there is an infinity of ping pong. More we use the tuya-mqtt command,, more the memory usage increases exponentially.

8:35AM i kill the process tuya-mqtt.js , 38MB , it's ok , 577MB available openhab+ 13179 19.6 3.8 131740 38376 ? Ssl 08:27 0:00 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js

01:52 PM 564MB available and I launch some command on tuya devices.

09:16 PM 519 MB available openhab+ 13179 0.1 7.6 179136 75928 ? Ssl 08:27 0:48 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js

Thank you

[tsightler]

Can you share what version of node you are running? I'm not currently able to reproduce this issue. I have 18 switches and I have toggled all of them on/off and left it running for a few hours and memory usage has not exceeded 40MB.

[Fiftiz]

I used node v10.16.0 look at the end, there is ping pong , here 2 times, but it's infinite, never it disconnect socket

[00:30:33] openhabian@openHABianPi:/etc/openhab2/scripts/tuyaapi_mqtt$ DEBUG=* node tuya-mqtt.js
  TuyAPI:mqtt MQTT-Server nicht verbunden. +0ms
  TuyAPI:mqtt Verbindung mit MQTT-Server hergestellt +44ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/command","action":"command","message":"1","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +8s
  TuyAPI:device Search device in network +0ms
  TuyAPI Finding missing IP undefined or ID 46200800c44f33889bfa +0ms
  TuyAPI Received UDP message. +289ms
  TuyAPI UDP data: +9ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.91',
  TuyAPI      gwId: '34236438840d8e9c0016',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'r15JnoGqWPM4PPtl',
  TuyAPI      version: '3.1' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 0,
  TuyAPI   sequenceN: 0 } +0ms
  TuyAPI Received UDP message. +15ms
  TuyAPI UDP data: +2ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.20',
  TuyAPI      gwId: '46200800dc4f22a6fcd0',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'ON0fHlAaAf7Pt7ja',
  TuyAPI      version: '3.3' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 19,
  TuyAPI   sequenceN: 0 } +0ms
  TuyAPI Received UDP message. +153ms
  TuyAPI UDP data: +1ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.42',
  TuyAPI      gwId: '86478067b4e62d6452df',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'r15JnoGqWPM4PPtl',
  TuyAPI      version: '3.1' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 0,
  TuyAPI   sequenceN: 0 } +1ms
  TuyAPI Received UDP message. +78ms
  TuyAPI UDP data: +1ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.31',
  TuyAPI      gwId: '605306003c71bf27f5fd',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'r15JnoGqWPM4PPtl',
  TuyAPI      version: '3.1' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 0,
  TuyAPI   sequenceN: 0 } +1ms
  TuyAPI Received UDP message. +552ms
  TuyAPI UDP data: +3ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.89',
  TuyAPI      gwId: '46200800dc4f22a6f2c5',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'ON0fHlAaAf7Pt7ja',
  TuyAPI      version: '3.3' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 19,
  TuyAPI   sequenceN: 0 } +0ms
  TuyAPI Received UDP message. +29ms
  TuyAPI UDP data: +2ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.7',
  TuyAPI      gwId: '87046830cc50e320d66f',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'r15JnoGqWPM4PPtl',
  TuyAPI      version: '3.1' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 0,
  TuyAPI   sequenceN: 0 } +1ms
  TuyAPI Received UDP message. +1s
  TuyAPI UDP data: +3ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.40',
  TuyAPI      gwId: '86478067ecfabca490b2',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'r15JnoGqWPM4PPtl',
  TuyAPI      version: '3.1' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 0,
  TuyAPI   sequenceN: 0 } +0ms
  TuyAPI Received UDP message. +901ms
  TuyAPI UDP data: +2ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.24',
  TuyAPI      gwId: '46200800dc4f22a6f863',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'ON0fHlAaAf7Pt7ja',
  TuyAPI      version: '3.3' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 19,
  TuyAPI   sequenceN: 0 } +1ms
  TuyAPI Received UDP message. +157ms
  TuyAPI UDP data: +2ms
  TuyAPI { payload:
  TuyAPI    { ip: '192.168.1.98',
  TuyAPI      gwId: '46200800c44f33889bfa',
  TuyAPI      active: 2,
  TuyAPI      ability: 0,
  TuyAPI      mode: 0,
  TuyAPI      encrypt: true,
  TuyAPI      productKey: 'ON0fHlAaAf7Pt7ja',
  TuyAPI      version: '3.3' },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 19,
  TuyAPI   sequenceN: 0 } +1ms
  TuyAPI:device Device found in network +3s
  TuyAPI Connecting to 192.168.1.98... +12ms
  TuyAPI Socket connected. +10ms
  TuyAPI:device Connected to device.  (undefined, 46200800c44f33889bfa, edd1197f2fd2ddcd) +17ms
  TuyAPI GET Payload: +4ms
  TuyAPI { gwId: '46200800c44f33889bfa', devId: '46200800c44f33889bfa' } +0ms
  TuyAPI:mqtt receive command { set: true } +3s
  TuyAPI:device set: { set: true } +22ms
  TuyAPI SET Payload: +23ms
  TuyAPI { devId: '46200800c44f33889bfa',
  TuyAPI   gwId: '46200800c44f33889bfa',
  TuyAPI   uid: '',
  TuyAPI   t: 1564180292,
  TuyAPI   dps: { '1': true } } +0ms
  TuyAPI Received data: 000055aa000000010000000a0000005c00000000d5020582b49a614c1df7e244153653aa8ee6a14200dfead082a36e26a3f9407f6830d00c6f15ba971439d19d5c767a34a467e5f5dc99fa6f486a9816bd160f8e7549ff9507e69826831d76f898f3f7c4b81ad01c0000aa55 +16ms
  TuyAPI Parsed: +2ms
  TuyAPI { payload:
  TuyAPI    { devId: '46200800c44f33889bfa',
  TuyAPI      dps: { '1': false, '101': '0', '102': 0 } },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 10,
  TuyAPI   sequenceN: 1 } +0ms
  TuyAPI:device Data from device: { devId: '46200800c44f33889bfa',
  dps: { '1': false, '101': '0', '102': 0 } } +68ms
  TuyAPI:mqtt:device Data from device undefined : { devId: '46200800c44f33889bfa',
  dps: { '1': false, '101': '0', '102': 0 } } +0ms
  TuyAPI:mqtt:device mqtt status updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/state -> OFF +3ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps ->  {"1":false,"101":"0","102":0} +1ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/1 -> dps[1] false +2ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/101 -> dps[101] "0" +0ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/102 -> dps[102] 0 +1ms
  TuyAPI Received data: 000055aa00000002000000070000000c0000000018cfc5da0000aa55 +57ms
  TuyAPI Parsed: +1ms
  TuyAPI { payload: false, leftover: false, commandByte: 7, sequenceN: 2 } +0ms
  TuyAPI Got SET ack. +1ms
  TuyAPI Received data: 000055aa00000000000000080000006b00000000332e33000000000000008800000001d5020582b49a614c1df7e244153653aa8ee6a14200dfead082a36e26a3f9407f0c73ebb30d041c6297d9a88693e2fbc3149ea04973f2889121a49df94e8c79b8598ba312926810b3fcfe583aa43ceaf3ff7a3cde0000aa55 +38ms
  TuyAPI Parsed: +2ms
  TuyAPI { payload:
  TuyAPI    { devId: '46200800c44f33889bfa',
  TuyAPI      dps: { '1': true },
  TuyAPI      t: 1564180292 },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 8,
  TuyAPI   sequenceN: 0 } +0ms
  TuyAPI:device Data from device: { devId: '46200800c44f33889bfa',
  dps: { '1': true },
  t: 1564180292 } +54ms
  TuyAPI:mqtt:device Data from device undefined : { devId: '46200800c44f33889bfa',
  dps: { '1': true },
  t: 1564180292 } +47ms
  TuyAPI:mqtt:device mqtt status updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/state -> ON +2ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps ->  {"1":true} +1ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/1 -> dps[1] true +1ms
  TuyAPI GET Payload: +10ms
  TuyAPI { gwId: '46200800c44f33889bfa', devId: '46200800c44f33889bfa' } +1ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/state","action":"state","message":"OFF","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +137ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps","action":"dps","message":"{\"1\":false,\"101\":\"0\",\"102\":0}","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +2ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/1","action":"dps","message":"false","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +1ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/101","action":"dps","message":"\"0\"","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +2ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/102","action":"dps","message":"0","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +4ms
  TuyAPI Received data: 000055aa000000030000000a0000005c00000000d5020582b49a614c1df7e244153653aa8ee6a14200dfead082a36e26a3f9407f71aadd25dccd472dcf3a4d8e97b9d1e1bae473963c3bebc57a8fe220525eee0fbbd3c158f49dcdaebee388dcb570d63c68ca27bb0000aa55 +27ms
  TuyAPI Parsed: +2ms
  TuyAPI { payload:
  TuyAPI    { devId: '46200800c44f33889bfa',
  TuyAPI      dps: { '1': true, '101': '0', '102': 0 } },
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 10,
  TuyAPI   sequenceN: 3 } +0ms
  TuyAPI:device Data from device: { devId: '46200800c44f33889bfa',
  dps: { '1': true, '101': '0', '102': 0 } } +39ms
  TuyAPI:mqtt:device Data from device undefined : { devId: '46200800c44f33889bfa',
  dps: { '1': true, '101': '0', '102': 0 } } +35ms
  TuyAPI:mqtt:device mqtt status updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/state -> ON +0ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps ->  {"1":true,"101":"0","102":0} +1ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/1 -> dps[1] true +1ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/101 -> dps[101] "0" +1ms
  TuyAPI:mqtt:device mqtt dps updated to:tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/102 -> dps[102] 0 +1ms
  TuyAPI:device set completed  +7ms
  TuyAPI:mqtt set device status completed { devId: '46200800c44f33889bfa',
  dps: { '1': true },
  t: 1564180292 } +24ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/state","action":"state","message":"ON","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +15ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps","action":"dps","message":"{\"1\":true}","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +15ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/1","action":"dps","message":"true","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +2ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/state","action":"state","message":"ON","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +41ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps","action":"dps","message":"{\"1\":true,\"101\":\"0\",\"102\":0}","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +4ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/1","action":"dps","message":"true","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +3ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/101","action":"dps","message":"\"0\"","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +1ms
  TuyAPI:mqtt receive settings {"topic":"tuya/46200800c44f33889bfa/edd1197f2fd2ddcd/discover/dps/102","action":"dps","message":"0","options":{"id":"46200800c44f33889bfa","key":"edd1197f2fd2ddcd","ip":"discover"}} +2ms
  TuyAPI Pinging 192.168.1.98 +10s
  TuyAPI Received data: 000055aa00000000000000090000000c00000000b051ab030000aa55 +8ms
  TuyAPI Parsed: +1ms
  TuyAPI { payload: false, leftover: false, commandByte: 9, sequenceN: 0 } +0ms
  TuyAPI Pong from 192.168.1.98 +1ms
  TuyAPI Pinging 192.168.1.98 +10s
  TuyAPI Received data: 000055aa00000000000000090000000c00000000b051ab030000aa55 +23ms
  TuyAPI Parsed: +1ms
  TuyAPI { payload: false, leftover: false, commandByte: 9, sequenceN: 0 } +1ms
  TuyAPI Pong from 192.168.1.98 +0ms
  TuyAPI Pinging 192.168.1.98 +10s
  TuyAPI Received data: 000055aa00000000000000090000000c00000000b051ab030000aa55 +50ms
  TuyAPI Parsed: +1ms
  TuyAPI { payload: false, leftover: false, commandByte: 9, sequenceN: 0 } +1ms
  TuyAPI Pong from 192.168.1.98 +1ms

[tsightler]

Yes, as I've stated before, the ping-pong is normal whenever TuyAPI has an active connection to the tuya device, this shows that the connection is active and the device is online. Keeping the active connection is required to get state updates from the device, for example if you turn the device on and off manually or via the Smart Life app. If we didn't maintain the connection to the device these state updates would be lost. You should see these for every device to which a connection is made.

But, I can't really explain why you are seeing such memory usage, I'm unable to reproduce this at all. I'm also running node 10.16.0 and, after 4 hours of monitoring all 18 devices in my house, with many state changes, etc. the script is only using 34MB of RAM. I will continue to attempt to reproduce, but I can't figure out what might be different.

[tsightler]

OK, maybe something a little interesting, even if I don't really understand it at this point. My development platform for this code is an small x86 VM with 512MB of RAM running CentOS 7 with the nodesource distro of node 10.16.0 installed. I see no significant memory usage on this machine when running the script no matter what I do.

However, when I run the code on my RPi 3b+, which is running Raspbian Buster with distro default version of node 10.15.2, I think I may be seeing the behavior you describe, i.e. constant memory increase when the script is running as, even though it's only been running a short while, it's already at 48MB and seems to be slowly but consistently increasing.

I'm not sure what this means however, why the code would behave differently just due to underlying architecture is definitely not clear to me, but I'll keep digging. At least I'm seeing something that appears to match the behavior you describe.

[Fiftiz]

perfect if you can to reproduce the problem, for my part I seek the solution too. But this problem was not present with the previous version (without 3.3 protocol) so we must, I think, look at this side.

[Fiftiz]

I tried without discover, same problem

[11:22:22] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.1  3.8 144212 38024 ?        Ssl  10:58   0:02 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 15383  0.0  0.0   5992   524 pts/0    S+   11:22   0:00 grep --color=auto tuya-mqtt.js
[11:22:43] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.2  3.8 143364 38152 ?        Ssl  10:58   0:04 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 15572  0.0  0.0   5992   540 pts/0    S+   11:27   0:00 grep --color=auto tuya-mqtt.js
[11:27:46] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.2  4.2 146060 42536 ?        Ssl  10:58   0:08 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 16897  0.0  0.0   5992   544 pts/0    S+   12:04   0:00 grep --color=auto tuya-mqtt.js
[12:04:20] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.2  4.4 148108 44588 ?        Ssl  10:58   0:11 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 17728  0.0  0.0   5992   540 pts/0    S+   12:26   0:00 grep --color=auto tuya-mqtt.js
[12:26:46] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.2  4.6 149800 46264 ?        Ssl  10:58   0:12 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 18388  0.0  0.0   5992   548 pts/0    S+   12:44   0:00 grep --color=auto tuya-mqtt.js
[12:44:04] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.1  4.7 151848 47580 ?        Ssl  10:58   0:14 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 18974  0.0  0.0   5992   536 pts/0    S+   12:59   0:00 grep --color=auto tuya-mqtt.js
[12:59:18] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.1  4.8 152496 48868 ?        Ssl  10:58   0:15 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 19442  0.0  0.0   5992   572 pts/0    S+   13:11   0:00 grep --color=auto tuya-mqtt.js
[13:11:35] openhabian@openHABianPi:/etc/openhab2/scripts/bash_scripts$ ps aux | grep tuya-mqtt.js
openhab+ 14432  0.1  5.7 160916 57344 ?        Ssl  10:58   0:24 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js
openhab+ 22832  0.0  0.0   5992   536 pts/0    S+   14:34   0:00 grep --color=auto tuya-mqtt.js

[tsightler]

I believe this problem is not related to the code in tuya-mqtt, but rather in tuyapi. To support the newer protocol required upgrading from 4.0.3 tuyapi to latest 5.1.1. There is another project (ioBroker.tuya) that has recently moved to the 5.1.1 tuyapi codebase and some users are reporting a similar issue (100's of MB of RAM used).

I've opened an issue with tuyapi team and have been doing a lot of testing. Hopefully we'll get to the bottom of it. In the interim, I have an idea on how I might be able to work around it the issue on my side, but it may take me a few days before I have time to code that up.

[Fiftiz]

Thank you very much @tsightler. I wait :)

[tsightler]

Hi @Fiftiz, I have pushed an update to my repo which includes a hackish workaround that should keep the memory leak from occurring. Basically, it just sets a timer which disconnects and reconnects all devices every hour. This is of course not the final fix, but until we manage to track down what is eating the memory in tuyapi, this seems to at least be a method to keep the code from growing without bounds with very little side effect.

I've only tested the code for ~8 hours, but after that time the memory size was still <40MB, while, without this workaround, the code was easily using 70-80MB in that same time. Could you please give it a spin and let me know if it helps in your case? Thanks!

[Fiftiz]

HI tsightler , I tried but error :

[14:33:21] openhabian@openHABianPi:/etc/openhab2/scripts/tuyaapi_mqtt$ DEBUG=* node tuya-mqtt.js
  TuyAPI:mqtt Connection established to MQTT server +0ms
  TuyAPI:mqtt:error ReferenceError: ip is not defined
  TuyAPI:mqtt:error     at getDeviceFromTopic (/etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js:80:9)
  TuyAPI:mqtt:error     at MqttClient.<anonymous> (/etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js:333:24)
  TuyAPI:mqtt:error     at MqttClient.emit (events.js:198:13)
  TuyAPI:mqtt:error     at Arguments.<anonymous> (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/mqtt/lib/client.js:1150:27)
  TuyAPI:mqtt:error     at Object.MqttClient.options.customHandleAcks (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/mqtt/lib/client.js:152:150)
  TuyAPI:mqtt:error     at MqttClient._handlePublish (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/mqtt/lib/client.js:1143:15)
  TuyAPI:mqtt:error     at MqttClient._handlePacket (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/mqtt/lib/client.js:351:12)
  TuyAPI:mqtt:error     at work (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/mqtt/lib/client.js:283:12)
  TuyAPI:mqtt:error     at Writable.writable._write (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/mqtt/lib/client.js:294:5)
  TuyAPI:mqtt:error     at doWrite (/etc/openhab2/scripts/tuyaapi_mqtt/node_modules/readable-stream/lib/_stream_writable.js:428:64) +0ms

OK it work with discover fonction but not with ver3.3 + IP

[tsightler]

Sorry, it was late last night and I didn't test it with "discover" option, stupid mistake. I'll fix it shortly and make sure to test all of the topic scenarios and then push another update in the next couple of hours.

[Fiftiz]

perfect @tsightler , thank you :)

[tsightler]

Just pushed an update that should fix the workaround code. I've tested all topics types and formats so hopefully this will squash it, it was a silly oversight due to my haste to get the workaround coded up last night.

[Fiftiz]

Thank you, thank you very much @tsightler , now let's go to test ;) I send you a message in a few hours

[Fiftiz]

Hello @tsightler , for the moment there is 61MB used by the script from 19hours

openhab+ 11269 0.1 6.1 164996 61304 ? Ssl Jul29 1:27 /usr/bin/node /etc/openhab2/scripts/tuyaapi_mqtt/tuya-mqtt.js

The memory increase is slower. I will continu to check today ;) I don't see side effect. Just before to disconnect device, I see an increase of memory used (1 or 2MB) and after a few minutes it decreased.

[tsightler]

Thanks for the update. Let's see how it behaves over the next 24 hours.

In the interim, I think I've identified the issue in tuyapi that is causing the leak (at least the significant one) and have a fix for it in my in-house tree. I'll get a pull request submitted for it and hopefully it will be merged in the next few days so the hacking workaround in the script won't be needed.

This is good because I want to get back on working on the cleanup and other features I have planned, as well as finalizing the protocol 3.3 support and getting it pushed to the main tree.

[Fiftiz]

After 24hours, the scripts used between 56MB and 61MB so the hack work :) thank you. Just to know, where is the problem with tuya 5.1.1 ?

[tsightler]

Something to do with the resolver code. I think we've decided that it's just not needed for heartbeat. I've been testing with this patch and it seems to completely eliminate the problem.

[tsightler]

The tuyapi team accepted the patch I submitted for the heartbeat code and pushed it to version 5.1.2. This should eliminate 99% of the memory leak. There's still a small leak in set command but only a few bytes per command so it would take many months and lots of commands to really be noticeable. The plan is to fix this for tuyapi 6.0 which is currently in planning phase. I'll follow this development and make sure tuya-mqtt is ready for it.

In the meantime, I've pushed a new version of tuya-mqtt to my tree which removes the hack I implemented and bumps the tuyapi dependencies to 5.1.2. If you would have some time to test it would be appreciated.

Be warned, the topic format used in my tree at the moment is likely to change before I push this code into the main tree. I'm just not happy with how the protocol and discover terms are used so I'm going to work on that a bit before I try to push my changes into the official tree.

[Fiftiz]

Hi @tsightler , wow perfect, ok I install this new version today , I test and I will make a feeboack. Thank you very much for your help, really appreciate

[Fiftiz]

Hi, 8 hours , always 38 / 39 MB, it's perfect. I will check in 24 hours :)

[Fiftiz]

Hello, After more than 24hours, 40MB :) it's really nice, thank you @tsightler

[tsightler]

Thanks for reporting and for providing detailed information, sorry it took a while but memory leaks are always a tough one, especially since this one wasn't even in the script itself, but in an external library.