search

mblackstock / node-red-contrib-influxdb

Node-RED nodes to save and query data from an influxdb time series database
Other
64 stars 46 forks source link

Issue wit Node Red and Influx 1.7 #88

Closed ajaywazir closed 3 years ago

ajaywazir commented 3 years ago

Hi I am using node red with 4 mobbus TCP devices being read in parallel. the output is sent to Influx node. I cannot seem to write faster than 1 row per second. I have to insert a delay of 1s between each message. Then I am able to save all points.

When I use my same influxDB with Telegraph (for MQTT), I have no problem in writing to same database at a much faster rate? Can you advise what could be an issue? Node v12.16.1 NPM v 6.13.4 node-red-contrib-influxdb v 0.5.4 Node Red V 1.2.2

Regards Ajay

[ { "id": "83e35657.864148", "type": "tab", "label": "Data Collect", "disabled": false, "info": "" }, { "id": "c3a5acc1.1e1ff", "type": "comment", "z": "83e35657.864148", "name": "", "info": " \"site_settings\": \n {\n \"site_id\": \"NUS_SDE1\",\n \"resource_id\": \"117566\",\n \"sender_id\": \"IPC-117566-01\"\n },\n \n\n//60\n192.168.1.101:502\nDEVICE_ID = \"NUS_SDE1_DB01_101\"\nTAG_NAME= IIF(MODBUS_ADDRESS = 2999,\"KW\",MODBUS_ADDRESS)\nTAG_NAME= IIF(MODBUS_ADDRESS = 3203,\"KWH\",TAG_NAME)\nTREND_NAME = \"PV_ACDB001\"\n\n\n//59\n192.168.1.111:502\nDEVICE_ID = \"NUS_SDE1_DB01_111\"\nTAG_NAME= IIF(MODBUS_ADDRESS = 2999,\"KW\",MODBUS_ADDRESS)\nTAG_NAME= IIF(MODBUS_ADDRESS = 3203,\"KWH\",TAG_NAME)\nTREND_NAME = \"PV_ACDB001_CHK\"\n\n//57\n192.168.1.102:502\nDEVICE_ID = \"NUS_SDE1_DB02_102\"\nTAG_NAME= IIF(MODBUS_ADDRESS = 2999,\"KW\",MODBUS_ADDRESS)\nTAG_NAME= IIF(MODBUS_ADDRESS = 3203,\"KWH\",TAG_NAME)\nTREND_NAME = \"PV_ACDB002\"\n\n//58\n192.168.1.122:502\nDEVICE_ID = \"NUS_SDE1_DB02_122\"\nTAG_NAME= IIF(MODBUS_ADDRESS = 2999,\"KW\",MODBUS_ADDRESS)\nTAG_NAME= IIF(MODBUS_ADDRESS = 3203,\"KWH\",TAG_NAME)\nTREND_NAME = \"PV_ACDB002_CHK\"", "x": 130, "y": 120, "wires": [] }, { "id": "8f188d2a.3893b", "type": "modbus-flex-getter", "z": "83e35657.864148", "name": "NUS_SDE1_DB01_101", "showStatusActivities": true, "showErrors": true, "logIOActivities": false, "server": "fcaddb6a.9b1fa8", "useIOFile": false, "ioFile": "", "useIOForPayload": false, "emptyMsgOnFail": false, "keepMsgProperties": true, "x": 770, "y": 200, "wires": [ [], [ "a29dfdc8.a3477" ] ] }, { "id": "edbf4eaa.54f93", "type": "inject", "z": "83e35657.864148", "name": "", "props": [ { "p": "payload" }, { "p": "topic", "vt": "str" } ], "repeat": "", "crontab": "", "once": false, "onceDelay": 0.1, "topic": "", "payload": "", "payloadType": "date", "x": 120, "y": 400, "wires": [ [ "a5e4f668.721c08" ] ] }, { "id": "76c88d91.e5b1f4", "type": "debug", "z": "83e35657.864148", "name": "", "active": false, "tosidebar": true, "console": false, "tostatus": false, "complete": "payload", "targetType": "msg", "statusVal": "", "statusType": "auto", "x": 1750, "y": 320, "wires": [] }, { "id": "759e4ab7.6136b4", "type": "function", "z": "83e35657.864148", "name": "", "func": "const _g_config = global.get(\"g_config\").site_settings;\nconst sc_ResourceId = _g_config.resource_id;\nconst sc_SenderId = _g_config.sender_id;\nconst sc_site_name = _g_config.site_id;\n//var sc_SenderId = \"IPC-\" + sc_ResourceId + \"-01\";\n\n// \"site_settings\": \n// {\n// \"site_id\": \"NUS_SDE1\",\n// \"resource_id\": \"117566\",\n// \"sender_id\": \"IPC-117566-01\"\n// },\n\nvar d = new Date();\nvar date_now = d.getFullYear() + (\"0\" + (d.getMonth() + 1)).slice(-2) + (\"0\" + d.getDate()).slice(-2);\nvar time_now = (\"0\" + d.getHours()).slice(-2) + (\"0\" + d.getMinutes()).slice(-2) + (\"0\" + d.getSeconds()).slice(-2);\nvar GetEventStamped = date_now + time_now;\n\nvar DEVICE_ID = msg.sensor_id;\nvar sc_sensorid=\"Electrical-Mirai-\"+sc_ResourceId + \"-\" + DEVICE_ID;\n\nvar sc_EventId = \"EV-\" + sc_ResourceId;\n sc_EventId = sc_EventId + \"-\" + DEVICE_ID + \"-\" + GetEventStamped;\n\nvar _payload = {};\n\n _payload.senderid = sc_SenderId;\n _payload.sensorid = sc_sensorid;\n _payload.eventid = sc_EventId;\n \n _payload.eventtype = msg.topic ;\n _payload.sensorstatus = 'normal';\n _payload.severity = 6;\n _payload.tag_value = Math.abs(msg.payload);\n \nmsg.payload = _payload;\n\n\nreturn msg;\n\n", "outputs": 1, "noerr": 0, "initialize": "", "finalize": "", "x": 1600, "y": 380, "wires": [ [ "9dcf23e7.d9dc2" ] ] }, { "id": "a29dfdc8.a3477", "type": "change", "z": "83e35657.864148", "name": "", "rules": [ { "t": "set", "p": "sensor_id", "pt": "msg", "to": "NUS_SDE1_DB01_101", "tot": "str" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 1010, "y": 200, "wires": [ [ "52d23431.e9643c" ] ] }, { "id": "b2b1a852.3d5b28", "type": "influxdb out", "z": "83e35657.864148", "influxdb": "39bb3475.b7e60c", "name": "sensor_data_test", "measurement": "sensor_data_trend", "precision": "s", "retentionPolicy": "OneWeek", "database": "sensor_data_instant", "precisionV18FluxV20": "s", "retentionPolicyV18Flux": "OneWeek", "org": "", "bucket": "", "x": 1730, "y": 600, "wires": [] }, { "id": "ad0ab455.2dca78", "type": "change", "z": "83e35657.864148", "name": "Set Tag Names", "rules": [ { "t": "change", "p": "topic", "pt": "msg", "from": "actPowerTot", "fromt": "str", "to": "EXPORT_KW", "tot": "str" }, { "t": "change", "p": "topic", "pt": "msg", "from": "toEnergyActExp", "fromt": "str", "to": "KWH_DELIVERED", "tot": "str" }, { "t": "change", "p": "topic", "pt": "msg", "from": "toEnergyActImp", "fromt": "str", "to": "KWH_RECEIVED", "tot": "str" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 1420, "y": 380, "wires": [ [ "759e4ab7.6136b4" ] ] }, { "id": "af6faad8.cc81b8", "type": "delay", "z": "83e35657.864148", "name": "", "pauseType": "delay", "timeout": "100", "timeoutUnits": "milliseconds", "rate": "1", "nbRateUnits": "1", "rateUnits": "second", "randomFirst": "1", "randomLast": "5", "randomUnits": "seconds", "drop": false, "x": 310, "y": 460, "wires": [ [ "fedae738.d74aa8", "8559f3b9.9e765", "6571c66.d61b538", "5e40bcb5.1d8534" ] ] }, { "id": "68a861af.c46b5", "type": "pm5500", "z": "83e35657.864148", "name": "kW_60", "unitId": "60", "data": "actPowerTot", "x": 490, "y": 200, "wires": [ [ "8f188d2a.3893b" ] ] }, { "id": "fedae738.d74aa8", "type": "pm5500", "z": "83e35657.864148", "name": "kWh_60", "unitId": "60", "data": "toEnergyActExp", "x": 500, "y": 260, "wires": [ [ "8f188d2a.3893b" ] ] }, { "id": "52d23431.e9643c", "type": "converter", "z": "83e35657.864148", "name": "", "unitId": "", "data": "", "x": 1240, "y": 400, "wires": [ [ "ad0ab455.2dca78", "5a80af8.18e735" ], [] ] }, { "id": "971686e5.ed7118", "type": "cron", "z": "83e35657.864148", "name": "every 30 sec", "crontab": "0,30 * * * * *", "x": 110, "y": 340, "wires": [ [ "a5e4f668.721c08" ] ] }, { "id": "67a8a4f1.1077fc", "type": "modbus-flex-getter", "z": "83e35657.864148", "name": "NUS_SDE1_DB01_111", "showStatusActivities": true, "showErrors": true, "logIOActivities": false, "server": "f30cd4fd.aad048", "useIOFile": false, "ioFile": "", "useIOForPayload": false, "emptyMsgOnFail": false, "keepMsgProperties": true, "x": 770, "y": 340, "wires": [ [], [ "ca1e912.e583e7" ] ] }, { "id": "ca1e912.e583e7", "type": "change", "z": "83e35657.864148", "name": "", "rules": [ { "t": "set", "p": "sensor_id", "pt": "msg", "to": "NUS_SDE1_DB01_111", "tot": "str" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 1010, "y": 340, "wires": [ [ "52d23431.e9643c" ] ] }, { "id": "5569de7d.6be25", "type": "pm5500", "z": "83e35657.864148", "name": "kW_59", "unitId": "59", "data": "actPowerTot", "x": 490, "y": 340, "wires": [ [ "67a8a4f1.1077fc" ] ] }, { "id": "8559f3b9.9e765", "type": "pm5500", "z": "83e35657.864148", "name": "kWh_59", "unitId": "59", "data": "toEnergyActExp", "x": 500, "y": 400, "wires": [ [ "67a8a4f1.1077fc" ] ] }, { "id": "d99d7588.6a6738", "type": "modbus-flex-getter", "z": "83e35657.864148", "name": "NUS_SDE1_DB02_102", "showStatusActivities": true, "showErrors": true, "logIOActivities": false, "server": "73ad6b92.de9ef4", "useIOFile": false, "ioFile": "", "useIOForPayload": false, "emptyMsgOnFail": false, "keepMsgProperties": true, "x": 790, "y": 480, "wires": [ [], [ "6d71890.52b7178" ] ] }, { "id": "6d71890.52b7178", "type": "change", "z": "83e35657.864148", "name": "", "rules": [ { "t": "set", "p": "sensor_id", "pt": "msg", "to": "NUS_SDE1_DB02_102", "tot": "str" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 1030, "y": 480, "wires": [ [ "52d23431.e9643c" ] ] }, { "id": "8a0b241b.8dfdf8", "type": "pm5500", "z": "83e35657.864148", "name": "kW_57", "unitId": "57", "data": "actPowerTot", "x": 510, "y": 480, "wires": [ [ "d99d7588.6a6738" ] ] }, { "id": "6571c66.d61b538", "type": "pm5500", "z": "83e35657.864148", "name": "kWh_57", "unitId": "57", "data": "toEnergyActExp", "x": 520, "y": 540, "wires": [ [ "d99d7588.6a6738" ] ] }, { "id": "9bde742f.683678", "type": "modbus-flex-getter", "z": "83e35657.864148", "name": "NUS_SDE1_DB02_122", "showStatusActivities": true, "showErrors": true, "logIOActivities": false, "server": "c4d769e.9a70a98", "useIOFile": false, "ioFile": "", "useIOForPayload": false, "emptyMsgOnFail": false, "keepMsgProperties": true, "x": 790, "y": 640, "wires": [ [], [ "3f99b144.e1961e" ] ] }, { "id": "3f99b144.e1961e", "type": "change", "z": "83e35657.864148", "name": "", "rules": [ { "t": "set", "p": "sensor_id", "pt": "msg", "to": "NUS_SDE1_DB02_122", "tot": "str" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 1030, "y": 640, "wires": [ [ "52d23431.e9643c" ] ] }, { "id": "3be4ed40.0b1002", "type": "pm5500", "z": "83e35657.864148", "name": "kW_58", "unitId": "58", "data": "actPowerTot", "x": 510, "y": 640, "wires": [ [ "9bde742f.683678" ] ] }, { "id": "5e40bcb5.1d8534", "type": "pm5500", "z": "83e35657.864148", "name": "kWh_58", "unitId": "58", "data": "toEnergyActExp", "x": 520, "y": 700, "wires": [ [ "9bde742f.683678" ] ] }, { "id": "764f7e9e.93e7a", "type": "debug", "z": "83e35657.864148", "name": "", "active": true, "tosidebar": true, "console": false, "tostatus": false, "complete": "true", "targetType": "full", "statusVal": "", "statusType": "auto", "x": 1710, "y": 900, "wires": [] }, { "id": "a5e4f668.721c08", "type": "delay", "z": "83e35657.864148", "name": "", "pauseType": "delay", "timeout": "100", "timeoutUnits": "milliseconds", "rate": "1", "nbRateUnits": "1", "rateUnits": "second", "randomFirst": "1", "randomLast": "5", "randomUnits": "seconds", "drop": false, "x": 290, "y": 380, "wires": [ [ "af6faad8.cc81b8", "68a861af.c46b5", "5569de7d.6be25", "8a0b241b.8dfdf8", "3be4ed40.0b1002" ] ] }, { "id": "9dcf23e7.d9dc2", "type": "delay", "z": "83e35657.864148", "name": "", "pauseType": "rate", "timeout": "1", "timeoutUnits": "seconds", "rate": "1", "nbRateUnits": "1", "rateUnits": "second", "randomFirst": "1", "randomLast": "5", "randomUnits": "seconds", "drop": false, "x": 1630, "y": 460, "wires": [ [ "b2b1a852.3d5b28", "76c88d91.e5b1f4" ] ] }, { "id": "472e23c9.cd770c", "type": "catch", "z": "83e35657.864148", "name": "sensor_data_test", "scope": [ "b2b1a852.3d5b28" ], "uncaught": false, "x": 1280, "y": 800, "wires": [ [ "402ccdf1.7a1dd4" ] ] }, { "id": "402ccdf1.7a1dd4", "type": "json", "z": "83e35657.864148", "name": "", "property": "payload", "action": "", "pretty": false, "x": 1510, "y": 800, "wires": [ [ "f30169e8.140718" ] ] }, { "id": "f30169e8.140718", "type": "file", "z": "83e35657.864148", "name": "", "filename": "c:\\temp\\influx.dat", "appendNewline": true, "createDir": true, "overwriteFile": "false", "encoding": "utf8", "x": 1690, "y": 800, "wires": [ [ "764f7e9e.93e7a" ] ] }, { "id": "53dc09f7.211cd8", "type": "function", "z": "83e35657.864148", "name": "", "func": "\nvar _payload = {};\n\n \n _payload.SENSORID = msg.sensor_id;\n _payload.TAG_NAME = msg.topic ;\n _payload.TAG_VALUE = Math.abs(msg.payload);\n \nmsg.payload = _payload;\n\n\nreturn msg;\n\n", "outputs": 1, "noerr": 0, "initialize": "", "finalize": "", "x": 1460, "y": 100, "wires": [ [ "991f7b62.ff1c88" ] ] }, { "id": "2dfb7bbd.f08514", "type": "influxdb out", "z": "83e35657.864148", "influxdb": "39bb3475.b7e60c", "name": "sensor_data", "measurement": "sensor_data", "precision": "s", "retentionPolicy": "OneWeek", "database": "sensor_data_instant", "precisionV18FluxV20": "s", "retentionPolicyV18Flux": "OneWeek", "org": "", "bucket": "", "x": 1690, "y": 100, "wires": [] }, { "id": "5a80af8.18e735", "type": "change", "z": "83e35657.864148", "name": "Set Tag Names", "rules": [ { "t": "change", "p": "topic", "pt": "msg", "from": "actPowerTot", "fromt": "str", "to": "KW", "tot": "str" }, { "t": "change", "p": "topic", "pt": "msg", "from": "toEnergyActExp", "fromt": "str", "to": "KWH", "tot": "str" }, { "t": "change", "p": "topic", "pt": "msg", "from": "toEnergyActImp", "fromt": "str", "to": "KWH_RECEIVED", "tot": "str" } ], "action": "", "property": "", "from": "", "to": "", "reg": false, "x": 1280, "y": 100, "wires": [ [ "53dc09f7.211cd8" ] ] }, { "id": "991f7b62.ff1c88", "type": "delay", "z": "83e35657.864148", "name": "", "pauseType": "rate", "timeout": "1", "timeoutUnits": "seconds", "rate": "1", "nbRateUnits": "1", "rateUnits": "second", "randomFirst": "1", "randomLast": "5", "randomUnits": "seconds", "drop": false, "x": 1570, "y": 180, "wires": [ [ "2dfb7bbd.f08514" ] ] }, { "id": "fcaddb6a.9b1fa8", "type": "modbus-client", "name": "192.168.1.101:502", "clienttype": "tcp", "bufferCommands": true, "stateLogEnabled": false, "queueLogEnabled": true, "tcpHost": "192.168.1.101", "tcpPort": "502", "tcpType": "DEFAULT", "serialPort": "/dev/ttyUSB", "serialType": "RTU-BUFFERD", "serialBaudrate": "9600", "serialDatabits": "8", "serialStopbits": "1", "serialParity": "none", "serialConnectionDelay": "100", "unit_id": "60", "commandDelay": "1", "clientTimeout": "1000", "reconnectOnTimeout": true, "reconnectTimeout": "2000", "parallelUnitIdsAllowed": true }, { "id": "39bb3475.b7e60c", "type": "influxdb", "hostname": "127.0.0.1", "port": "8086", "protocol": "http", "database": "sensor_data_instant", "name": "http://localhost:8086", "usetls": false, "tls": "", "influxdbVersion": "1.x", "url": "http://localhost:8086", "rejectUnauthorized": false }, { "id": "f30cd4fd.aad048", "type": "modbus-client", "name": "192.168.1.111:502", "clienttype": "tcp", "bufferCommands": true, "stateLogEnabled": false, "queueLogEnabled": true, "tcpHost": "192.168.1.111", "tcpPort": "502", "tcpType": "DEFAULT", "serialPort": "/dev/ttyUSB", "serialType": "RTU-BUFFERD", "serialBaudrate": "9600", "serialDatabits": "8", "serialStopbits": "1", "serialParity": "none", "serialConnectionDelay": "100", "unit_id": "1", "commandDelay": "1", "clientTimeout": "1000", "reconnectOnTimeout": true, "reconnectTimeout": "2000", "parallelUnitIdsAllowed": true }, { "id": "73ad6b92.de9ef4", "type": "modbus-client", "name": "192.168.1.102:502", "clienttype": "tcp", "bufferCommands": true, "stateLogEnabled": false, "queueLogEnabled": true, "tcpHost": "192.168.1.102", "tcpPort": "502", "tcpType": "DEFAULT", "serialPort": "/dev/ttyUSB", "serialType": "RTU-BUFFERD", "serialBaudrate": "9600", "serialDatabits": "8", "serialStopbits": "1", "serialParity": "none", "serialConnectionDelay": "100", "unit_id": "1", "commandDelay": "1", "clientTimeout": "1000", "reconnectOnTimeout": true, "reconnectTimeout": "2000", "parallelUnitIdsAllowed": true }, { "id": "c4d769e.9a70a98", "type": "modbus-client", "name": "192.168.1.122:502", "clienttype": "tcp", "bufferCommands": true, "stateLogEnabled": false, "queueLogEnabled": false, "tcpHost": "192.168.1.122", "tcpPort": "502", "tcpType": "DEFAULT", "serialPort": "/dev/ttyUSB", "serialType": "RTU-BUFFERD", "serialBaudrate": "9600", "serialDatabits": "8", "serialStopbits": "1", "serialParity": "none", "serialConnectionDelay": "100", "unit_id": "1", "commandDelay": "1", "clientTimeout": "1000", "reconnectOnTimeout": true, "reconnectTimeout": "2000", "parallelUnitIdsAllowed": true } ]

mblackstock commented 3 years ago

I've tested the node with InfluxDb 1.8 and it can definitely write more than once per second. While I haven't used version 1.7 recently, I don't think this is the problem since the node's code is the same for both databases. To verify, simply create a simple flow that writes data triggered by an inject node on repeat every 0.1 seconds for example. All that said, I'd suggest upgrading to 1.8 or even better 2.x when you can.

The flow you've provided is fairly complex, with a number of nodes that I am not familiar with and can't test. My guess is that the issue you are seeing is a result of the timing issue with the flow itself, or it has something to do with the precision you've selected on the influxdb out node (seconds). My guess is that rows are overwritten if the timestamp is in the same second, but not sure.

If you need help with the flow overall, the node-red slack or forum might be a good place to go.