ETS group address import from file and/or via payload

[weilhr]

Hi Supergiovane,

many thanks for the time and effort you are spending to make this project happen. To make my life easier to import the ETS GA into knx-ultimate I wrote a quick script which finds my last storage archive on my NAS share and extract all GA into a CSV.

#!/bin/bash
umask 0000

KNXPROJ="myHome"
PROJDIR="/net/diskstation/volume1/KNX/ETS6-Projektarchiv"
KNXEXPO="/data/docker/containers/shared/ets/ets-knxproj.csv"

# Use find to get list of KNX project files as named by KNXPROJ. Use file ctime in find to identify most recent version.
# ETS6 changes the path for each restore point. From my findings ETS5 store the project archive always with the same 
# filename. However, the parsing works like in the ETS6 as shown below.
find ${PROJDIR}/*.archive/ -name "${KNXPROJ}.knxproj" -printf "%C@;%h/%f\n"  | sort -n | tail -1 | cut -d \; -f 2 | while read line; do
   echo $line
   # unzip the projectfile and export GA to file KNXEXPO
   unzip -p "$line" */0.xml \
        | grep "<GroupAddress\ " \
        | sed -e 's/^.*Address="\([^\"]\+\).*Name="\([^\"]\+\).*DatapointType="\([^\"]\+\).*$/\1;\2;\3/' \
              -e 's/\(T-[0-9]\+\)$/\1-1/' > "${KNXEXPO}"
done

At this point the individual GA in exported CSV are in binary format along with the GA name and GA DPT. Example:

...
2108;L_HG_BUE_Decke;DPST-1-1
2110;L_HG_BUE_Strahler;DPST-1-1
2114;L_HG_BUE_Schreibtisch;DPST-1-1
2115;LD_HG_BUE_Schreibtisch;DPST-3-7
...

Even though it's possible to script, I use node red to convert the RAW-GA into a "usable" format with a function in a function node. My function do some other dynamic tweaking and prepare the context in node red for other functions, so it's not bad for me personally when I do the convertion in node red.

function raw2ga (raw) {
    var gaRet=((raw & 30720) >> 11)+"/"+((raw & 1792) >> 8)+"/"+(raw & 255)
return gaRet

At this point it would be nice if it's possible to pass the GA either as JSON or CSV into KNX-Ultimate for import - depending on the format which works better for you.

Another way, almost like I do manually at the moment, export of the CSV data nto a new file / or let it generate already by the script. and store it in a directory which can be accessed by node-red, and point knx-ultimate to that csv for import. The import should either happen during startup, or with a defined payload - like { "topic": "import", "payload": "/path/to/file.csv" }

Looking forward hearing from you if you like the idea.

Thanks, Rob

[Supergiovane]

Hi Weilhr, thank you to you for enjoying my node.

At this point it would be nice if it's possible to pass the GA either as JSON or CSV into KNX-Ultimate for import - depending on the format which works better for you.

If i've correctly understood, you need to pass the CSV at runtime, to KNX-Ultimate, using, for example a property like this, to be passed to KNX-Ultimate:

msg.setGAList = '.............'

Is that correct? In this case, you should use one of the two allowed ETS formats (ESF or CSV). You can look the video at the bottom of this page, export both formats from ETS (ESF and CSV) and see which fit your need.

[weilhr]

Hi Supergiovane,

If i've correctly understood, you need to pass the CSV at runtime, to KNX-Ultimate, using, for example a property like this, to be passed to KNX-Ultimate:

msg.setGAList = '.............'

Is that correct?

Yes, that's correct. Goal is a process which can be automated instead of running a manual export from ETS as shown in the video you are referring. With the process I have described it's possible to extract the raw (binary) GAs from a ETS projectarchive. From the extracted data it's possible to calculate the GA from the raw/binary GA with the formula as mentioned. This scriptable way to generate the CSV is an alternate way to export the GA as required for KNX-Ultimate. In a next step it would be nice to get a way to load the CSV automated/triggered from a file or by passing a payload as you mentioned (msg.setGAList = '...').

Thanks, Rob

[Supergiovane]

Ok Rob, i will start to work on that runtime import of CSV. Your script should save the exact ESF or CSV format file somewhere, like ETS export does. Then you can simply use a "file in" node to read that file and to feed KNX-Ultimate. I'll let you know here when it's done and when the new KNX-Ultimate version will be published.

[Supergiovane]

Hi Rob, in the version 1.4.1 you can now import the CSV at runtime, by using the Watchdog Node and pass this to it:

// IP: IP of your KNX/IP Router or Interface
// Port: Port of your KNX/IP Router or Interface
// PhysicalAddress: Physical address your KNX/IP Router or Interface (this is not a Group Address, this is a physical address indicating the physical device in your KNX installation)
// BindToEthernetInterface: "Auto" (for automatic detection) or the ethernet interface name, for example "en0".
// Protocol: "TunnelUDP" or "TunnelTCP" or "Multicast"
// importCSV: the ETS exported CSV or ESF. Please see the text format in the Gateway Config Wiki Page and in the youtube video.
// All these parameters are optional

msg.setGatewayConfig={ importCSV:`"Group name"  "Address"   "Central"   "Unfiltered"    "Description"   "DatapointType" "Security"
"Attuatori luci"    "0/-/-" ""  ""  ""  ""  "Auto"
"Luci primo piano"  "0/0/-" ""  ""  ""  ""  "Auto"
"Luce camera da letto"  "0/0/1" ""  ""  ""  "DPST-1-1"  "Auto"};`
return msg;

The sample above is based on the exported CSV file from ETS. Your exporter script should exactly write the file like ETS does (with " between field names and tabulators, etc..).