Open RobBie1221 opened 4 months ago
arjenhiemstra
commented
4 months ago Thanks for you this issue. I see that there is indeed a new version in the servicetool.
How would I be able to determine if they added / changed some settings / status registers? Do I need to enable sniffing?
Unfortunately there is no easy way to get these labels out of the tool, for starters because all labels are in Dutch... I'll work on it; maybe @tomkooij has an idea how to create this file in an easier way?
RobBie1221
commented
4 months ago Dutch is not an issue ;-) Do you need an account for the service tool to be able to see the labels?
Theoretically, if I would build a new version for myself to test where I copy the status registers from version 37, could that work? Then at least I can have monitoring again.
arjenhiemstra
commented
4 months ago Ok, the status label are easy, not a lot of changes, commit https://github.com/arjenhiemstra/ithowifi/commit/aa65d7bf8259f7d7a1e6d1f7f024ba8f8bdd4f69
settings labels need a bit more work
arjenhiemstra
commented
4 months ago Dutch is not an issue ;-) Do you need an account for the service tool to be able to see the labels?
Theoretically, if I would build a new version for myself to test where I copy the status registers from version 37, could that work? Then at least I can have monitoring again.
if you pull this repo and build it you should have status labels the service tool is free and downloadable from the itho website
arjenhiemstra
commented
4 months ago I theory the settings should be an easy fix as well. There is one change on an existing index (228) and there are 10 additions. But unfortunately I have some issues with the script to generate the .h header files, I' getting a totally different result than currently is in de firmware...
RobBie1221
commented
4 months ago Thanks for the fast response!
Took me some time to build, commit https://github.com/arjenhiemstra/ithowifi/commit/aa65d7bf8259f7d7a1e6d1f7f024ba8f8bdd4f69 doesn't actually build with me (gives 2 errors regarding type conversion).
I was able to build the change of this commit on top of 2.8.0. I now see status data again!
arjenhiemstra
commented
4 months ago Thanks for the fast response!
Took me some time to build, commit aa65d7b doesn't actually build with me (gives 2 errors regarding type conversion).
I was able to build the change of this commit on top of 2.8.0. I now see status data again!
Hmm, your correct. I accidentally checked in two files extra... trying to do too many things at once...
I'll fix that
here is a reference to the script in use to generate the files https://github.com/arjenhiemstra/ithowifi/issues/49#issuecomment-971422213
But somehow its not working anymore, need to check this one into version control as well cleary
RobBie1221
commented
4 months ago Thanks for the hint towards the script. I just manually opened the par file from the tool, still trying to understand how it's all built together.
For now at least my monitoring is running again. I'll see if I can get the script working to also get parameters in for version 41.
arjenhiemstra
commented
4 months ago Here is some useful info about the par files and how to download them: https://github.com/pommi/python-itho-wpu
and here is the "latest" version of the script. I'm far from an PHP expert and as said, should have better kept track of things using vc. If you can help out, that would be great!
<?php
$lang = 'NL';
//$lang = 'GB';
//$lang = 'other';
$product = 'WPU'; //(cve14, cve1B, demandflow, autotemp, hrueco, hru200, hru250-300, hru350, WPU)
$dbname = 'wpu.db';
$pwlevel = 500;
//use versionlist instead of DB query
//cve14
//$versions = [1,2,3,4,5,6];
//cve1B
//$versions = [6,7,8,9,10,11,17,18,20,21,22,24,25,26,27];
//demandflow
//$versions = [1,3,4,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21];
//autotemp
//$versions = [1,2,3,4,5,6,7,8,9,10,12,13];
//hrueco
//$versions = [3,4,6,7,8,10,11,12];
//hru200
//$versions = [12,13,14,15,17,18,19,20,21,25,26,27];
//hru250-300, $lang = 'GB';
//$product = 'HRU250_300';
//$versions = [1,2,3,4,6,7,8,9,10];
//hru350
//$versions = [1,3,4];
//wpu & set $verlist to TRUE
$versions = [];
$verlist = TRUE;
$db = new SQLite3($dbname);
$dl = [];
if ($verlist == TRUE) {
$q = 'select DataLabel from VersieBeheer';
$res = $db->query($q);
// All datalabel numbers in an array
while ($row = $res->fetchArray()) {
$dl[$row['DataLabel']] = [];
}
}
else {
foreach ($versions as $value) {
$dl[$value] = [];
}
}
if($lang == 'NL') {
$col_unit = 'Eenheid_NL';
$col_title = 'Tekst_NL';
$col_name = 'Naam';
}
else if($lang == 'GB') {
$col_unit = 'Eenheid_GB';
$col_title = 'Tooltip_GB';
$col_name = 'Naam';
}
else {
$col_unit = 'unit';
$col_title = 'title';
$col_name = 'name';
}
// Collect names per datalabel and the flash label text
foreach($dl as $k=>$v) {
$q = "select * from Datalabel_V$k ORDER BY `Index` ASC";
$res = $db->query($q);
while($row = $res->fetchArray()) {
// if (empty($row[$col_title])) {
// $col_title_save = $col_title;
// $col_title = 'Naam';
// }
echo print_r($row);
$fullname = $row[$col_title];
if (!empty($row[$col_unit])) {
$row[$col_unit] = preg_replace('/[\s]+/','',$row[$col_unit]);
//Strip off spaces and non-alpha-numeric
$fullname .= " (".$row[$col_unit].")";
}
if (!empty($row[$col_unit])) {
$unit = "";
if (strcmp($row[$col_unit], "%") == 0) $unit = 'perc';
else if (strcmp($row[$col_unit], "% ") == 0) $unit = 'perc';
else if (strcmp($row[$col_unit], "°c") == 0) $unit = 'degc';
else {
$unit = $row[$col_unit];
}
$unit = preg_replace('/[\W]+/','',$unit);
$row[$col_title] .= "_".$unit;
$row[$col_title] = preg_replace('/[[:space:]]+/', '-', $row[$col_title] ?? '');
}
//$dl[$k][$row[$col_name]] = strtolower($row[$col_title]);
$dl[$k][$row[$col_name]] = array(ucfirst($fullname ?? ''), strtolower($row[$col_title] ?? ''));
// $col_title = $col_title_save;
//echo print_r($dl[$k]);
echo "\n";
}
}
// Build array of labels with their associated datalabel number
$labels = [];
foreach($dl as $k=>$v) {
foreach($v as $label=>$id) {
if (! isset($labels[$label])) {
$labels[$label] = [];
}
$labels[$label][$k] = $id;
}
//p rint_r($labels);
echo "\n";
}
$nr = 0;
$f = [];
// Store label sequences
$labelsequence = [];
foreach($labels as $label => $version) {
$keys = array_keys($version);
$labelsequence[$label][$nr] = $keys;
// Flash message position
$f[$nr] = $version[$keys[array_key_last($keys)]];
//p rint array_key_last($keys)."\n";
//p rint_r($keys)."\n";
$nr++;
}
print("\n");
// Display the relevant labels numbers per version
foreach($dl as $labelnr=>$nameArr)
{
$fieldseq = [];
foreach($nameArr as $name=>$id) {
$n = $labelsequence[$name];
$fieldseq[] = array_keys($n)[0];
}
printf("const uint8_t itho_%sstatus%d[] { %s,255};\n",$product,$labelnr,join(",",$fieldseq));
}
print("\n");
print("const struct ithoLabels itho{$product}StatusLabels[] {\n");
//print("const __FlashStringHelper *itho{$product}StatusLabels[] = {\n");
// Flash messages
foreach($f as $pos => $flashlabel) {
$flashlabel[1] = preg_replace('/[[:space:]]+/', '-', $flashlabel[1] ?? '');
$flashlabel[1] = preg_replace('/^[\-]+/','',$flashlabel[1] ?? '');
// Strip off the starting hyphens
$flashlabel[1] = preg_replace('/[\-]+$/','',$flashlabel[1] ?? '');
// // Strip off the ending hyphens
//$ flashlabel = preg_replace('/[\s\W]+/','-',$flashlabel);
// Strip off spaces and non-alpha-numeric
printf(" { \"%s\", \"%s\" },\n",$flashlabel[0],$flashlabel[1]);
}
print("};\n");
$dl = [];
if ($verlist == TRUE) {
$q = 'select ParameterLijst from VersieBeheer';
$res = $db->query($q);
// All datalabel numbers in an array
while ($row = $res->fetchArray()) {
$dl[$row['ParameterLijst']] = [];
}
}
else {
foreach ($versions as $value) {
$dl[$value] = [];
}
}
if($lang == 'NL') {
$col_unit = 'Eenheid_NL';
$col_title = 'Tekst_NL';
$col_name = 'Naam';
if($product == 'DemandFlow') {
$col_title = 'Naam_fabriek';
}
if($product == 'WPU') {
$col_name = 'Tekst_NL';
}
}
else if($lang == 'GB') {
$col_unit = 'Eenheid_GB';
$col_title = 'Tekst_GB';
$col_name = 'Naam';
if($product == 'WPU') {
$col_name = 'Tekst_NL';
}
}
else {
$col_unit = 'unit';
$col_title = 'title';
$col_name = 'name';
}
// Collect names per settingslabel and the flash label text
foreach($dl as $k=>$v) {
$q = "select * from Parameterlijst_V$k where Paswoordnivo<$pwlevel ORDER BY `Index` ASC";
$res = $db->query($q);
while($row = $res->fetchArray()) {
if (empty($row[$col_title])) {
if($product == 'AutoTemp') {
$col_title = 'Naam';
}
if($product == 'WPU') {
$col_title = 'Naam_fabriek';
}
}
if(strcmp($row[$col_title] ?? '', "Minimum ventilation speed during Auto Night mode") == 0 || strcmp($row[$col_title] ?? '', "Minimum ventilation speed during Auto mode") == 0) {
$col_title = 'Naam';
}
if (!empty($row[$col_unit])) {
$row[$col_unit] = preg_replace('/[\s]+/','',$row[$col_unit]);
//Strip off spaces and non-alpha-numeric
$row[$col_title] .= " (".$row[$col_unit].")";
}
$dl[$k][$row[$col_name]] = ucfirst($row[$col_title] ?? '');
if($product == 'AutoTemp') { //check if correct
$col_title = 'Tekst_GB';
}
if($product == 'WPU') { //check if correct
$col_title = 'Tekst_GB';
}
}
}
// Build array of labels with their associated datalabel number
$labels = [];
foreach($dl as $k=>$v) {
foreach($v as $label=>$id) {
if (! isset($labels[$label])) {
$labels[$label] = [];
}
$labels[$label][$k] = $id;
}
}
$nr = 0;
$f = [];
// Store label sequences
$labelsequence = [];
foreach($labels as $label => $version) {
$keys = array_keys($version);
$labelsequence[$label][$nr] = $keys;
// Flash message position
$f[$nr] = $version[$keys[array_key_last($keys)]];
$nr++;
}
print("\n");
// Display the relevant labels numbers per version
foreach($dl as $labelnr=>$nameArr)
{
$fieldseq = [];
foreach($nameArr as $name=>$id) {
$n = $labelsequence[$name];
$fieldseq[] = array_keys($n)[0];
}
printf("const uint16_t itho_%ssetting%d[] { %s,999};\n",$product,$labelnr,join(",",$fieldseq));
}
print("\n");
print("const char* itho{$product}SettingsLabels[] = {\n");
// Flash messages
foreach($f as $pos => $flashlabel) {
printf(" \"%s\",\n",$flashlabel);
}
print("};\n");
Looking at my own earlier reply in the mentioned issue:
Hmm, in the DB apparently not all data info is sorted the same way...
It probably has something to do with the data alignment (rows in the DB do not appear to be stored in index order), the script does not correct for that
tomkooij
commented
4 months ago I have a tool that needs cleaning up that does just this...
Shall I dig into this and add the setting labels for v41, or is this already fixed by @RobBie1221 ?
RobBie1221
commented
4 months ago I didn't fix anything yet, I'm still in a learning curve ;)
If you can do this that would be perfect.
tomkooij
commented
4 months ago Unfortunately my (spaghetti) code just reads all the StatusLabels and exports a list. However, the order does not seem to match wpu.h. So it has the same issue that is described by @arjenhiemstra above :-(
I guess last time I just manually edited/added some stuff. (Just like I manually editted the double 'utc-time' key).
I'll try to write the records to sqllite and try with the PHP tool above.
RobBie1221
commented
4 months ago If I look in the database, parameterlijst_v41 is inserted in the database between v4 and v5, which probably means scripts parse v41 before v5, v6 etc. This is I think also what @arjenhiemstra means by correction for ordering?
If this is the case, I wonder how parsing has been done with scripts before, because e.g. v37 is also inserted before v4...
I can make a script which parses from v1 to v41 (first to last), but that would also maybe change wpu.h quite a bit?
RobBie1221
commented
4 months ago I'm debuggingconvert-itho-db.py script, but something goes seriously wrong with this script. From Parameterlijst_v17 onwards, it is not able to parse the Index. Some get a strange number, most get nan
tomkooij
commented
4 months ago @RobBie1221 I just read the tables from the Access database (parameter file). Using code copied from convert-itho-db.py.
I have two problems which prevent me from auto-generating much of wpu.h:
wpu.h. I cannot recreate them from the parameter file. wpu.h. Many of the settinglabels are in the list twice. https://github.com/arjenhiemstra/ithowifi/blob/master/software/NRG_itho_wifi/main/devices/wpu.h#L28The StatusLabels seems to autogen just fine (except from different descriptions, but the order is an exact match)
I'll upload my python notebook to GitHub so you can have a look: https://github.com/tomkooij/itho_parameters/blob/master/parse_parameters.ipynb
(This is a "work in progress" notebook, I have not cleaned it up)
RobBie1221
commented
4 months ago I also noticed StatusLabels are different in sense of descriptions, this probably has to do with different descriptions. Some StatusLabels have among versions the same Naam but different Tekst_NL and Tekst_GB. It seems to depend on order of parsing which label ends up in the header.
I'm trying to wrap my head around this: https://github.com/pommi/python-itho-wpu/blob/7048014a223431d9024f0c5fca820f5ad146e19c/convert-itho-db.py#L55-L58
That should simply fetch the Index (among other keys) but somehow for Parameterlijst 17 and up this does not result in indices. Can't figure out what is wrong here.
arjenhiemstra
commented
4 months ago If I look in the database,
parameterlijst_v41is inserted in the database between v4 and v5, which probably means scripts parse v41 before v5, v6 etc. This is I think also what @arjenhiemstra means by correction for ordering?If this is the case, I wonder how parsing has been done with scripts before, because e.g. v37 is also inserted before v4...
I can make a script which parses from v1 to v41 (first to last), but that would also maybe change
wpu.hquite a bit?
Also if I order the versions ASC the result is the same, also with old db files unfortunately
arjenhiemstra
commented
4 months ago @RobBie1221 I just read the tables from the Access database (parameter file). Using code copied from
convert-itho-db.py.I have two problems which prevent me from auto-generating much of
wpu.h:
- The english descriptions from the .mdb are quite different from
wpu.h. I cannot recreate them from the parameter file.- The settings are really different between v27 and v31 in
wpu.h. Many of the settinglabels are in the list twice. https://github.com/arjenhiemstra/ithowifi/blob/master/software/NRG_itho_wifi/main/devices/wpu.h#L28The StatusLabels seems to autogen just fine (except from different descriptions, but the order is an exact match)
I'll upload my python notebook to GitHub so you can have a look: https://github.com/tomkooij/itho_parameters/blob/master/parse_parameters.ipynb
(This is a "work in progress" notebook, I have not cleaned it up)
I think I manually translated the labels because no english version was available. Some other devices have english labels and some luckily don't get updated as often :)
arjenhiemstra
commented
4 months ago I also noticed StatusLabels are different in sense of descriptions, this probably has to do with different descriptions. Some StatusLabels have among versions the same
Naambut differentTekst_NLandTekst_GB. It seems to depend on order of parsing which label ends up in the header.
Thats correct, the goal is to store labels with as little memory as possible. User a certain labels (Naam, Tekst_NL etc.) as keys. The keys need to be unique in the sense that they generate enough unique labels for all settings across all firmware versions to be covered. This sometimes means a value can be repeated, keys were different in that case.
Unfortunately, the db files of the service tool are a just as much spaghetti as our code :) It differs between devices what the best to use as keys, that is why there are all kind of exceptions in my adapted script.
What would be great is if we could generate the .h file and with that output check agains the db if all labels line up with device fw version x and index no y etc.
RobBie1221
commented
4 months ago Just checking here, I'm now manually reading and trying to interpret the v37 parameter list. Via Access I can see the following:
Here:
It seems that index 228 refers to position 409 from ithoWPUSettingsLabels.
Position 409 should be:
Right? Or am I seeing this wrong? Seems this should point to 9?
arjenhiemstra
commented
4 months ago Just checking here, I'm now manually reading and trying to interpret the v37 parameter list. Via Access I can see the following:
Here:
It seems that index 238 refers to position 409 from ithoWPUSettingsLabels.
Position 409 should be:
Right? Or am I seeing this wrong? Seems this should point to 9?
You should read it as follows: setting in database for version 37 column index value: 228 then itho_WPUsetting37[228] = 409
ithoWPUSettingsLabels[409] = "Log interval (sec)"
which indeed is in the db:
| 228.0 | 228.0 | P_Data_Interval | dataLogInterval | 5.0 | 0x10 | 300.0 | 5.0 | loginterval tijd | tijd tussen 2 metingen |
|---|
First of all, kudos for the very nice prints and code, I use them on my WPU as well as all my AutoTemp units!
Today, they exchanged my heatpump. The print used to work perfectly, but they bumped me to hw version 83, fw version 41. It seems from wpu.h that up to fw version 37 is currently supported.
Obviously, I'd like to get support in for fw version 41. I can add some effort, but need some guidance. It seems I need to extend wpu.h? Could I test this by locally running build_script.py from Python to generate a binary? How would I be able to determine if they added / changed some settings / status registers? Do I need to enable sniffing?