Mode: Scraping - cant handle non numeric values (e.g. On / Off)

[svenoone]

When attempting to start the Docker container with the ISG_URL, ISG_PASSWORD, and ISG_USER variables, the container stops after a few seconds with the following log message:

panic: runtime error: index out of range [1] with length 0 goroutine 22 [running]: main.normalizeValue({0xc000704c99, 0x3}) /build/scraping.go:150 +0x2c6 main.parsePage.func1(0xc0000120f8?, 0x0?) /build/scraping.go:110 +0xfb github.com/PuerkitoBio/goquery.(*Selection).Each(0xc0003af890, 0xc00060fd08) /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.8.1/iteration.go:10 +0x46 main.parsePage({0xba6637?, 0x0?}, 0xc00060fd70) /build/scraping.go:95 +0x11b main.gatherScrapingData() /build/scraping.go:73 +0x18e main.gatherData() /build/main.go:167 +0x108 main.main.func1() /build/main.go:88 +0x19 created by main.main /build/main.go:86 +0x10b

The ISG is set to English, and the ISG version is v12.2.1.

[svenoone]

I have fiddled around - I never did something in GO but it seems i found the point of failure.

The value "RÜCKLAUFISTTEMPERATUR WP | 34,0°C" seems to be handeled correct then comes the value "FESTWERTBETRIEB | Aus" and "Aus" seem not beeing handeled by the RegEx at Line 145 in scrapping.go

Hope this is a better detailed description :)

[svenoone]

It seems I was at the right road. I just simply overwrite "Aus" and "Ein" values to 0 and 1 with an statement and the scrapper works - wohoo!

[sfudeus]

Hey, good that you found a solution. Didn't you mention setting the language to English? Which kind of System do you have? I don't have "FESTWERTBETRIEB" with my LWZ. Did you happen to test the modbus integration as well? You wouldn't see those user interface parsing problems there.

[svenoone]

Yes - initially the interface was EN - I tried out many thing :P - I preffer to use Modbus but there I have absolutely no knowledge :(

I can tell you that also does not work with my heating unit. Its a "WPE-I 12H 230 Premium"

I think you have a Air/Water Heat pump since i red some where something about a fan, which mine does not have. Mine is a Sole/Water with geothermal probe.

About Modbus I can tell you it stops also with a exception: time="2023-03-18T00:21:24+01:00" level=fatal msg="modbus: exception '2' (illegal data address), function '132'"

It seems one or more address is not readable - I tried to read some addresses, unfornunately I got at ech request such a error in the response.

[sfudeus]

Sorry for the late feedback, I'm currently providing a fix for this using some of your pull request, thanks for that. To enhance the test suite, could you provide a dump of one of the pages of the web interface which do have this problem? With that I could better test the result. Just a html dump, as the examples in the test_resources directory.

[svenoone]

@sfudeus I will upload the HTML dump of the affected page. I was sure that another page was also affected. Off the top of my head I don't know which one it was. I'll look into it a bit more in the next few days and give you some feedback.

STIEBEL ELTRON Reglersteuerung_files.zip

[sfudeus]

Thx, added the page already to the testcases.

[svenoone]

@sfudeus

First run prompts directly an error and let crash the container.

panic: runtime error: index out of range [1] with length 0

goroutine 13 [running]:
main.normalizeValue({0xc0002fa759, 0x3})
        /build/scraping.go:151 +0x25d
main.parsePage.func1(0x0?, 0xc0005cd710)
        /build/scraping.go:110 +0xe5
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc0005cca80, 0xc0004d9d00)
        /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.9.1/iteration.go:10 +0x62
main.parsePage({0xc44709?, 0x0?}, 0xc0004d9ec8)
        /build/scraping.go:95 +0x114
main.gatherScrapingData()
        /build/scraping.go:73 +0x19b
main.gatherData()
        /build/main.go:178 +0x10a
main.main.func1()
        /build/main.go:96 +0x13
created by main.main in goroutine 1
        /build/main.go:94 +0x111
panic: runtime error: index out of range [1] with length 0

I just switched to your new tag 1.4.2 and runs my docker-compose.yml. It was just a quick try out. I try to check later more details what's failing.

[sfudeus]

1.4.2 doesn't contain the fix, either use image docker.io/sfudeus/isg_exporter:preview-92 or the source from branch fix_onoff_parsing

[svenoone]

That was a typical brain AFK mistake.

Now I am on the right branch. I can tell you for German is working. But something else is not working when the language is English.

I noticed, the old code from April 2023 is crashing as well on the same point, so I assume it has nothing to do with the changes.

panic: descriptor Desc{fqName: "isg_set_superheating,_suction_gas,_compresso", help: "Metric set_superheating,_suction_gas,_compresso in K", constLabels: {}, variableLabels: {unit}} is invalid: "isg_set_superheating,_suction_gas,_compresso" is not a valid metric name

goroutine 51 [running]:
github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x1292c20, {0xc0002882c0?, 0x0?, 0x0?})
        /go/pkg/mod/github.com/prometheus/client_golang@v1.19.0/prometheus/registry.go:405 +0x66
github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGaugeVec({{0xd79898?, 0x1292c20?}}, {{0xc446ee, 0x3}, {0x0, 0x0}, {0xc0005311d0, 0x28}, {0xc000437e00, 0x34}, ...}, ...)
        /go/pkg/mod/github.com/prometheus/client_golang@v1.19.0/prometheus/promauto/auto.go:308 +0x163
github.com/prometheus/client_golang/prometheus/promauto.NewGaugeVec(...)
        /go/pkg/mod/github.com/prometheus/client_golang@v1.19.0/prometheus/promauto/auto.go:198
main.createOrRetrieve({0xc0005311d0, 0x28}, {0xc000402407, 0x1}, 0x0)
        /build/main.go:208 +0x610
main.parsePage.func1(0x0?, 0xc000566330)
        /build/scraping.go:118 +0x24b
github.com/PuerkitoBio/goquery.(*Selection).Each(0xc000556390, 0xc0000add00)
        /go/pkg/mod/github.com/!puerkito!bio/goquery@v1.9.1/iteration.go:10 +0x62
main.parsePage({0xc44718?, 0x0?}, 0xc0000adec8)
        /build/scraping.go:96 +0x114
main.gatherScrapingData()
        /build/scraping.go:79 +0x218
main.gatherData()
        /build/main.go:178 +0x10a
main.main.func1()
        /build/main.go:96 +0x13
created by main.main in goroutine 1
        /build/main.go:94 +0x111

[sfudeus]

Oh, yeah. It tries to generate a prometheus metrics names isg_set_superheating,_suction_gas,_compresso, which is not an accepted name because of the commas. This is an unrelated bug. Can you try to find out on which page in the ISG this happens and provide an html dump of the page? With that I could fix this.

But the on/off is solved for you now? Then I could merge this already.

[svenoone]

On/Off is fixed!

Sure, I'll open another issue for the "isg_set_superheating,_suction_gas,_compresso" issue

[sfudeus]

thx for confirming, merging this one then now.