Telegraf errors for several hours

[jgleigh]

These errors happened for several hours and then resolved on their own. Any ideas?

2022-05-24T15:10:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:54Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:10:59Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

2022-05-24T15:11:00Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/aggregates]: Get "http://pypowerwall:8675/aggregates": EOF

2022-05-24T15:11:00Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/soe]: Get "http://pypowerwall:8675/soe": EOF

2022-05-24T15:11:00Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/freq]: Get "http://pypowerwall:8675/freq": EOF

2022-05-24T15:11:00Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/temps/pw]: Get "http://pypowerwall:8675/temps/pw": EOF

2022-05-24T15:11:00Z E! [inputs.http] Error in plugin: [url=http://pypowerwall:8675/strings]: Get "http://pypowerwall:8675/strings": EOF

[jasonacox]

Hi @jgleigh - if these happened during the start of the docker containers, it means that telegraf was up and operational (polling) before the pypowerwall container was running. Technically docker-compose should start pypowerwall before telegraf but I have seen this occur. It is generally harmless as it will go away when the pypowerwall container is fully operational.

If this is occurring randomly, look at the pypowerwall logs (docker logs pypowerwall) to see what errors are happening when this occurs. It could be getting timeouts with your Powerwall. This will occur more if you Powerwall is using WiFi instead of a hardware ethernet connection.

[jgleigh]

My Gateway is hooked to Ethernet. It only happened this one time and I've restarted Docker since then. I'll see if it happens again, but for now it was just the one time and only for a few hours. I don't think there were any errors in the pypowerwall log, just in the telegraf log. Very odd. It was during the night and early morning so I didn't notice. I didn't notice any data gaps on the Tesla app so I believe the Gateway was fully functional the whole time.

[jgleigh]

Here's what the graph looked like during the errors:

[jgleigh]

This was running t8 proxy at the time, so could be related to that old version too.

[jasonacox]

Ok, that helps. Some theories:

• There was a memory/resource leak in "t8" that could have caused issues via out of memory/swapping (fixed in "t10") which may have eventually resulted in docker restarting it.
• The error indicates telegraf is unable to make a network connection to that endpoint on pypowerwall, which could be due to some host issue (is this a Raspberry Pi?) during an automated update or other system level activity (would need to check host logs like dmesg -T).

[jgleigh]

It's running on a Mac desktop which was up at the time. I'm going to guess it was probably the memory leak since nothing else was going on with the computer during this time.

[jgleigh]

Closing for now. Will monitor for future occurence.

[BJReplay]

I've just run into this issue. Restarting Docker hasn't helped, so far. Just about to restart the host (Windows 11). Found this issue searching for Error in plugin. In my case, InfluxDB, pypowerwall and weather411 are all started, but there are errors in pypowerwall. Will open in separate issue.

[BuongiornoTexas]

@BJReplay One possible explanation is that your router has lost your powerwall or your powerwall has lost your network - my symptom is that the powerwall has no IP address in the router device list and I can't access the powerwall via https://<powerwall ip>/.

I get something similar, which I can fix by opening the app and reconnecting the Powerwall to wifi - the easiest way to do this is add a manual network with a garbage name and password - this will fail, and then it falls back to the normal wifi network and data starts to flow again. (I'm hoping a longer term fix will be an upgrade to a more modern router - if that fails, it's on to hardwiring!)

Interestingly, once pypowerwall has connected, it maintains the connection even if the router drops the powerwall from its list of known devices with IP addresses allocation.

[BJReplay]

@BuongiornoTexas thanks for the reply.

I had written up an issue, and nearly submitted, but it came good after the second host reboot, after multiple docker container and then linux instance restarts,

The Powerwall is connected to router via LAN rather than WiFi (WiFi isn't set up).

(Because the gateway and my router are in the same equipment cupboard, I went to some trouble to separate them onto separate WiFi channels to improve WiFi in my home;

• I put them both on a locally heavily congested channel,
• I then disassociated the powerwall from my home router SSID (which left the Powerwall on the congested channel),
• I then put my router on an uncongested channel.

Which reduced the interference on that channel from the gateway's TEG-NNN access point).

It is also set up with a DHCP static lease with a name assigned and, because I've got other (I'll call it "legacy") software running on this host (controlling backup reserve for my own ToU control, based on Solcast forecasting, and my own load forecasting) and that kept on trucking (including pushing six second data to a PowerBI streaming dataset for the duration), I knew it wasn't the Powerwall dropping off the air, though I thought it might have been a firmware upgrade at first.

In fact, the name resolution is that good that I can ping powerwall from my desktop, from the windows host, from WSL on the host, or from the pypowerwall container, and they all resolve, so I don't need to have have it configured with the IP, but just the name set up in the static lease.

[BuongiornoTexas]

The Powerwall is connected to router via LAN rather than WiFi (WiFi isn't set up).

Mea culpa - you are correct - my issue only applies to a wifi connection. LAN connection won't have this problem.

[BJReplay]

Mea culpa - you are correct - my issue only applies to a wifi connection. LAN connection won't have this problem.

You weren't to know - and every response helps! Something funky went on, and caused one process to lose visibility while another didn't. That visibility loss survived one host reboot, but not a second. 🤷

[jasonacox]

Puzzling. If it happens again (and I hope it doesn't), there is a feature in pypowerwall to scan your network for Powerwalls. It would be interesting to see what it can find:

 python3 -m pypowerwall scan

Additionally, I would love to see what pypowerwall is reporting (via: http://localhost:8675/vitals and http://localhost:8675/stats where localhost is the host running the pypowerwall container) and via logs:

docker logs pypowerwall

[BJReplay]

there is a feature in pypowerwall to scan your network for Powerwalls

I'll report back if it happens again (or better still, raise a new issue - I nearly had, and it resolved after a second reboot).