Connectivity error while in session with PicoStill

[marcchabot]

I am unable to successfully complete a distillation because every 5-10 minutes or so, Z will give me a "Connectivity error" and alarm go on. I manually have to select Reconnect. It will reconnect but 5-10 minutes later same thing again.

Any fix for this?

Thank you.

[wiltdavi]

I saw a similar behavior but it ran hours without an issue in my case so I just reconnected and continued. I did not get any logs so I did not open an issue for it. If you can reproduce it so quickly can you provided logs of the issue?

What is your setup? I was using a RPi 3 Model B with a Wi-Fi connection to the Z.

[tmack8001]

For some reason the Z has more wifi and/or network stability issues than any other device (surprising to me). If you have an ethernet port (or you can get a $10-15 USB<->ethernet adapter) I'd encourage anyone to go that route until this can be determined what the underlying issues are.

[marcchabot]

Hello,

So connect directly to the rpi4 via ethernet? Of course I could do that.

Right now I'm using rpi4 with wifi to PICOBREW.

Here is my log file:

● rc-local.service - /etc/rc.local Compatibility Loaded: loaded (/lib/systemd/system/rc-local.service; enabled-runtime; vendor preset: enabled) Drop-In: /usr/lib/systemd/system/rc-local.service.d └─debian.conf /etc/systemd/system/rc-local.service.d └─ttyoutput.conf Active: active (running) since Tue 2021-01-12 11:32:32 EST; 23min ago Docs: man:systemd-rc-local-generator(8) Process: 355 ExecStart=/etc/rc.local start (code=exited, status=0/SUCCESS) Tasks: 6 (limit: 2063) CGroup: /system.slice/rc-local.service ├─552 python3 server.py 0.0.0.0 8080 ├─610 /bin/sh -c systemctl status rc.local -n 20000 └─611 systemctl status rc.local -n 20000 Jan 12 11:32:18 picoserver systemd[1]: Starting /etc/rc.local Compatibility... Jan 12 11:32:32 picoserver systemd[1]: Started /etc/rc.local Compatibility. Jan 12 11:32:35 picoserver rc.local[355]: WebSocket transport not available. Install eventlet or gevent and gevent-websocket for improved performance. Jan 12 11:32:35 picoserver rc.local[355]: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit) Jan 12 11:46:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:46:46] "PUT /Vendors/input.cshtml?type=ZState&token=30aea4c73494 HTTP/1.0" 200 - Jan 12 11:47:30 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:47:30] "POST /Vendors/input.cshtml?type=StillRequest&token=30aea4c73494 HTTP/1.0" 200 - Jan 12 11:47:34 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:47:34] "POST /Vendors/input.cshtml?type=StillRequest&token=30aea4c73494 HTTP/1.0" 200 - Jan 12 11:47:50 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:47:50] "GET /API/PicoStill/getFirmwareAddress?uid=30aea41c5640&version=0.0.30 HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET / HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/css/bootstrap/bootstrap.min.css HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/css/fontawesome/fontawesome.min.css HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/css/fontawesome/all.min.css HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/css/tabulator/tabulator_midnight.min.css HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/css/main.css HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/jquery/jquery.min.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/bootstrap/bootstrap.min.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/tabulator/tabulator.min.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/highcharts/annotations.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/highcharts/highcharts.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/highcharts/export-data.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/highcharts/dark-unica.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/highcharts/exporting.js HTTP/1.0" 200 - Jan 12 11:55:46 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:46] "GET /static/js/socketio/socket.io.slim.js HTTP/1.0" 200 - Jan 12 11:55:47 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:47] "GET /static/js/index.js HTTP/1.0" 200 - Jan 12 11:55:47 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:47] "GET /static/js/brew_graph_socketio.js HTTP/1.0" 200 - Jan 12 11:55:47 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:47] "GET / socket.io/?EIO=3&transport=polling&t=NRtXbn0 HTTP/1.1" 200 - Jan 12 11:55:47 picoserver rc.local[355]: 127.0.0.1 - - [12/Jan/2021 11:55:47] "GET /static/css/webfonts/fa-solid-900.woff2 HTTP/1.0" 200 -

Le mar. 12 janv. 2021 11 h 54, Trevor Mack notifications@github.com a écrit :

For some reason the Z has more wifi and/or network stability issues than any other device (surprising to me). If you have an ethernet port (or you can get a $10-15 USB<->ethernet adapter) I'd encourage anyone to go that route until this can be determined what the underlying issues are.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/chiefwigms/picobrew_pico/issues/190#issuecomment-758791463, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDGHOLHEYQTOURGVZLN2HLSZR5ERANCNFSM4V7MDBZQ .

[wiltdavi]

@tmack8001 I tried to use the Ethernet connect to the Z but I was not able to get the PicoStill to start a session properly in this configuration. I was going to look into the logs but I have not done that yet. Should it work for the Z to connect to the PicoStill over Wi-Fi while the Z is connected to the RPi over Ethernet?

[tmack8001]

Not sure I have both connected over WiFi here with no issues so far 🤞 just trying to reproduce what others have mentioned.

We bridge the two interfaces (ap0 and eth0) explicitly as the ascii diagram on my blog post showed so they are still on the same network. The Z doesn't connect "WiFi" to the Still as then it would need to disconnect from out wireless network, but rather communicates to it via the network they both are attached to by IP address I'm fairly certain.

[tmack8001]

@marcchabot if you could restart and update the server (no need to power down use the option in system in the navbar).

I recently added a better logs experience. Use that to then post the nginx error log to this ticket would be helpful.

[marcchabot]

Here it is.

Thanks.

For your information I connected to ethernet but still having the issue. I'd say it doesn't happen as much but still will disconnect every 20 mins approx.

Marc

Le mar. 12 janv. 2021 12 h 15, Trevor Mack notifications@github.com a écrit :

@marcchabot https://github.com/marcchabot if you could restart and update the server (no need to power down use the option in system in the navbar).

I recently added a better logs experience. Use that to then post the nginx error log to this ticket would be helpful.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chiefwigms/picobrew_pico/issues/190#issuecomment-758808026, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDGHOPVRJFWDX3GWXS5YHLSZR7SFANCNFSM4V7MDBZQ .

[tmack8001]

@marcchabot don't think GitHub attached files from an email properly all the time. Best to do that via the website instead

[marcchabot]

Oh sorry about that. Did you receive error log file this time?

Marc

Le mar. 12 janv. 2021 13 h 04, Trevor Mack notifications@github.com a écrit :

@marcchabot https://github.com/marcchabot don't think GitHub attached files from an email properly all the time. Best to do that via the website instead

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chiefwigms/picobrew_pico/issues/190#issuecomment-758837511, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDGHOPPDR5GZV447JUTN6DSZSFK7ANCNFSM4V7MDBZQ .

nginx.error.log

[tmack8001]

No @marcchabot I haven't gotten those, you can email them over to me as well if that is easier for you.

[tmack8001]

or wait yea actually I see it now there is a link to the file in the latest commit I missed 🤦‍♂️. Or at least only the nginx.error.log

[tmack8001]

As I suspected there are some communication failures between 2 pieces of technology within the server:

2021/01/12 09:25:08 [error] 411#411: *1 connect() failed (111: Connection refused) while connecting to upstream, 
client: 192.168.72.115, server: www.picobrew.com, request: "PUT /Vendors/input.cshtml?type=ZState&token=<token> HTTP/1.1", 
upstream: "http://127.0.0.1:8080/Vendors/input.cshtml?type=ZState&token=<token>", host: "www.picobrew.com"

However when I see this it only occurs right at server startup if I catch one process not fully started yet (or after doing a server update and restart task). Can you confirm if this is an ongoing problem even after the website of the raspberrypi server stabilizes after startup? If you can view the index page http://raspberrypi/ I classify that as stabilized.

I see you say this happens now every 20 minutes do you restart any electronics when this happens (the Z, the raspberrypi, or the PicoStill)?

[marcchabot]

This is an ongoing problem. Rightr now, it's beeping every 5 mins or so. I gave plenty of time to the server to boot. I don't have to reatart anything, just select "try to reconnect" but 5 mins (10 if I'm lucky) I'll have that same error agasin. Here is todays' error log.

Thanks nginx.error.log

[marcchabot]

Could it be that Z is switching from ETH to Wifi and then giving me Connectivity error? Couldtry to disable Z wifi (by giving wrong password to PICOBREW AP so it stays on ethernet and not wifi?

[tmack8001]

@marcchabot I had another theory about what was going on, but unfortunately that error log shows that isn't the case. I had a suspicion that nginx (the http and https proxy in front of our server) is switching between ipv4 and ipv6 internally for talking to the python+flask service port, however all the upstream timeouts in that error log weren't from the Z unit itself and instead from your browser is which not great, but ok.

Could you also upload the flask+server logs? I want to put to rest any assumption that there was an error in the server vs having unstable connection to the server.

When you run non PicoStill sessions do you have issues (drain, rinse, brew, etc)? Should see a fairly frequent data log show up on the brew graphs with no visible gap of data being reported. Data is reported every minute or so should be obvious if there is any loss of connectivity.

When ethernet is enabled on the Z I'm fairly certain it takes user action to switch back to WiFi via the settings menu of the Z.

[marcchabot]

Here is the other log you need.

As I'm just starting using your Pico server, I don't know if Z behavior is different with non-Still tasks. Will do some more tests and report back to you. picobrew_pico.log

[tmack8001]

Ok in this I also don't see any reported errors. However, that doesn't necessarily mean anything either.

Jan 14 13:42:14 raspberrypi rc.local[386]: 127.0.0.1 - - [14/Jan/2021 13:42:14] "PUT /Vendors/input.cshtml?type=ZState&token=<z-uid> HTTP/1.0" 200 -
Jan 14 13:43:37 raspberrypi rc.local[386]: 127.0.0.1 - - [14/Jan/2021 13:43:37] "PUT /Vendors/input.cshtml?type=ZState&token=<z-uid> HTTP/1.0" 200 -
Jan 14 13:43:59 raspberrypi rc.local[386]: 127.0.0.1 - - [14/Jan/2021 13:43:59] "GET /API/PicoStill/getFirmwareAddress?uid=<still-uid>&version=0.0.30 HTTP/1.0" 200 -
Jan 14 13:44:17 raspberrypi rc.local[386]: 127.0.0.1 - - [14/Jan/2021 13:44:17] "POST /Vendors/input.cshtml?type=StillRequest&token=<z-uid> HTTP/1.0" 200 -
Jan 14 13:44:25 raspberrypi rc.local[386]: 127.0.0.1 - - [14/Jan/2021 13:44:25] "POST /Vendors/input.cshtml?type=StillRequest&token=<z-uid> HTTP/1.0" 200 -

What this log tells me though is that you didn't actually start a session just went through the clean acknowledgements. Did you start the session? If you did there should be some session log requests from the Z followed by minutely data log points in the graph of the water heating.