Bad gateway - firmware and update page

[cdnninja]

I am getting a bad gateway error on both the flash firmware page and the update page. This is ever since the most recent tools update.

I have been having disconnects to my Arduino so thought it was time to move away from 2.10 firmware but looks like I can't.

[thorrak]

Unfortunately we’re traveling this week, so I’m away from my test environment but I’ll take a look as soon as I can — no later than this weekend if I can’t spin something up on my laptop.

Just to clarify - are you using the latest release of Fermentrack installed using Docker?

[cdnninja]

Yes running via docker. Can't easily confirm version since the update page won't load but two days ago ran the install.sh script again which I assume should mean I am latest.

It's prompting to update from GitHub though which makes me wonder if not on latest.

[thorrak]

I can see the "upgrade" issue hitting Sentry, but it seems weird as it's acting like it's just the system shutting down. All I can do is try to catch it in case it's some weird cascade of events relating to Git -- I've opened an issue to track against this as #752 .

Given both of these, I would recommend trying to rerun the install.sh script when you have a chance just in case these issues relate to something not downloading properly. If it doesn't do anything, try running ./install.sh -i docker-dev which should force it to use a different (but near-identical) image.

[thorrak]

I got an image booted on my end, but the firmware flash routines seem to be working as expected.

[cdnninja]

Interesting. I can't even hit the page to start flashing firmware.

This comes out of me trying to address it loosing connection every few days to the temp controller. Good news is with this change the tilt is stable though.

I'm not by a computer but will see what I can do with this in a few days.

[cdnninja]

Same gateway error after the update. I don't have the prompt to update from github anymore. Most likely because I used to be set to the dev mode in fermentrack and now running that branch. Looks like the add backup feature does bad gateway too. Was thinking of running a backup and clean install.

As a note I ran sudo ./install as I have that old permissions issue that won't let it install without sudo.

[cdnninja]

Hunting for logs I found the following in the postgres container logs when I go to the backup page or the flash firmware page, but not the github page.

2023-08-11 15:33:39.670 UTC [104] LOG: unexpected EOF on client connection with an open transaction

Not sure if related but the tilt container is also filled with the following. This timeout could be a symptom though since it seems the interface takes awhile to throw the gateway error.

httplib_response = super().getresponse() File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/stdlib.py", line 121, in getresponse rv = real_getresponse(self, *args, **kwargs) File "/usr/local/lib/python3.9/http/client.py", line 1377, in getresponse response.begin() File "/usr/local/lib/python3.9/http/client.py", line 320, in begin version, status, reason = self._read_status() File "/usr/local/lib/python3.9/http/client.py", line 281, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/usr/local/lib/python3.9/socket.py", line 704, in readinto return self._sock.recv_into(b) socket.timeout: timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) File "/usr/local/lib/python3.9/site-packages/urllib3/util/util.py", line 39, in reraise raise value File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 538, in _make_request self._raise_timeout(err=e, url=url, timeout_value=read_timeout) File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout raise ReadTimeoutError( urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=80): Read timed out. (read timeout=5) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/app/data_targets/legacy_fermentrack_target.py", line 64, in process r=requests.post(self.target_url, json=target_dict, timeout=5) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 532, in send raise ReadTimeout(e, request=request) requests.exceptions.ReadTimeout: HTTPConnectionPool(host='127.0.0.1', port=80): Read timed out. (read timeout=5) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/logging/__init__.py", line 1083, in emit msg = self.format(record) File "/usr/local/lib/python3.9/logging/__init__.py", line 927, in format return fmt.format(record) File "/usr/local/lib/python3.9/logging/__init__.py", line 666, in format s = self.formatMessage(record) File "/usr/local/lib/python3.9/logging/__init__.py", line 635, in formatMessage return self._style.format(record) File "/usr/local/lib/python3.9/logging/__init__.py", line 434, in format return self._format(record) File "/usr/local/lib/python3.9/logging/__init__.py", line 430, in _format return self._fmt % record.__dict__ ValueError: unsupported format character ':' (0x3a) at index 24 Call stack: File "/app/tiltbridge_junior.py", line 174, in <module> asyncio.run(async_main()) File "/usr/local/lib/python3.9/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/local/lib/python3.9/asyncio/base_events.py", line 634, in run_until_complete self.run_forever() File "/usr/local/lib/python3.9/asyncio/base_events.py", line 601, in run_forever self._run_once() File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1905, in _run_once handle._run() File "/usr/local/lib/python3.9/asyncio/events.py", line 80, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 811, in _read_ready self._read_ready_cb() File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 868, in _read_ready__data_received self._protocol.data_received(data) File "/usr/local/lib/python3.9/site-packages/aioblescan/aioblescan.py", line 1855, in data_received self.process(packet) File "/app/tiltbridge_junior.py", line 133, in process_ble_beacon data_target_handler.process_data(tilts) File "/app/data_targets/data_target_handler.py", line 14, in process_data target_legacy_fermentrack.process(tilts) File "/app/data_targets/legacy_fermentrack_target.py", line 66, in process LOG.error(e) File "/usr/local/lib/python3.9/logging/__init__.py", line 1475, in error self._log(ERROR, msg, args, **kwargs) File "/usr/local/lib/python3.9/logging/__init__.py", line 1589, in _log self.handle(record) File "/usr/local/lib/python3.9/logging/__init__.py", line 1599, in handle self.callHandlers(record) File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/logging.py", line 96, in sentry_patched_callhandlers return old_callhandlers(self, record) Message: ReadTimeout(ReadTimeoutError("HTTPConnectionPool(host='127.0.0.1', port=80): Read timed out. (read timeout=5)")) Arguments: ()

[thorrak]

This is admittedly the most asinine way I could have solved the flashing problem, but I just released a new version of BrewFlasher that works on the command line called BrewFlasher CLI edition. It’s available onPyPI and the announcement post is here on HBT .

I’m now back home with my Pi’s so I should be able to debug a bit further.

[cdnninja]

It's been stable since pulling from dev. All works now. Going to close this.

[thorrak]

Cheers - thank you!