iSpindel no longer reporting, code 422

[sickchilly]

Some server update about Feb 5th caused my iSpindel to stop reporting. I thought maybe the battery died, but I started a new ferm session today (after a full charge) and it’s still not working. My PicoFerm is working and I see error 422 in the logs for the iSpindel. picobrew_pico.log

[BuckoWA]

@sickchilly Just tried on my setup with iSpindel. No issue seen with the latest release. I definitely see the 422 in your logs. Can you check the iSpindel/sessions/active folder and see if there’s something in there?

[sickchilly]

Nope, nothing there. Throughout the day, I’ve rebooted the Pi multiple times, stopped and started the iSpindel session, and still nothing. Haven’t changed anything other than updating the server as new versions have been released over the past couple of days.

[BuckoWA]

@sickchilly Tried again this morning on second server with different iSpindel. I'm not able to replicate the issue you are seeing. Tried multiple ways to induce a failure. Will try more troubleshooting later. Can you check the nginx logs for issues? Sorry for the issues you're having.

[sickchilly]

Here are the full logs as of now. It's still not reporting and the active sessions directory is still empty. (The NGINX error log is also empty, so not attached.) I'm not seeing anything obvious. I don't think I've done this in Github before... is there an easy way to roll back to a previous build/snapshot of the released code package? I would go back to Feb 3 or 4 to see if it starts working again. nginx.access.log picobrew_pico.log

[tmack8001]

@sickchilly you can go and reset the commit your local git directory is tracking.

From the command line and specifically the git directory of the server /picobrew_pico run the following command to go back to February 1st, https://github.com/chiefwigms/picobrew_pico/commit/ff399d862152ebdde7dae02f385a335767471a5b

git reset --hard ff399d862152ebdde7dae02f385a335767471a5b

[sickchilly]

Thanks for that. It wasn't obvious where to obtain that full UUID or whatever it is. I see where it comes from now.

But, I've narrowed down the breaking point...

1. I first went back to that Feb 1 build and my iSpindel I still got the 422 errors with no reporting.
2. I then looked at the comments for an obvious iSpindel major feature commit and decided to go back to d2304961c1125e1d02ef0cc036875ad2b9779859 (2nd commit on Jan 28 just before start/stop was added to iSpindel). With this build, my iSpindel reports correctly again.
3. Then I started working forward. It worked until I got to d41a44fabd8892c0f2b9059aff4a6ded615788d8, when it broke again. This commit was from Feb 3 with note "update iSpindel routes to support iSpindle, but also support both set…"
4. If I go back to the build prior (ff399d862152ebdde7dae02f385a335767471a5b), my iSpindel works again.

I completely restarted the Pi between each downgrade/upgrade. That was quite painful having to wait 5-10 minutes and a restart between builds for confirmation, but hope this helps isolate things and points to a cause.

Note to anyone who might go spelunking into the release history like this: Be sure you set "update_boot: false" in config.yaml or else you'll be chasing your tail!

[tmack8001]

Note to anyone who might go spelunking into the release history like this: Be sure you set "update_boot: false" in config.yaml or else you'll be chasing your tail!

Nope no reason to restart the Pi like ever 😜 .

Can simply change source files in git and restart the process via

systemctl restart rc.local

[tmack8001]

@sickchilly

I then looked at the comments for an obvious iSpindel major feature commit and decided to go back to d230496 (2nd commit on Jan 28 just before start/stop was added to iSpindel). With this build, my iSpindel reports correctly again.

obvious questions maybe...

• did you click on "start fermentation"?
• what is the route you configured your ispindel with?

[sickchilly]

Ha! It’s my test background coming through... as soon as you file a bug without mentioning you restarted, the dev will ask, “Well, did you try a reboot?”

[sickchilly]

This iSpindel is in active fermentation, so when I rolled back to a version before the start/stop function, it created a session and started graphing. When I updated through and past the version with the start/stop function, it was already started and graphing since a session existed. I can’t get the settings off it at the moment, but I haven’t changed them in months. I may try setting up another server instance and factory reset, and connect my second iSpindel to it.

[tmack8001]

If there is an active session file detected on server startup (as mentioned doesn't have to be a "reboot" event) the server will mark the in-memory session as session.active == True and log the data from the iSpindel API requests made. If there is not an active session file AND the end user (you the brewer) don't click Start Fermentation a session file will not be created and data logs flowing from the iSpindel to the server will be simply ignored.

[sickchilly]

I know I'm flirting with disaster here, but as a Hail Mary, I upgraded to the latest version (7d33f59) but replaced routes_iSpindel_api.py from ff399d8. It's working! I'll try factory resetting my 2nd iSpindel that isn't being used right now and attempt to get it going with the latest build on a new server instance. If that works, maybe I need to reset and reconfigure my 1st iSpindel.

[tmack8001]

I just tested locally on the latest master source (7d33f59) and tested both spellings (which was what was added in one of the more recent ispindel changes).

curl --location --request POST 'localhost/API/iSpindel' \
--header 'Content-Type: application/json' \
--data-raw '{
    "ID": 1234567,
    "temperature": 36.8867,
    "temp_units": "C",
    "battery": 5.0,
    "gravity": 1.065
}'

and

curl --location --request POST 'localhost/API/iSpindel' \
--header 'Content-Type: application/json' \
--data-raw '{
    "ID": 1234567,
    "temperature": 36.8867,
    "temp_units": "C",
    "battery": 5.0,
    "gravity": 1.065
}'

both of these patterns worked to log to the session file if and only if there was a session.active == True detected by the server (tested both with pre-existing session file aka server restart during fermentation and manually clicking start).

[sickchilly]

OK, experimenting with my 2nd iSpindel and a new server instance, I think I've found the problem.

I factory reset and re-configured 2nd iSpindel with a new server instance and the latest build. Configured it exactly the same except for the name "iSpindel-2" vs "iSpindel-1" and the server address "piconew.com" vs "picobrew.com".

What I found was I had the Token field populated with my old Ubidots token even though the service type is HTTP.

Whatever changes were made recently no longer tolerates that. Once I deleted the Token field, it started reporting as expected. Put anything in the Token field again and it stops reporting. 422 error.

I'll have to fish the 1st iSpindel out of fermentation and confirm this finding on it.

[BuckoWA]

That explains 422 error - properly formatted just unparsable from Python. The @arg in the API doesn't contain a token. I can update the Support file to include this. I've never connected my iSpindels to Ubidots so never even thought to test with this. Mine have worked consistently through the latest updates. Thanks @sickchilly for all the troubleshooting and @tmack8001 for assist.

[tmack8001]

Ah yes... the classic "you sent more than the server understands and it rejected you" problem. https://github.com/chiefwigms/picobrew_pico/pull/253 fixes that and I tested with several JSON key/value pairs that weren't explicitly defined in the structure.

[tmack8001]

reference for those curious: https://webargs.readthedocs.io/en/latest/advanced.html#setting-unknown

[BuckoWA]

Teach a man to fish... Thanks for the reference!

[sickchilly]

I manually applied the changes in PR 253. It's working for me. Did not adjust my iSpindel, it's still in fermentation. Thanks!