Improving CRC handling. (On pi zero)

[tzachi-dar]

Looking at the logs of pi zero runs from the nights, I can see the following line 2020-01-20 00:11:17 openaps.pump-loop 2020/01/20 00:11:13 computed CRC AF but received 00 about 5-6 times every hour. Not to mention lines like: 2020-01-20 01:05:22 openaps.pump-loop 2020/01/20 01:05:11 6b/4b decoding failure All this lines represent communication error. Many times this happen in pumphistory, but other commands enjoy it as well. Our handling of this issues is actually through retrying the command. We exit the immediate command, and stop the loop. So, most times retry will happen after a few minutes. Furthermore, in order for an entire loop to complete we need to retry all communication. This increases the chance of another error. My suggestion is to try the fix in the low level driver version. For example, replace the function that reads a page with a function that tries to read the page up to 3 times (until the first page is read correctly). I have tried this, and it seems to allow much more loops to complete per hour.

Second issue is whether our CRC can protect us from failures: Assuming that communication is reliable (and let say that reliable means an error every day) and assuming our checksum is 16 bytes, this means that we will have an error that checksum did not detect once in 65,000 days. (Assuming that there are 1000 loopers, it means that we will have a wrong loop every 2 days). But since we get an error 100 times a day, it means that for a specific user an error will happen once in every 650 days, and for a 1000 users there will be an error every day. So, what is my suggestion? I suggest to read every page twice, and compare. In the case that the pages are not the same read another one and look for 3 similar pages together. I guess that the second suggestion might have influence on batteries and other things that I might not think about. Looking for feedback

Appendix 1: Code for reading up to 3 times: func (pump *Pump) Download(cmd Command, page int) []byte { var data []byte for i:=0 ; i < 3; i++ { data = pump.DownloadIn(cmd, page); if data != nil { return data } fmt.Fprintln(os.Stderr,"Error - retrying ", i) pump.SetError(nil) } fmt.Fprintln(os.Stderr,"Error - quitting") return nil }

[juehv]

I think your second point is not a real issue since the wrong data will be overwritten in the next loop...

[ecc1]

I'm having a hard time coming up with a noisy enough RF environment to test this. Can you try a version of the pumphistory or historypage commands with and without your change, and convince me that there's a difference?

There's already retry logic in the initial execPage (3 times) and for each page fragment in handleNoResponse (10 times) so I don't see how this would help (more than just dialing up the default retry count, say)

[tzachi-dar]

I have updated my printing to be something like the following:

func (pump *Pump) Download(cmd Command, page int) []byte { var data []byte for i:=0 ; i < 10; i++ { data = pump.DownloadIn(cmd, page); if data != nil { return data } fmt.Fprintln(os.Stderr,"Error - retrying ", i) pump.SetError(nil) } fmt.Fprintln(os.Stderr,"Error - quting") return nil }

With this code a print of, "Error - retrying " displays what errors have been found and when they have been fixed:

Two more points: I don't think that my environment is noisy. It is in a storeroom on the ground floor of my building. I didn't find there any internet, so I had to put there a cell phone to have connectivity.

If there are other variables to try and create retries, I'll be happy to try them.

And thanks for your help.

2020-01-29 00:09:38 openaps.pump-loop Error - retrying 0 2020-01-29 00:12:44 openaps.pump-loop Error - retrying 0 2020-01-29 00:17:12 openaps.pump-loop Error - retrying 0 2020-01-29 00:17:12 openaps.pump-loop Error - retrying 1 2020-01-29 00:17:12 openaps.pump-loop Error - retrying 2 2020-01-29 00:17:12 openaps.pump-loop Error - retrying 3 2020-01-29 00:25:35 openaps.pump-loop Error - retrying 0 2020-01-29 00:25:36 openaps.pump-loop Error - retrying 1 2020-01-29 00:25:36 openaps.pump-loop Error - retrying 2 2020-01-29 00:25:36 openaps.pump-loop Error - retrying 3 2020-01-29 00:25:36 openaps.pump-loop Error - retrying 4 2020-01-29 00:39:20 openaps.pump-loop Error - retrying 0 2020-01-29 00:39:20 openaps.pump-loop Error - retrying 1 2020-01-29 00:39:20 openaps.pump-loop Error - retrying 2 2020-01-29 00:55:15 openaps.pump-loop Error - retrying 0 2020-01-29 01:18:24 openaps.pump-loop Error - retrying 0 2020-01-29 01:22:16 openaps.pump-loop Error - retrying 0 2020-01-29 01:37:56 openaps.pump-loop Error - retrying 0 2020-01-29 01:57:21 openaps.pump-loop Error - retrying 0 2020-01-29 01:57:21 openaps.pump-loop Error - retrying 1 2020-01-29 02:00:49 openaps.pump-loop Error - retrying 0 2020-01-29 02:00:50 openaps.pump-loop Error - retrying 1 2020-01-29 02:05:43 openaps.pump-loop Error - retrying 0 2020-01-29 02:05:43 openaps.pump-loop Error - retrying 1 2020-01-29 02:05:43 openaps.pump-loop Error - retrying 2 2020-01-29 02:23:31 openaps.pump-loop Error - retrying 0 2020-01-29 02:29:28 openaps.pump-loop Error - retrying 0 2020-01-29 02:42:33 openaps.pump-loop Error - retrying 0 2020-01-29 02:47:35 openaps.pump-loop Error - retrying 0 2020-01-29 02:47:36 openaps.pump-loop Error - retrying 1 2020-01-29 02:52:24 openaps.pump-loop Error - retrying 0 2020-01-29 02:52:24 openaps.pump-loop Error - retrying 1 2020-01-29 03:14:08 openaps.pump-loop Error - retrying 0 2020-01-29 03:14:08 openaps.pump-loop Error - retrying 1 2020-01-29 03:14:08 openaps.pump-loop Error - retrying 2 2020-01-29 03:14:08 openaps.pump-loop Error - retrying 3 2020-01-29 03:14:09 openaps.pump-loop Error - retrying 4 2020-01-29 03:17:45 openaps.pump-loop Error - retrying 0 2020-01-29 03:28:31 openaps.pump-loop Error - retrying 0 2020-01-29 03:28:31 openaps.pump-loop Error - retrying 1 2020-01-29 03:28:31 openaps.pump-loop Error - retrying 2 2020-01-29 03:31:35 openaps.pump-loop Error - retrying 0 2020-01-29 03:31:35 openaps.pump-loop Error - retrying 1 2020-01-29 03:31:35 openaps.pump-loop Error - retrying 2 2020-01-29 03:37:27 openaps.pump-loop Error - retrying 0 2020-01-29 03:37:27 openaps.pump-loop Error - retrying 1 2020-01-29 03:47:50 openaps.pump-loop Error - retrying 0 2020-01-29 03:47:50 openaps.pump-loop Error - retrying 1 2020-01-29 03:51:19 openaps.pump-loop Error - retrying 0 2020-01-29 03:57:49 openaps.pump-loop Error - retrying 0 2020-01-29 03:57:49 openaps.pump-loop Error - retrying 1 2020-01-29 04:25:13 openaps.pump-loop Error - retrying 0 2020-01-29 04:32:48 openaps.pump-loop Error - retrying 0 2020-01-29 04:41:27 openaps.pump-loop Error - retrying 0 2020-01-29 04:46:24 openaps.pump-loop Error - retrying 0 2020-01-29 04:46:24 openaps.pump-loop Error - retrying 1 2020-01-29 04:56:44 openaps.pump-loop Error - retrying 0 2020-01-29 05:19:54 openaps.pump-loop Error - retrying 0 2020-01-29 05:36:27 openaps.pump-loop Error - retrying 0 2020-01-29 05:36:27 openaps.pump-loop Error - retrying 1 2020-01-29 05:39:58 openaps.pump-loop Error - retrying 0 2020-01-29 05:39:58 openaps.pump-loop Error - retrying 1 2020-01-29 05:48:14 openaps.pump-loop Error - retrying 0 2020-01-29 05:48:14 openaps.pump-loop Error - retrying 1 2020-01-29 06:04:08 openaps.pump-loop Error - retrying 0 2020-01-29 06:04:08 openaps.pump-loop Error - retrying 1 2020-01-29 06:07:42 openaps.pump-loop Error - retrying 0 2020-01-29 06:07:42 openaps.pump-loop Error - retrying 1 2020-01-29 06:20:39 openaps.pump-loop Error - retrying 0 2020-01-29 06:20:39 openaps.pump-loop Error - retrying 1 2020-01-29 06:20:39 openaps.pump-loop Error - retrying 2 2020-01-29 06:20:40 openaps.pump-loop Error - retrying 3 2020-01-29 06:45:14 openaps.pump-loop Error - retrying 0 2020-01-29 06:45:14 openaps.pump-loop Error - retrying 1 2020-01-29 06:45:14 openaps.pump-loop Error - retrying 2 2020-01-29 06:55:20 openaps.pump-loop Error - retrying 0 2020-01-29 07:04:55 openaps.pump-loop Error - retrying 0 2020-01-29 07:08:55 openaps.pump-loop Error - retrying 0 2020-01-29 07:08:55 openaps.pump-loop Error - retrying 1 2020-01-29 07:12:33 openaps.pump-loop Error - retrying 0 2020-01-29 07:12:33 openaps.pump-loop Error - retrying 1 2020-01-29 07:23:18 openaps.pump-loop Error - retrying 0 2020-01-29 07:31:26 openaps.pump-loop Error - retrying 0 2020-01-29 07:31:26 openaps.pump-loop Error - retrying 1 2020-01-29 07:56:27 openaps.pump-loop Error - retrying 0 2020-01-29 08:20:25 openaps.pump-loop Error - retrying 0 2020-01-29 08:47:13 openaps.pump-loop Error - retrying 0 2020-01-29 08:47:13 openaps.pump-loop Error - retrying 1 2020-01-29 09:00:16 openaps.pump-loop Error - retrying 0 2020-01-29 09:00:16 openaps.pump-loop Error - retrying 1 2020-01-29 10:31:21 openaps.pump-loop Error - retrying 0 2020-01-29 10:31:21 openaps.pump-loop Error - retrying 1 2020-01-29 10:31:21 openaps.pump-loop Error - retrying 2 2020-01-29 10:31:21 openaps.pump-loop Error - retrying 3 2020-01-29 10:44:32 openaps.pump-loop Error - retrying 0 2020-01-29 10:44:32 openaps.pump-loop Error - retrying 1 2020-01-29 10:57:34 openaps.pump-loop Error - retrying 0 2020-01-29 10:57:34 openaps.pump-loop Error - retrying 1 2020-01-29 11:10:38 openaps.pump-loop Error - retrying 0 2020-01-29 11:10:38 openaps.pump-loop Error - retrying 1 2020-01-29 11:10:38 openaps.pump-loop Error - retrying 2 2020-01-29 11:10:39 openaps.pump-loop Error - retrying 3 2020-01-29 11:12:25 openaps.pump-loop Error - retrying 0 2020-01-29 11:12:26 openaps.pump-loop Error - retrying 1 2020-01-29 11:20:33 openaps.pump-loop Error - retrying 0 2020-01-29 11:34:28 openaps.pump-loop Error - retrying 0 2020-01-29 11:46:25 openaps.pump-loop Error - retrying 0 2020-01-29 11:46:25 openaps.pump-loop Error - retrying 1 2020-01-29 11:59:21 openaps.pump-loop Error - retrying 0 2020-01-29 11:59:21 openaps.pump-loop Error - retrying 1 2020-01-29 11:59:21 openaps.pump-loop Error - retrying 2 2020-01-29 11:59:21 openaps.pump-loop Error - retrying 3 2020-01-29 12:39:23 openaps.pump-loop Error - retrying 0 2020-01-29 12:39:24 openaps.pump-loop Error - retrying 1 2020-01-29 12:51:35 openaps.pump-loop Error - retrying 0 2020-01-29 12:51:35 openaps.pump-loop Error - retrying 1 2020-01-29 12:58:25 openaps.pump-loop Error - retrying 0 2020-01-29 12:58:25 openaps.pump-loop Error - retrying 1 2020-01-29 13:11:48 openaps.pump-loop Error - retrying 0 2020-01-29 13:11:48 openaps.pump-loop Error - retrying 1 2020-01-29 13:11:48 openaps.pump-loop Error - retrying 2 2020-01-29 13:24:25 openaps.pump-loop Error - retrying 0 2020-01-29 13:24:25 openaps.pump-loop Error - retrying 1 2020-01-29 13:24:25 openaps.pump-loop Error - retrying 2 2020-01-29 13:39:05 openaps.pump-loop Error - retrying 0 2020-01-29 13:52:27 openaps.pump-loop Error - retrying 0 2020-01-29 13:52:27 openaps.pump-loop Error - retrying 1 2020-01-29 13:52:27 openaps.pump-loop Error - retrying 2 2020-01-29 13:52:27 openaps.pump-loop Error - retrying 3 2020-01-29 14:00:24 openaps.pump-loop Error - retrying 0 2020-01-29 14:00:24 openaps.pump-loop Error - retrying 1 2020-01-29 14:00:24 openaps.pump-loop Error - retrying 2 2020-01-29 14:09:06 openaps.pump-loop Error - retrying 0 2020-01-29 14:09:06 openaps.pump-loop Error - retrying 1 2020-01-29 14:09:07 openaps.pump-loop Error - retrying 2 2020-01-29 14:12:29 openaps.pump-loop Error - retrying 0 2020-01-29 14:12:29 openaps.pump-loop Error - retrying 1 2020-01-29 14:12:31 openaps.pump-loop Error - retrying 2 2020-01-29 14:23:21 openaps.pump-loop Error - retrying 0 2020-01-29 14:28:20 openaps.pump-loop Error - retrying 0 2020-01-29 14:39:28 openaps.pump-loop Error - retrying 0 2020-01-29 14:39:28 openaps.pump-loop Error - retrying 1 2020-01-29 14:52:24 openaps.pump-loop Error - retrying 0 2020-01-29 14:52:24 openaps.pump-loop Error - retrying 1 2020-01-29 14:52:24 openaps.pump-loop Error - retrying 2 2020-01-29 15:04:39 openaps.pump-loop Error - retrying 0 2020-01-29 15:51:31 openaps.pump-loop Error - retrying 0 2020-01-29 15:51:32 openaps.pump-loop Error - retrying 1 2020-01-29 15:59:55 openaps.pump-loop Error - retrying 0 2020-01-29 15:59:55 openaps.pump-loop Error - retrying 1 2020-01-29 16:13:54 openaps.pump-loop Error - retrying 0 2020-01-29 16:25:55 openaps.pump-loop Error - retrying 0 2020-01-29 16:25:55 openaps.pump-loop Error - retrying 1 2020-01-29 16:25:55 openaps.pump-loop Error - retrying 2 2020-01-29 16:39:32 openaps.pump-loop Error - retrying 0 2020-01-29 16:54:19 openaps.pump-loop Error - retrying 0 2020-01-29 16:54:19 openaps.pump-loop Error - retrying 1 2020-01-29 16:54:19 openaps.pump-loop Error - retrying 2 2020-01-29 17:07:20 openaps.pump-loop Error - retrying 0 2020-01-29 17:07:21 openaps.pump-loop Error - retrying 1 2020-01-29 17:19:55 openaps.pump-loop Error - retrying 0 2020-01-29 17:19:55 openaps.pump-loop Error - retrying 1 2020-01-29 17:59:21 openaps.pump-loop Error - retrying 0 2020-01-29 17:59:21 openaps.pump-loop Error - retrying 1 2020-01-29 18:42:19 openaps.pump-loop Error - retrying 0 2020-01-29 18:55:27 openaps.pump-loop Error - retrying 0 2020-01-29 18:55:27 openaps.pump-loop Error - retrying 1 2020-01-29 19:07:36 openaps.pump-loop Error - retrying 0 2020-01-29 19:07:36 openaps.pump-loop Error - retrying 1 2020-01-29 19:20:56 openaps.pump-loop Error - retrying 0 2020-01-29 19:20:56 openaps.pump-loop Error - retrying 1 2020-01-29 19:39:47 openaps.pump-loop Error - retrying 0 2020-01-29 19:58:00 openaps.pump-loop Error - retrying 0 2020-01-29 20:07:20 openaps.pump-loop Error - retrying 0 2020-01-29 20:07:20 openaps.pump-loop Error - retrying 1 2020-01-29 20:07:20 openaps.pump-loop Error - retrying 2 2020-01-29 20:07:21 openaps.pump-loop Error - retrying 3 2020-01-29 20:19:51 openaps.pump-loop Error - retrying 0 2020-01-29 20:26:31 openaps.pump-loop Error - retrying 0 2020-01-29 20:55:12 openaps.pump-loop Error - retrying 0 2020-01-29 20:55:12 openaps.pump-loop Error - retrying 1 2020-01-29 20:59:20 openaps.pump-loop Error - retrying 0 2020-01-29 21:03:13 openaps.pump-loop Error - retrying 0 2020-01-29 21:03:13 openaps.pump-loop Error - retrying 1 2020-01-29 21:49:28 openaps.pump-loop Error - retrying 0 2020-01-29 21:49:28 openaps.pump-loop Error - retrying 1 2020-01-29 21:49:29 openaps.pump-loop Error - retrying 2 2020-01-29 22:02:24 openaps.pump-loop Error - retrying 0 2020-01-29 22:02:24 openaps.pump-loop Error - retrying 1 2020-01-29 22:02:24 openaps.pump-loop Error - retrying 2 2020-01-29 22:15:37 openaps.pump-loop Error - retrying 0 2020-01-29 22:43:28 openaps.pump-loop Error - retrying 0 2020-01-29 22:43:29 openaps.pump-loop Error - retrying 1 2020-01-29 22:57:26 openaps.pump-loop Error - retrying 0 2020-01-29 23:10:00 openaps.pump-loop Error - retrying 0 2020-01-29 23:10:00 openaps.pump-loop Error - retrying 1 2020-01-29 23:16:18 openaps.pump-loop Error - retrying 0 2020-01-29 23:16:18 openaps.pump-loop Error - retrying 1 2020-01-29 23:16:18 openaps.pump-loop Error - retrying 2 2020-01-29 23:42:25 openaps.pump-loop Error - retrying 0 2020-01-29 23:55:45 openaps.pump-loop Error - retrying 0

[ecc1]

Interesting, but I'd still like to understand what the underlying errors are.

You could use fmt.Fprintf(os.Stderr,"Error (%v) - retrying %d\n", pump.Error(), i) to print a little more info. Are you able to run without your change for a similar amount of time, for comparison?

[tzachi-dar]

I'll do that tomorrow and let you know how it is working. Without my changes, there are (sometimes) pumphistory failures. Anything more that you want to know about what is happening without my changes?

[tzachi-dar]

Here is one more update:

I have compiled the code the way you want, stopped the loop and run "pumphistory " about 15 times "pumphistory -n 27 " about 5 times "pumphistory -n 270 " about 5 times All went well, without an error

Than I have started the loop and got the following results:

2020-02-10 00:10:48 openaps.pump-loop Error - retrying 0 2020-02-10 00:10:48 openaps.pump-loop Error (computed CRC BC but received 30) - retrying 0 2020-02-10 00:10:48 openaps.pump-loop Error - retrying 1 2020-02-10 00:10:48 openaps.pump-loop Error (6b/4b decoding failure) - retrying 1 2020-02-10 00:45:27 openaps.pump-loop Error - retrying 0 2020-02-10 00:45:27 openaps.pump-loop Error (no response to historyPage) - retrying 0 2020-02-10 00:45:27 openaps.pump-loop Error - retrying 1 2020-02-10 00:45:27 openaps.pump-loop Error (no response to historyPage) - retrying 1 2020-02-10 01:06:31 openaps.pump-loop Error - retrying 0 2020-02-10 01:06:31 openaps.pump-loop Error (6b/4b decoding failure) - retrying 0 2020-02-10 01:06:31 openaps.pump-loop Error - retrying 1 2020-02-10 01:06:31 openaps.pump-loop Error (no response to historyPage) - retrying 1 2020-02-10 01:06:31 openaps.pump-loop Error - retrying 2 2020-02-10 01:06:31 openaps.pump-loop Error (6b/4b decoding failure) - retrying 2 2020-02-10 01:06:32 openaps.pump-loop Error - retrying 3 2020-02-10 01:06:32 openaps.pump-loop Error (no response to historyPage) - retrying 3 2020-02-10 01:19:35 openaps.pump-loop Error - retrying 0 2020-02-10 01:19:35 openaps.pump-loop Error (6b/4b decoding failure) - retrying 0 2020-02-10 01:19:35 openaps.pump-loop Error - retrying 1 2020-02-10 01:19:35 openaps.pump-loop Error (6b/4b decoding failure) - retrying 1 2020-02-10 01:19:35 openaps.pump-loop Error - retrying 2 2020-02-10 01:19:35 openaps.pump-loop Error (6b/4b decoding failure) - retrying 2 2020-02-10 01:19:35 openaps.pump-loop Error - retrying 3 2020-02-10 01:19:35 openaps.pump-loop Error (computed CRC 90 but received 00) - retrying 3 2020-02-10 01:19:35 openaps.pump-loop Error - retrying 4 2020-02-10 01:19:35 openaps.pump-loop Error (6b/4b decoding failure) - retrying 4 2020-02-10 01:27:20 openaps.pump-loop Error - retrying 0 2020-02-10 01:27:20 openaps.pump-loop Error (6b/4b decoding failure) - retrying 0 2020-02-10 01:27:20 openaps.pump-loop Error - retrying 1 2020-02-10 01:27:20 openaps.pump-loop Error (no response to historyPage) - retrying 1 2020-02-10 01:27:20 openaps.pump-loop Error - retrying 2 2020-02-10 01:27:20 openaps.pump-loop Error (6b/4b decoding failure) - retrying 2

[ecc1]

I've add this to the dev branch (it also requires the dev branch of my nightscout package to build). I've migrated it all to Go modules, so it should work if you do go get github.com/ecc1/medtronic@dev

[tzachi-dar]

Thanks for adding this. I'm starting to test the dev branch, hope everything will be fine.