grin-miner does not resume mining after connection to stratum server severed

I’ve got 3 miners connected to 1 stratum server. One of the miners runs locally (on the same machine as the stratum server), 2 of them run remotely (on 2 MacBook Pro’s conected over the internet). When the miners lose connection to the stratum server (either because of a flaky network connection or because the stratum server was stopped/restarted), they create a new connection successfully (with a new worker ID) but fail to resume mining.

May 07 08:22:37.000 DEBG Received response with id: 0
May 07 08:22:37.000 INFO Status for worker 0 - Height: 56723, Difficulty: 1, (1440/166/841)
May 07 08:22:38.009 DEBG Mining: Plugin 0 - Device 0 (CPU) Status: OK : Last Graph time: 3.722362434s; Graphs per second: 0.269 - Total Attempts: 109831
May 07 08:22:38.009 INFO Mining: Cuckoo30 at 0.2686465968133612 gps (graphs per second)
May 07 08:22:41.005 DEBG Mining: Plugin 0 - Device 0 (CPU) Status: OK : Last Graph time: 3.741235456s; Graphs per second: 0.267 - Total Attempts: 109832
May 07 08:22:41.005 INFO Mining: Cuckoo30 at 0.2672913832237561 gps (graphs per second)
May 07 08:22:44.006 DEBG Mining: Plugin 0 - Device 0 (CPU) Status: OK : Last Graph time: 3.657733812s; Graphs per second: 0.273 - Total Attempts: 109833
May 07 08:22:44.007 INFO Mining: Cuckoo30 at 0.27339332258659177 gps (graphs per second)
May 07 08:22:47.006 DEBG Mining: Plugin 0 - Device 0 (CPU) Status: OK : Last Graph time: 3.602388513s; Graphs per second: 0.278 - Total Attempts: 109834
May 07 08:22:47.006 INFO Mining: Cuckoo30 at 0.27759360113193876 gps (graphs per second)
May 07 08:22:49.000 ERRO Error reading message: ConnectionError("broken pipe")
May 07 08:22:49.000 WARN Connection Status: Can't establish server connection to 127.0.0.1:13416. Will retry every 5 seconds
May 07 08:22:49.019 DEBG Miner recieved message: StopJob
May 07 08:22:49.255 DEBG Stopping jobs
May 07 08:22:55.000 WARN Connection Status: Connected to Grin server at 127.0.0.1:13416.
May 07 08:22:55.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"getjobtemplate","params":null}
May 07 08:23:06.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 08:23:15.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56723,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd938c746ec80d0dad8db5d27367453d8327fb302767987e649f208f918030641690000000005aeffece00000000000135eaaa4a7ec93c5cd92b4aa0f13c4e1adb44dabfee59bfbacef817ec1f1700a9e26038d55efa9b1d5d0667aff833126b67ddaa3ebf49405b8692075012447813916a650209f1a804b4d38f9dbb4be65aa2c78ef829a34eb64ba881db427b05d99e361f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

May 07 08:23:15.000 DEBG Received request type: job
May 07 08:23:15.000 INFO Got a new job: JobTemplate { height: 56723, difficulty: 1, pre_pow: "0001000000000000dd938c746ec80d0dad8db5d27367453d8327fb302767987e649f208f918030641690000000005aeffece00000000000135eaaa4a7ec93c5cd92b4aa0f13c4e1adb44dabfee59bfbacef817ec1f1700a9e26038d55efa9b1d5d0667aff833126b67ddaa3ebf49405b8692075012447813916a650209f1a804b4d38f9dbb4be65aa2c78ef829a34eb64ba881db427b05d99e361f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 08:23:15.000 DEBG Received message: {"id":"0","jsonrpc":"2.0","method":"getjobtemplate","result":null,"error":{"code":-32701,"message":"Node is syncing - Please wait"}}

May 07 08:23:15.000 DEBG Received response with id: 0
May 07 08:23:15.000 ERRO Failed to get a job template: RpcError { code: -32701, message: "Node is syncing - Please wait" }
May 07 08:23:35.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56724,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd94013b26c698d2969f012338dacc66c91422a605d992a6ffe37bcf5d98d5e8c9c9000000005aeffee200000000000135ebef5725ff15e9a4e1f4e44c1aae9f72ef16a76a4ab252da5f218eeeac87caaa72d72478ac9fa61d243e0b0afe8ab72a32743e87463f186fb5ba78815898e05496f62972149dee78f78e134eb1f48ac7dd2459c54922af3b8658a3dd644bdc56ab1f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

May 07 08:23:35.000 DEBG Received request type: job
May 07 08:23:35.000 INFO Got a new job: JobTemplate { height: 56724, difficulty: 1, pre_pow: "0001000000000000dd94013b26c698d2969f012338dacc66c91422a605d992a6ffe37bcf5d98d5e8c9c9000000005aeffee200000000000135ebef5725ff15e9a4e1f4e44c1aae9f72ef16a76a4ab252da5f218eeeac87caaa72d72478ac9fa61d243e0b0afe8ab72a32743e87463f186fb5ba78815898e05496f62972149dee78f78e134eb1f48ac7dd2459c54922af3b8658a3dd644bdc56ab1f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 08:23:35.000 DEBG Received message: {"id":"0","jsonrpc":"2.0","method":"status","result":"{\"id\":\"0\",\"height\":56724,\"difficulty\":1,\"accepted\":0,\"rejected\":0,\"stale\":0}","error":null}

May 07 08:23:35.000 DEBG Received response with id: 0
May 07 08:23:35.000 INFO Status for worker 0 - Height: 56724, Difficulty: 1, (0/0/0)
May 07 08:23:37.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 08:23:53.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56726,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd9623fe44663ee00a7051902935f13da41f3c8c16b423bd36263205627d2059cd57000000005aeffef500000000000135eddbdcbe83024770107f779977695e48a2ddc6173bd5ffb2572e601cf0057eb82fa093fb84983d0ef08c08eb23a14695f8310ec85022cf71401fd86b0625e5dbd71a6bb7351af6ebf28dceae82327ac444dbb5401cca824acf72c535644e1cae5a1f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

May 07 08:23:53.000 DEBG Received request type: job
May 07 08:23:53.000 INFO Got a new job: JobTemplate { height: 56726, difficulty: 1, pre_pow: "0001000000000000dd9623fe44663ee00a7051902935f13da41f3c8c16b423bd36263205627d2059cd57000000005aeffef500000000000135eddbdcbe83024770107f779977695e48a2ddc6173bd5ffb2572e601cf0057eb82fa093fb84983d0ef08c08eb23a14695f8310ec85022cf71401fd86b0625e5dbd71a6bb7351af6ebf28dceae82327ac444dbb5401cca824acf72c535644e1cae5a1f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 08:23:53.000 DEBG Received message: {"id":"0","jsonrpc":"2.0","method":"status","result":"{\"id\":\"0\",\"height\":56726,\"difficulty\":1,\"accepted\":0,\"rejected\":0,\"stale\":0}","error":null}

May 07 08:23:53.000 DEBG Received response with id: 0
May 07 08:23:53.000 INFO Status for worker 0 - Height: 56726, Difficulty: 1, (0/0/0)
May 07 08:24:08.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 08:24:13.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56727,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd97c725f4a63f20273d669a852f33570ab890d30ae7b77d6b7d437bc8132174a207000000005aefff0800000000000135ee5039422e68a14ba30fb598f86237a54efc3336fde346c1090eabc3d9499d6aae62332147fd6d737de255e6e7e4af6318530cd45fd4d91a9948ec6c09ac6bda4f1661c3ef2e98bfed3aa58bb1627b88307a4b5b4a1ee9015ddedc246827e2a8481f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

May 07 08:24:13.000 DEBG Received request type: job
May 07 08:24:13.000 INFO Got a new job: JobTemplate { height: 56727, difficulty: 1, pre_pow: "0001000000000000dd97c725f4a63f20273d669a852f33570ab890d30ae7b77d6b7d437bc8132174a207000000005aefff0800000000000135ee5039422e68a14ba30fb598f86237a54efc3336fde346c1090eabc3d9499d6aae62332147fd6d737de255e6e7e4af6318530cd45fd4d91a9948ec6c09ac6bda4f1661c3ef2e98bfed3aa58bb1627b88307a4b5b4a1ee9015ddedc246827e2a8481f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 08:24:13.000 DEBG Received message: {"id":"0","jsonrpc":"2.0","method":"status","result":"{\"id\":\"0\",\"height\":56727,\"difficulty\":1,\"accepted\":0,\"rejected\":0,\"stale\":0}","error":null}

May 07 08:24:13.000 DEBG Received response with id: 0
May 07 08:24:13.000 INFO Status for worker 0 - Height: 56727, Difficulty: 1, (0/0/0)
May 07 08:24:31.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56728,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd98d9a1b6faa8bbc4165df4f8b9f8d9746c0b55dfaa75e242d5f7c5df90cc64376a000000005aefff1b00000000000135efd84df979025358be033774ad47ee40d878d4bd3ed5072f3e6908070b51fedec701f3cfafd78d7bc9932dd96ef74d4135145acc729e786a98b45addc209067b2f287bd49f2bb2983ad6a801a1c78abaed0d47d27d67e8e7bacdb0a65f6736a2911f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

May 07 08:24:31.000 DEBG Received request type: job
May 07 08:24:31.000 INFO Got a new job: JobTemplate { height: 56728, difficulty: 1, pre_pow: "0001000000000000dd98d9a1b6faa8bbc4165df4f8b9f8d9746c0b55dfaa75e242d5f7c5df90cc64376a000000005aefff1b00000000000135efd84df979025358be033774ad47ee40d878d4bd3ed5072f3e6908070b51fedec701f3cfafd78d7bc9932dd96ef74d4135145acc729e786a98b45addc209067b2f287bd49f2bb2983ad6a801a1c78abaed0d47d27d67e8e7bacdb0a65f6736a2911f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 08:24:39.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 08:24:51.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56729,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd99f794a71ffdaab682c1b928fe5183cdedf6c537a6562f89eb23e01384b34758e3000000005aefff2e00000000000135f092b357b92439d1a1fb6b32b79482d634d0d01e08e6c89e93d727633312904cdd9bffbbe2bb8b1dcdbe4a603bbe9163f63d0a848fa69ddde28f779032657f5eae7a4f37d0b4f28243fff10bf96a8dd9106b2d167a72108984807abc0adbc2e5411f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

May 07 08:24:51.000 DEBG Received request type: job
May 07 08:24:51.000 INFO Got a new job: JobTemplate { height: 56729, difficulty: 1, pre_pow: "0001000000000000dd99f794a71ffdaab682c1b928fe5183cdedf6c537a6562f89eb23e01384b34758e3000000005aefff2e00000000000135f092b357b92439d1a1fb6b32b79482d634d0d01e08e6c89e93d727633312904cdd9bffbbe2bb8b1dcdbe4a603bbe9163f63d0a848fa69ddde28f779032657f5eae7a4f37d0b4f28243fff10bf96a8dd9106b2d167a72108984807abc0adbc2e5411f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 08:24:51.000 DEBG Received message: {"id":"0","jsonrpc":"2.0","method":"status","result":"{\"id\":\"0\",\"height\":56729,\"difficulty\":1,\"accepted\":0,\"rejected\":0,\"stale\":0}","error":null}

May 07 08:24:51.000 DEBG Received response with id: 0
May 07 08:24:51.000 INFO Status for worker 0 - Height: 56729, Difficulty: 1, (0/0/0)
May 07 08:25:10.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 08:25:15.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56730,\"difficulty\":1,\"pre_pow\":\"0001000000000000dd9ad377d42b43abe92e8ff66bf4d393a5e6f1972caa90ee0a5b44bc6057bbd1397d000000005aefff4600000000000135f164776439dc7b8edfb0290a601f1825cb187a48305829ec441103cdb135fe0b3ffea096166fa92df0775baa327774c32c73e2724462befe67de290beb6353bf7fe90b8763d3cd3c0c2fca33ebc6eda070843b39406a067f93cebb535d3cdb1d7c1f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}

Confirmed. Stopped 'grin server' and started after a few seconds new. On both miners:

May 07 11:01:59.000 ERRO Communication error with stratum server: Connection reset by peer (os error 104)
May 07 11:01:59.000 ERRO Error reading message: ConnectionError("broken pipe")
May 07 11:01:59.023 WARN Connection Status: Can't establish server connection to 193.175.82.53:13416. Will retry every 5 seconds
May 07 11:02:05.022 WARN Connection Status: Can't establish server connection to 193.175.82.53:13416. Will retry every 5 seconds
May 07 11:02:11.022 WARN Connection Status: Can't establish server connection to 193.175.82.53:13416. Will retry every 5 seconds
May 07 11:02:17.022 WARN Connection Status: Can't establish server connection to 193.175.82.53:13416. Will retry every 5 seconds
May 07 11:02:23.022 WARN Connection Status: Can't establish server connection to 193.175.82.53:13416. Will retry every 5 seconds
May 07 11:02:29.023 WARN Connection Status: Connected to Grin server at 193.175.82.53:13416.
May 07 11:02:29.023 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"getjobtemplate","params":null}
May 07 11:02:29.023 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 11:03:00.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 11:03:31.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 11:03:45.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56879,\"difficulty\":1,\"pre_pow\":\"0001000000000000de2f78f9a023eb167d3a9cd09ebe38f479d81047ab6bef2da400d5559758a6f6c8e1000000005af016220000000000013693437f12ef638824bbb8dff9551a6e61f8bdc03bde6d0a93736fcfdc215d0544525eca16102b2ea0f83573a8121be25f26fa35e4e9f8ead9934ffcc96067316fecf34b20fc5f5235d2c810b89f34c99e03a3a0c39e7a31458369d6b72b90f306c41f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}
May 07 11:03:45.000 DEBG Received request type: job
May 07 11:03:45.000 INFO Got a new job: JobTemplate { height: 56879, difficulty: 1, pre_pow: "0001000000000000de2f78f9a023eb167d3a9cd09ebe38f479d81047ab6bef2da400d5559758a6f6c8e1000000005af016220000000000013693437f12ef638824bbb8dff9551a6e61f8bdc03bde6d0a93736fcfdc215d0544525eca16102b2ea0f83573a8121be25f26fa35e4e9f8ead9934ffcc96067316fecf34b20fc5f5235d2c810b89f34c99e03a3a0c39e7a31458369d6b72b90f306c41f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 11:03:45.000 DEBG Received message: {"id":"1","jsonrpc":"2.0","method":"getjobtemplate","result":"{\"height\":56879,\"difficulty\":1,\"pre_pow\":\"0001000000000000de2f78f9a023eb167d3a9cd09ebe38f479d81047ab6bef2da400d5559758a6f6c8e1000000005af016220000000000013693437f12ef638824bbb8dff9551a6e61f8bdc03bde6d0a93736fcfdc215d0544525eca16102b2ea0f83573a8121be25f26fa35e4e9f8ead9934ffcc96067316fecf34b20fc5f5235d2c810b89f34c99e03a3a0c39e7a31458369d6b72b90f306c41f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}","error":null}
May 07 11:03:45.000 DEBG Received response with id: 1
May 07 11:03:45.001 INFO Got a job at height 56879 and difficulty 1
May 07 11:04:02.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 11:04:33.000 DEBG sending request: {"id":"0","jsonrpc":"2.0","method":"status","params":null}
May 07 11:04:39.000 DEBG Received message: {"id":"Stratum","jsonrpc":"2.0","method":"job","params":"{\"height\":56881,\"difficulty\":1,\"pre_pow\":\"0001000000000000de316820fdfc32a8d0b3bb4b1c3c5708fc69520efcdcc6a72583924657cea6179973000000005af0166e00000000000136953c3726b8b47b99a8c6a2452efb02f5bb8c324298d81db11b1449fd5f95cc5ae81e3699b4614431ed3d1452ab5853e03bd3aa2463e212d0821822f0162811a9587a33499ee48fdc99ec5e802ee8e6535a0c3c957ac166826e792f3f100d6c5fb61f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d\"}"}
May 07 11:04:39.000 DEBG Received request type: job
May 07 11:04:39.000 INFO Got a new job: JobTemplate { height: 56881, difficulty: 1, pre_pow: "0001000000000000de316820fdfc32a8d0b3bb4b1c3c5708fc69520efcdcc6a72583924657cea6179973000000005af0166e00000000000136953c3726b8b47b99a8c6a2452efb02f5bb8c324298d81db11b1449fd5f95cc5ae81e3699b4614431ed3d1452ab5853e03bd3aa2463e212d0821822f0162811a9587a33499ee48fdc99ec5e802ee8e6535a0c3c957ac166826e792f3f100d6c5fb61f9c6284dfee7be0fff46f5e3b53d05161464b4be24a8b8774c1e25f60417e1d" }
May 07 11:04:39.000 DEBG Received message: {"id":"1","jsonrpc":"2.0","method":"status","result":"{\"id\":\"1\",\"height\":56881,\"difficulty\":1,\"accepted\":0,\"rejected\":0,\"stale\":0}","error":null}

Both miners (one arm64 with 6 cores, the other arm64 with 4 cores) use two of the available cores for 'whatever':

[grin@rock64 miner]$ top -b -1 -n1 | head -n20
top - 11:21:22 up 9 days, 19:02,  1 user,  load average: 2.18, 2.12, 2.10
Tasks: 102 total,   2 running, 100 sleeping,   0 stopped,   0 zombie
%Cpu0  :  0.0 us, 11.8 sy,  0.0 ni, 88.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  : 23.5 us,  0.0 sy,  0.0 ni, 76.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3986.2 total,   3278.0 free,     66.9 used,    641.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   3880.0 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
11690 grin      20   0  645428   5940   4656 R 218.8   0.1   1767:41 grin-miner
12007 grin      20   0    9444   3312   2788 R   6.2   0.1   0:00.02 top
    1 root      20   0  162072   8236   6260 S   0.0   0.2   0:11.52 systemd

top - 09:26:26 up 2 days, 10:57,  1 user,  load average: 2.27, 2.18, 2.17
Tasks: 102 total,   2 running, 100 sleeping,   0 stopped,   0 zombie
%Cpu0  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  : 16.3 us,  0.3 sy,  0.0 ni, 83.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  4035764 total,  3465248 free,    49200 used,   521316 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  3703436 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                 
 2260 grin      20   0  642648   9484   3104 R 215.9  0.2  17316:09 grin-miner                                                              
    1 root      20   0    6836   4432   2744 S   0.0  0.1   0:09.16 systemd                                                                 
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.02 kthreadd

After restarting both of the miners by pressing 'q' in the tui and starting grin-miner new, the mining works as expected.

mimblewimble / grin-miner

grin-miner does not resume mining after connection to stratum server severed #5