Closed dstewen closed 2 years ago
@dstewen I've not tried this on a Pi Zero - there's a chance that the device does not have enough memory for the stack being used here. Can you check memory usage? My guess is that the OOM killer is killing the speedtest exe.
I am trying to install this in an RPI0 and the 'speedtest' never starts, it continues attempting to download, and has never gotten past more than 17%.
It doesn't appear memory is an issue:
root@4a68129:~# free -m
total used free shared buff/cache available
Mem: 490304 165420 150828 18136 174056 306924
Swap: 122572 6144 116428
I got some errors from the logs:
Failed to start service 'mqtt sha256:246cd51101f808074ee0d0aeeaf9d634d54fa9aa8cee59f83a9e3213b86f5b4a' due to 'Could not parse status code from docker error: Error: socket hang up'
Failed to start service 'connector sha256:e5f06c9c72c47777851659d78654bdeff0aa77a6930f1c9bb5d86724eefb7a00' due to 'Could not parse status code from docker error: Error: socket hang up'
Failed to start service 'influxdb sha256:0866da0fd5bb684b2b9b13002956f1d9806bc8eac6d62a98bf1056d4268a25c6' due to 'Could not parse status code from docker error: Error: socket hang up'
Failed to download image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7' due to 'connect ECONNREFUSED /var/run/balena-engine.sock'
Supervisor starting
At which point it restarts Downloading delta for image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
.
I'm guessing for my problem it is a lack of device space:
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 232M 0 232M 0% /dev
tmpfs tmpfs 240M 4.0K 240M 1% /tmp
/dev/disk/by-state/active ext4 300M 243M 38M 87% /mnt/sysroot/active
/dev/disk/by-state/resin-state ext4 19M 224K 17M 2% /mnt/state
none aufs 300M 243M 38M 87% /
tmpfs tmpfs 240M 476K 239M 1% /dev/shm
tmpfs tmpfs 240M 25M 215M 11% /run
tmpfs tmpfs 240M 0 240M 0% /sys/fs/cgroup
tmpfs tmpfs 240M 20K 240M 1% /var/volatile
/dev/mmcblk0p1 vfat 40M 8.3M 32M 21% /mnt/boot
/dev/mmcblk0p6 ext4 1.2G 1.2G 1.9M 100% /mnt/data
It was not due to the 2G SD card. I added an 8G and the same problem exists. The only [filtered] debugging I can see is this:
26.02.21 16:05:30 (+0000) Downloading image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
26.02.21 16:47:24 (+0000) Downloading delta for image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
26.02.21 16:55:20 (+0000) Failed to download image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7' due to '(HTTP code 404) no such image - no such image: registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a:delta-beb9e543ffa7cc0e: No such image: registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a:delta-beb9e543ffa7cc0e '
26.02.21 17:10:43 (+0000) Downloading delta for image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
26.02.21 17:32:19 (+0000) Downloading delta for image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
26.02.21 17:52:47 (+0000) Downloading delta for image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
26.02.21 18:19:38 (+0000) Downloading delta for image 'registry2.balena-cloud.com/v2/12aa67f7e4e13d318d48bbfdd1ec392a@sha256:d8e0d14421283ae69f33524addacde945c09959b51399081e0bd49665b9c8ef7'
At which point, as noted above, the download starts over again.
Made some progress, though a number of variables changed at the same time sooo ... here are my observations / actions:
All services are in a constant state of restarting, no idea why:
Restarting service 'dashboard sha256:8d36c94964aa50df8cac4aa56ca69becbd333fbab3f7320d20933a69773dfb72'
Service exited 'connector sha256:e5f06c9c72c47777851659d78654bdeff0aa77a6930f1c9bb5d86724eefb7a00'
Service exited 'dashboard sha256:8d36c94964aa50df8cac4aa56ca69becbd333fbab3f7320d20933a69773dfb72'
Restarting service 'speedtest sha256:3e2b380add093153b1e626be66e83e4d42c67c04a5ae1b982ac6ac7bbf4a252a'
Restarting service 'influxdb sha256:0866da0fd5bb684b2b9b13002956f1d9806bc8eac6d62a98bf1056d4268a25c6'
Service exited 'speedtest sha256:3e2b380add093153b1e626be66e83e4d42c67c04a5ae1b982ac6ac7bbf4a252a'
Restarting service 'connector sha256:e5f06c9c72c47777851659d78654bdeff0aa77a6930f1c9bb5d86724eefb7a00'
Restarting service 'dashboard sha256:8d36c94964aa50df8cac4aa56ca69becbd333fbab3f7320d20933a69773dfb72'
Service exited 'influxdb sha256:0866da0fd5bb684b2b9b13002956f1d9806bc8eac6d62a98bf1056d4268a25c6'
Restarting service 'influxdb sha256:0866da0fd5bb684b2b9b13002956f1d9806bc8eac6d62a98bf1056d4268a25c6'
Restarting service 'speedtest sha256:3e2b380add093153b1e626be66e83e4d42c67c04a5ae1b982ac6ac7bbf4a252a'
Service exited 'connector sha256:e5f06c9c72c47777851659d78654bdeff0aa77a6930f1c9bb5d86724eefb7a00'
Service exited 'dashboard sha256:8d36c94964aa50df8cac4aa56ca69becbd333fbab3f7320d20933a69773dfb72'
Service exited 'influxdb sha256:0866da0fd5bb684b2b9b13002956f1d9806bc8eac6d62a98bf1056d4268a25c6'
Service exited 'speedtest sha256:3e2b380add093153b1e626be66e83e4d42c67c04a5ae1b982ac6ac7bbf4a252a'
etc.
Any ideas?
Attempts to run 'device diagnostics' produces an error ssh client socket error while initiating SSH connection
.
I get this from journalctl:
Feb 26 23:36:10 641ce56 balenad[1398]: time="2021-02-26T23:36:10.341271934Z" level=error msg="Handler for GET /containers/a6881aba394191dce214eb9a749a6c13060c2d93bbbb545aded9405a63707c7d/json returned error: write unix /var/run/balena-engine.sock->@: write: broken pipe"
Feb 26 23:36:10 641ce56 balenad[1398]: time="2021-02-26T23:36:10.351433934Z" level=error msg="Handler for GET /v1.40/containers/influxdb_3329913_1711990/json returned error: write unix /var/run/balena-engine.sock->@: write: broken pipe"
That log shows output for influxdb, but it happens for all services.
From diagnostics Supervisor is running, but may be unhealthy
hmmm :thinking:
Hey @n1md4 - that doesn't sound like a healthy device, rather than it being specific to running this repo. I'd pop along to the balena forums and get some support on the device itself.
I have been meaning to try and run this up on a PiZero, but haven't got round to it yet.
Just to clarify here. The Dockerfiles on this project are all pinned to raspberrypi3
or arm32v6
, neither will run on a Pi Zero. So as it stands Pi Zero devices are not supported.
[tmigone] This issue has attached support thread https://jel.ly.fish/4ae32e2a-f6b3-4d54-a0d2-056335fe83b4
With the release of the PiZero2, I don't personally have the time or energy to add support for the PiZeroV1. PRs welcome, but I won't be working on this myself.
I downloaded this about a week or 10 days ago. It was working with errors on my pi zero w.
Now getting the following errors when I try to restart the speedtest container via the Balena interface:
speedtest ============================================================================== speedtest
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 21, in test
speedtest print ("Timestamp = " + str(result['timestamp']))
speedtest KeyError: 'timestamp'
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 21, in test
speedtest print ("Timestamp = " + str(result['timestamp']))
speedtest KeyError: 'timestamp'
speedtest [2020-12-05 01:29:39.598] [error] Configuration - Timeout was reached (TimeoutException)
speedtest [2020-12-05 01:29:39.617] [error] Configuration - Cannot retrieve configuration document (0)
speedtest [2020-12-05 01:29:39.627] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest [2020-12-05 01:29:39.639] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest {"type":"log","timestamp":"2020-12-05T01:29:39Z","message":"Configuration - Could not retrieve or read configuration (ConfigurationError)","level":"error"}
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 20, in test
speedtest result = json.loads(response[0])
speedtest File "/usr/local/lib/python3.7/json/init.py", line 348, in loads
speedtest return _default_decoder.decode(s)
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
speedtest obj, end = self.raw_decode(s, idx=_w(s, 0).end())
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
speedtest raise JSONDecodeError("Expecting value", s, err.value) from None
speedtest json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
speedtest [2020-12-05 01:29:39.598] [error] Configuration - Timeout was reached (TimeoutException)
speedtest [2020-12-05 01:29:39.617] [error] Configuration - Cannot retrieve configuration document (0)
speedtest [2020-12-05 01:29:39.627] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest [2020-12-05 01:29:39.639] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest {"type":"log","timestamp":"2020-12-05T01:29:39Z","message":"Configuration - Could not retrieve or read configuration (ConfigurationError)","level":"error"}
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 20, in test
speedtest result = json.loads(response[0])
speedtest File "/usr/local/lib/python3.7/json/init.py", line 348, in loads
speedtest return _default_decoder.decode(s)
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
speedtest obj, end = self.raw_decode(s, idx=_w(s, 0).end())
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
speedtest raise JSONDecodeError("Expecting value", s, err.value) from None
speedtest json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 21, in test
speedtest print ("Timestamp = " + str(result['timestamp']))
speedtest KeyError: 'timestamp'
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 21, in test
speedtest print ("Timestamp = " + str(result['timestamp']))
speedtest KeyError: 'timestamp'
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 21, in test
speedtest print ("Timestamp = " + str(result['timestamp']))
speedtest KeyError: 'timestamp'
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 21, in test
speedtest print ("Timestamp = " + str(result['timestamp']))
speedtest KeyError: 'timestamp'
speedtest [2020-12-05 01:31:38.818] [error] Configuration - Timeout was reached (TimeoutException)
speedtest [2020-12-05 01:31:38.834] [error] Configuration - Cannot retrieve configuration document (0)
speedtest [2020-12-05 01:31:38.845] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest [2020-12-05 01:31:38.853] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest {"type":"log","timestamp":"2020-12-05T01:31:38Z","message":"Configuration - Could not retrieve or read configuration (ConfigurationError)","level":"error"}
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 20, in test
speedtest result = json.loads(response[0])
speedtest File "/usr/local/lib/python3.7/json/init.py", line 348, in loads
speedtest return _default_decoder.decode(s)
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
speedtest obj, end = self.raw_decode(s, idx=_w(s, 0).end())
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
speedtest raise JSONDecodeError("Expecting value", s, err.value) from None
speedtest json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
speedtest [2020-12-05 01:31:38.818] [error] Configuration - Timeout was reached (TimeoutException)
speedtest [2020-12-05 01:31:38.834] [error] Configuration - Cannot retrieve configuration document (0)
speedtest [2020-12-05 01:31:38.845] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest [2020-12-05 01:31:38.853] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest {"type":"log","timestamp":"2020-12-05T01:31:38Z","message":"Configuration - Could not retrieve or read configuration (ConfigurationError)","level":"error"}
speedtest Traceback (most recent call last):
speedtest File "speedtest.py", line 34, in
speedtest result = speedtest.test()
speedtest File "speedtest.py", line 20, in test
speedtest result = json.loads(response[0])
speedtest File "/usr/local/lib/python3.7/json/init.py", line 348, in loads
speedtest return _default_decoder.decode(s)
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
speedtest obj, end = self.raw_decode(s, idx=_w(s, 0).end())
speedtest File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
speedtest raise JSONDecodeError("Expecting value", s, err.value) from None
speedtest json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
speedtest [2020-12-05 01:32:24.706] [error] Configuration - Timeout was reached (TimeoutException)
speedtest [2020-12-05 01:32:24.725] [error] Configuration - Cannot retrieve configuration document (0)
speedtest [2020-12-05 01:32:24.736] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
speedtest You may only use this Speedtest software and information generated speedtest from it for personal, non-commercial use, through a command line speedtest interface on a personal computer. Your use of this software is subject speedtest to the End User License Agreement, Terms of Use and Privacy Policy at speedtest these URLs: speedtest
speedtest https://www.speedtest.net/about/eula speedtest https://www.speedtest.net/about/terms speedtest https://www.speedtest.net/about/privacy speedtest
speedtest ============================================================================== speedtest
speedtest License acceptance recorded. Continuing. speedtest
speedtest Traceback (most recent call last): speedtest File "speedtest.py", line 34, in
Note. I found I had to change the speedtest Dockerfile to the following to be able to download an image that includes python
So my first question would be if the above python is the correct one I should be using.
Second question I have is I have noticed big gaps in the Grafana display of the data. I've got FREQUENCY set to 1800 (30 minutes) with the SERVER_ID variable set to the 1 server I want to test against and if I restart the container I will normally get 1 or 2 runs of the speedtest data collection script before it stops collecting for me and I have to restart it again.