krasi-georgiev
commented
5 years ago just tested that if I remove the : it downloads the image mainfest data:
Closed krasi-georgiev closed 5 years ago
krasi-georgiev
commented
5 years ago just tested that if I remove the : it downloads the image mainfest data:
pdcastro
commented
5 years ago It sounds like balenaEngine (the equivalent to the Docker daemon, running on a balena device -- I assume that 192.168.0.113 is a balena device) is getting "connection reset by peer" when it tries to download "external images". Can you confirm that either your docker-compose.yml file or your Dockerfiles are attempting to pull images from "cloudfare.docker.com"? Perhaps you could share the relevant lines from those files.
just tested that if I remove the
:it downloads the image manifest data
Does this mean that you have found the problem, and solved it? Or perhaps this is just additional debugging information.
If the problem persists, could it be a matter of authentication? Are those images being pulled from a private Dockerhub-like repository that requires a username and password to download from? If so, check the following docs and use the --registry-secrets command-line option:
krasi-georgiev
commented
5 years ago It sounds like balenaEngine (the equivalent to the Docker daemon, running on a balena device -- I assume that 192.168.0.113 is a balena device) is getting "connection reset by peer" when it tries to download "external images". Can you confirm that either your docker-compose.yml file or your Dockerfiles are attempting to pull images from "cloudfare.docker.com"? Perhaps you could share the relevant lines from those files.
Here is the full compose file. You can simplify it even to a single service and it still throws the same error. https://github.com/arribada/SMARTLoraTracker/blob/b3679f0bd1bb7ae487f6fac669ad446b023342c1/receiver/docker-compose.yml
just tested that if I remove the : it downloads the image manifest data
Does this mean that you have found the problem, and solved it? Or perhaps this is just additional debugging information.
this is just additional debug info. It seems to me that balena cli is missing some url sanitization and it is sending the requests to the wrong url.
If the problem persists, could it be a matter of authentication? Are those images being pulled from a private Dockerhub-like repository that requires a username and password to download from?
nope these are all public images.
pdcastro
commented
5 years ago @krasi-georgiev, thanks for sharing the docker-compose.yml file. I have tried using it with balena-cli 11.11.2 (standalone zip package) on Ubuntu 18.04.3, pushing the project to a Raspberry Pi 3 running balenaOS 2.38.0+rev1, and the arribada/* images were downloaded without errors:
DEBUG=1 /mnt/data/cli/balena-cli-11.11.2/balena push 192.168.0.20 --env DEBUG=1
[debug] original argv0="/mnt/data/cli/balena-cli-11.11.2/balena" argv=[/mnt/data/cli/balena-cli-11.11.2/balena,/snapshot/versioned-source/bin/balena,push,192.168.0.20,--env,DEBUG=1] length=6
[debug] Using . as build source
[Debug] Checking we can access device
[Debug] Sending request to http://192.168.0.20:48484/ping
[Debug] Checking device version: 9.15.7
[Info] Starting build on device 192.168.0.20
[Debug] Loading project...
[Debug] Resolving project...
[Info] Compose file detected
[Debug] Creating project...
[Debug] Tarring all non-ignored files...
[Debug] Sending request to http://192.168.0.20:48484/v2/local/device-info
[Debug] Found build tasks:
[Debug] packet-forwarder: image pull [arribada/packet-forwarder]
[Debug] loraserver: image pull [arribada/loraserver]
[Debug] Resolving services with [raspberrypi3|armv7hf]
[Debug] Found project types:
[Debug] packet-forwarder: External image
[Debug] loraserver: External image
[Debug] Probing remote daemon for cache images
[Debug] Using 11 on-device images for cache...
[Debug] Starting builds...
[Build] [loraserver] [> ] 90.15kB/8.822MB
...
[Build] [loraserver] [==================================================>] 8.822MB/8.822MB
[Build] [packet-forwarder] [> ] 473.8kB/45.92MB
...
[Build] [packet-forwarder] [==================================================>] 45.92MB/45.92MB
[Debug] Setting device state...
[Debug] Sending request to http://192.168.0.20:48484/v2/local/target-state
[Debug] Sending target state: {"local":{"name":"pi-sky","config":{"HOST_CONFIG_disable_splash":"1","HOST_CONFIG_dtparam":"\"i2c_arm=on\",\"spi=on\",\"audio=on\"","HOST_CONFIG_enable_uart":"1","HOST_CONFIG_gpu_mem":"192","SUPERVISOR_LOCAL_MODE":"1","SUPERVISOR_POLL_INTERVAL":"900000","SUPERVISOR_VPN_CONTROL":"true","SUPERVISOR_INSTANT_UPDATE_TRIGGER":"true","SUPERVISOR_CONNECTIVITY_CHECK":"true","SUPERVISOR_LOG_CONTROL":"true","SUPERVISOR_DELTA":"false","SUPERVISOR_DELTA_REQUEST_TIMEOUT":"30000","SUPERVISOR_DELTA_APPLY_TIMEOUT":"0","SUPERVISOR_DELTA_RETRY_COUNT":"30","SUPERVISOR_DELTA_RETRY_INTERVAL":"10000","SUPERVISOR_DELTA_VERSION":"2","SUPERVISOR_OVERRIDE_LOCK":"false","SUPERVISOR_PERSISTENT_LOGGING":"false"},"apps":{"1":{"name":"localapp","commit":"localrelease","releaseId":"1","services":{"1":{"environment":{"CONCENTRATOR_RESET_PIN":"22","CONCENTRATOR_CONFIG":"{\n \"SX1301_conf\": {\n \"lorawan_public\": true,\n \"clksrc\": 1,\n \"antenna_gain\": 0,\n \"radio_0\": {\n \"enable\": true,\n \"type\": \"SX1257\",\n \"freq\": 867500000,\n \"rssi_offset\": -166.0,\n \"tx_enable\": true,\n \"tx_freq_min\": 863000000,\n \"tx_freq_max\": 870000000\n },\n \"radio_1\": {\n \"enable\": true,\n \"type\": \"SX1257\",\n \"freq\": 868500000,\n \"rssi_offset\": -166.0,\n \"tx_enable\": false\n },\n \"chan_multiSF_0\": {\n \"enable\": true,\n \"radio\": 1,\n \"if\": -400000\n },\n \"chan_multiSF_1\": {\n \"enable\": true,\n \"radio\": 1,\n \"if\": -200000\n },\n \"chan_multiSF_2\": {\n \"enable\": true,\n \"radio\": 1,\n \"if\": 0\n },\n \"chan_multiSF_3\": {\n \"enable\": true,\n \"radio\": 0,\n \"if\": -400000\n },\n \"chan_multiSF_4\": {\n \"enable\": true,\n \"radio\": 0,\n \"if\": -200000\n },\n \"chan_multiSF_5\": {\n \"enable\": true,\n \"radio\": 0,\n \"if\": 0\n },\n \"chan_multiSF_6\": {\n \"enable\": true,\n \"radio\": 0,\n \"if\": 200000\n },\n \"chan_multiSF_7\": {\n \"enable\": true,\n \"radio\": 0,\n \"if\": 400000\n },\n \"chan_Lora_std\": {\n \"enable\": true,\n \"radio\": 1,\n \"if\": -200000,\n \"bandwidth\": 250000,\n \"spread_factor\": 7\n },\n \"chan_FSK\": {\n \"enable\": true,\n \"radio\": 1,\n \"if\": 300000,\n \"bandwidth\": 125000,\n \"datarate\": 50000\n },\n \"tx_lut_0\": {\n \"pa_gain\": 0,\n \"mix_gain\": 8,\n \"rf_power\": -6,\n \"dig_gain\": 0\n },\n \"tx_lut_1\": {\n \"pa_gain\": 0,\n \"mix_gain\": 10,\n \"rf_power\": -3,\n \"dig_gain\": 0\n },\n \"tx_lut_2\": {\n \"pa_gain\": 0,\n \"mix_gain\": 12,\n \"rf_power\": 0,\n \"dig_gain\": 0\n },\n \"tx_lut_3\": {\n \"pa_gain\": 1,\n \"mix_gain\": 8,\n \"rf_power\": 3,\n \"dig_gain\": 0\n },\n \"tx_lut_4\": {\n \"pa_gain\": 1,\n \"mix_gain\": 10,\n \"rf_power\": 6,\n \"dig_gain\": 0\n },\n \"tx_lut_5\": {\n \"pa_gain\": 1,\n \"mix_gain\": 12,\n \"rf_power\": 10,\n \"dig_gain\": 0\n },\n \"tx_lut_6\": {\n \"pa_gain\": 1,\n \"mix_gain\": 13,\n \"rf_power\": 11,\n \"dig_gain\": 0\n },\n \"tx_lut_7\": {\n \"pa_gain\": 2,\n \"mix_gain\": 9,\n \"rf_power\": 12,\n \"dig_gain\": 0\n },\n \"tx_lut_8\": {\n \"pa_gain\": 1,\n \"mix_gain\": 15,\n \"rf_power\": 13,\n \"dig_gain\": 0\n },\n \"tx_lut_9\": {\n \"pa_gain\": 2,\n \"mix_gain\": 10,\n \"rf_power\": 14,\n \"dig_gain\": 0\n },\n \"tx_lut_10\": {\n \"pa_gain\": 2,\n \"mix_gain\": 11,\n \"rf_power\": 16,\n \"dig_gain\": 0\n },\n \"tx_lut_11\": {\n \"pa_gain\": 3,\n \"mix_gain\": 9,\n \"rf_power\": 20,\n \"dig_gain\": 0\n }\n },\n\n \"gateway_conf\": {\n \"gateway_ID\": \"0242acfffe110006\",\n \"server_address\": \"gatewaybridge\",\n \"serv_port_up\": 1700,\n \"serv_port_down\": 1700,\n \"keepalive_interval\": 10,\n \"stat_interval\": 30,\n \"push_timeout_ms\": 100,\n \"forward_crc_valid\": true,\n \"forward_crc_error\": false,\n \"forward_crc_disabled\": false,\n #\"gps_tty_path\": \"/dev/ttyAMA0\",\n \"beacon_period\": 128,\n \"beacon_freq_hz\": 869525000,\n \"beacon_datarate\": 9,\n \"beacon_bw_hz\": 125000,\n \"beacon_power\": 14,\n \"beacon_infodesc\": 0\n }\n}","DEBUG":"1"},"labels":{},"privileged":true,"imageId":1,"serviceName":"packet-forwarder","serviceId":1,"image":"local_image_packet-forwarder:latest","running":true},"2":{"environment":{"POSTGRESQL.DSN":"postgres://loraserver_ns:loraserver_ns@postgresql/loraserver_ns?sslmode=disable","REDIS.URL":"redis://redis:6379","NETWORK_SERVER.BAND.NAME":"EU_863_870","NETWORK_SERVER.GATEWAY.BACKEND.MQTT.SERVER":"tcp://mosquitto:1883","JOIN_SERVER.DEFAULT.SERVER":"http://appserver:8003","GEOLOCATION_SERVER.SERVER":"geoserver:8005","DEBUG":"1"},"labels":{},"imageId":2,"serviceName":"loraserver","serviceId":2,"image":"local_image_loraserver:latest","running":true}},"volumes":{"smart-postgres":{},"smart-connect":{},"lora-postgres":{},"lora-redis":{},"configs":{}},"networks":{}}}},"dependent":{"apps":[],"devices":[]}}
[Debug] Sending request to http://192.168.0.20:48484/v2/local/target-state
[Debug] Sending request to http://192.168.0.20:48484/v2/local/device-info
[Live] Waiting for device state to settle...
[Info] Streaming device logs...
[Live] Watching for file changes...
[Debug] Device state not settled, retrying in 1000ms
[Logs] [9/23/2019, 3:56:17 PM] Installing service 'packet-forwarder sha256:5f2b15d4533c3a60c14bed2ff516fc14e8759fb44bd607868b8df95620266c4b'
[Logs] [9/23/2019, 3:56:18 PM] Installed service 'packet-forwarder sha256:5f2b15d4533c3a60c14bed2ff516fc14e8759fb44bd607868b8df95620266c4b'
[Logs] [9/23/2019, 3:56:18 PM] Starting service 'packet-forwarder sha256:5f2b15d4533c3a60c14bed2ff516fc14e8759fb44bd607868b8df95620266c4b'
[Logs] [9/23/2019, 3:56:22 PM] Started service 'packet-forwarder sha256:5f2b15d4533c3a60c14bed2ff516fc14e8759fb44bd607868b8df95620266c4b'
[Logs] [9/23/2019, 3:56:22 PM] [packet-forwarder] Replacing the default global conf with a user defined config from the CONCENTRATOR_CONFIG env
[Logs] [9/23/2019, 3:56:22 PM] [packet-forwarder] Starting the CONCENTRATOR by reseting pin 22
[Debug] Device state not settled, retrying in 1000ms
[Logs] [9/23/2019, 3:56:22 PM] [packet-forwarder] Accessing concentrator reset pin through GPIO22...
...(For the output above, I commented out all services except for loraserver and packet-forwarder. The loraserver service eventually failed with BadRequestDeviceAPIError: Invalid apps, but the point is that there were no errors downloading the image.)
I.e., I could not reproduce the connection reset by peer errors. Could it be a network issue on the device's local network? Even perhaps firewall config? Are you also using a Raspberry Pi 3? Which version of balenaOS is it running? What else might I need to do to reproduce the issue?
krasi-georgiev
commented
5 years ago yes this is intermittent issue and trying the same thing today runs without the above error.
krasi-georgiev
commented
5 years ago I will reopen if it happens again
balena version 11.11.2
Linux Ubuntu installed from zip
let me know if you need anymore info