balena-io / balena-cli

The official balena CLI tool.
Apache License 2.0
455 stars 141 forks source link

Failed to preload image on openBalena #2068

Closed openedhardware closed 4 years ago

openedhardware commented 4 years ago

Specifications

I was able to deploy an application by using balena deploy command.

But cannot preload the deployed application to an img file:

ubuntu@ubuntu-desktop:/media$ balena apps
ID APP NAME  SLUG      DEVICE TYPE  ONLINE DEVICES DEVICE COUNT
1  yokosoApp yokosoapp raspberry-pi 0              0
ubuntu@ubuntu-desktop:/media$ balena preload ../balena-rpi-configured.img --app 1 --commit current --splash-image splash.jpg --debug
Building Docker preloader image. [===                     ] 12%
Step 1/7 : FROM docker:17.12.0-ce-dind
Building Docker preloader image. [======                  ] 25%
Step 2/7 : RUN apk update && apk add --no-cache python3 parted btrfs-progs util-linux sfdisk file coreutils sgdisk
 ---> Using cache
Building Docker preloader image. [=========               ] 37%
Step 3/7 : COPY ./requirements.txt /tmp/
 ---> Using cache
Building Docker preloader image. [============            ] 50%
Step 4/7 : RUN pip3 install -r /tmp/requirements.txt
 ---> Using cache
Building Docker preloader image. [===============         ] 62%
Step 5/7 : COPY ./src /usr/src/app
 ---> Using cache
Building Docker preloader image. [==================      ] 75%
Step 6/7 : WORKDIR /usr/src/app
 ---> Using cache
Building Docker preloader image. [=====================   ] 87%
Step 7/7 : CMD ["python3", "/usr/src/app/preload.py"]
 ---> Using cache
 ---> 95c91d1abee6
Successfully built 95c91d1abee6
Building Docker preloader image. [========================] 100%
| Checking that the image is a writable file
| Finding a free tcp port and getting balena settings
| Checking if the image is an edison zip archive
| Creating preloader container
| Starting preloader container
- Fetching application 1
- Cleaning up temporary files
BalenaRequestError: Request error: <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v6/application(1)</pre>
</body>
</html>

BalenaRequestError: Request error: <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot GET /v6/application(1)</pre>
</body>
</html>

    at /snapshot/versioned-source/node_modules/balena-request/build/request.js:189:17
From previous event:
    at /snapshot/versioned-source/node_modules/balena-request/build/request.js:181:62
From previous event:
    at Object.module.exports.exports.send (/snapshot/versioned-source/node_modules/balena-request/build/request.js:180:8)
    at /snapshot/versioned-source/node_modules/balena-pine/es2018/index.js:20:22
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
From previous event:
    at BalenaPine._request (/snapshot/versioned-source/node_modules/balena-pine/es2018/index.js:12:34)
    at BalenaPine.request (/snapshot/versioned-source/node_modules/pinejs-client-core/es2018/index.js:952:29)
    at BalenaPine.get (/snapshot/versioned-source/node_modules/pinejs-client-core/es2018/index.js:701:21)
    at /snapshot/versioned-source/node_modules/balena-preload/node_modules/balena-sdk/es2018/models/application.js:245:26
From previous event:
    at Object.get (/snapshot/versioned-source/node_modules/balena-preload/node_modules/balena-sdk/es2018/models/application.js:239:32)
    at Object.get (/snapshot/versioned-source/node_modules/balena-preload/node_modules/balena-sdk/es2018/util/callbacks.js:19:27)
    at _fetchApplication._runWithSpinner.Promise.try (/snapshot/versioned-source/node_modules/balena-preload/lib/preload.js:487:49)
From previous event:
    at _fetchApplication._runWithSpinner (/snapshot/versioned-source/node_modules/balena-preload/lib/preload.js:485:25)
    at Preloader._runWithSpinner (/snapshot/versioned-source/node_modules/balena-preload/lib/preload.js:305:12)
    at Preloader._fetchApplication (/snapshot/versioned-source/node_modules/balena-preload/lib/preload.js:481:17)
    at prepare._build.then.then.spread.then.then.then.then.then.then.then.then (/snapshot/versioned-source/node_modules/balena-preload/lib/preload.js:628:19)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:126:23)
From previous event:
    at Preloader.prepare (/snapshot/versioned-source/node_modules/balena-preload/lib/preload.js:627:6)
    at prepareAndPreload (/snapshot/versioned-source/build/actions/preload.js:173:21)
    at /snapshot/versioned-source/build/actions/preload.js:347:25
    at new Promise (<anonymous>)
    at Command.action (/snapshot/versioned-source/build/actions/preload.js:345:19)

Additional information may be available with the `--debug` flag.
For help, visit our support forums: https://forums.balena.io
For bug reports or feature requests, see: https://github.com/balena-io/balena-cli/issues/

ubuntu@ubuntu-desktop:/media$ balena version -a
balena-cli version "12.11.0"
Node.js version "10.21.0"
pdcastro commented 4 years ago

Thank you for reporting this issue and sharing all the details - the details are very helpful. I think you may be coming across some outstanding issues:

openedhardware commented 4 years ago

@pdcastro

Hmm, downgraded to 12.2.2 and tried again:

ubuntu@ubuntu-desktop:/media$ balena preload ../balena-rpi-configured.img --app yokosoApp --commit current --splash-image splash.jpg --debug
Building Docker preloader image. [===                     ] 12%
Step 1/7 : FROM docker:17.12.0-ce-dind
Building Docker preloader image. [======                  ] 25%
Step 2/7 : RUN apk update && apk add --no-cache python3 parted btrfs-progs util-linux sfdisk file coreutils sgdisk
 ---> Using cache
Building Docker preloader image. [=========               ] 37%
Step 3/7 : COPY ./requirements.txt /tmp/
 ---> Using cache
Building Docker preloader image. [============            ] 50%
Step 4/7 : RUN pip3 install -r /tmp/requirements.txt
 ---> Using cache
Building Docker preloader image. [===============         ] 62%
Step 5/7 : COPY ./src /usr/src/app
 ---> Using cache
Building Docker preloader image. [==================      ] 75%
Step 6/7 : WORKDIR /usr/src/app
 ---> Using cache
Building Docker preloader image. [=====================   ] 87%
Step 7/7 : CMD ["python3", "/usr/src/app/preload.py"]
 ---> Using cache
 ---> c0c061732ecc
Successfully built c0c061732ecc
Building Docker preloader image. [========================] 100%
| Checking that the image is a writable file
| Finding a free tcp port and getting balena settings
| Checking if the image is an edison zip archive
/ Creating preloader container
- Starting preloader container
\ Fetching application yokosoApp
\ Reading image informationWaiting for Docker to start...
- Reading image informationDocker started
\ Reading image information
- Resizing partitions and waiting for dockerd to startReplacing splash image
/ Resizing partitions and waiting for dockerd to startExpanding extended partition n°4 of /img/balena.img
Expanding logical partition n°6 of /img/balena.img
| Resizing partitions and waiting for dockerd to startResizing ext4 filesystem of partition n°6 of /img/balena.img using /dev/loop9
/ Resizing partitions and waiting for dockerd to startFile system OK
| Resizing partitions and waiting for dockerd to startWaiting for Docker to start...
\ Resizing partitions and waiting for dockerd to startDocker started

\ Cleaning up temporary files
(HTTP code 500) server error - Get https://registry.yokosotv.com/v2/: x509: certificate signed by unknown authority

Error: (HTTP code 500) server error - Get https://registry.yokosotv.com/v2/: x509: certificate signed by unknown authority
    at /snapshot/versioned-source/node_modules/dockerode/node_modules/docker-modem/lib/modem.js:257:17
    at IncomingMessage.<anonymous> (/snapshot/versioned-source/node_modules/dockerode/node_modules/docker-modem/lib/modem.js:284:9)
    at IncomingMessage.emit (events.js:203:15)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at endReadableNT (_stream_readable.js:1143:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)

Additional information may be available with the `--debug` flag.
For help, visit our support forums: https://forums.balena.io
For bug reports or feature requests, see: https://github.com/balena-io/balena-cli/issues/

root@ubuntu-desktop:/media# balena apps
ID APP NAME  SLUG      DEVICE TYPE  ONLINE DEVICES DEVICE COUNT
1  yokosoApp yokosoapp raspberry-pi 0              0
root@ubuntu-desktop:/media# balena version -a
balena-cli version "12.2.2"
Node.js version "10.17.0"

Seems certification issue, but I was able to login and get app info via balena apps command?

And why does it say that 10.17.0 is installed on my PC?

root@ubuntu-desktop:/media# node -v
v12.19.0

Thanks!

openedhardware commented 4 years ago

Found another issue on v12.2.2:

ubuntu@ubuntu-desktop:~$ sudo balena scan
Reporting scan results
-
  host:          yokoso.local
  address:       192.168.1.115
  dockerInfo:
    Containers:        4
    ContainersRunning: 2
    ContainersPaused:  0
    ContainersStopped: 2
    Images:            87
    Driver:            aufs
    SystemTime:        2020-10-20T11:38:43.173895986Z
    KernelVersion:     4.19.118
    OperatingSystem:   balenaOS 2.54.2+rev1
    Architecture:      armv6l
  dockerVersion:
    Version:    19.03.13-dev
    ApiVersion: 1.40
ubuntu@ubuntu-desktop:~$ sudo balena ssh 192.168.1.115 --debug
[debug] new argv=[/home/ubuntu/balena-cli/balena,/snapshot/versioned-source/bin/balena,ssh,192.168.1.115] length=4
Part of the CLI could not be loaded. This typically means your CLI install is in a broken state.
You can normally fix this by uninstalling and reinstalling the CLI.

Error: Cannot find module 'lodash/reduce'
1) If you want to compile the package/file into executable, please pay attention to compilation warnings and specify a literal in 'require' call. 2) If you don't want to compile the package/file into executable and want to 'require' it from filesystem (likely plugin), specify an absolute path in 'require' call using process.cwd() or process.execPath.
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:634:15)
    at Function.Module._resolveFilename (pkg/prelude/bootstrap.js:1351:46)
    at Function.resolveFilenameOptimized [as _resolveFilename] (/snapshot/versioned-source/node_modules/fast-boot2/index.js:57:33)
    at Function.Module._load (internal/modules/cjs/loader.js:560:25)
    at Module.require (internal/modules/cjs/loader.js:690:17)
    at Module.require (pkg/prelude/bootstrap.js:1230:31)
    at require (internal/modules/cjs/helpers.js:25:18)
    at Promise.resolve.then (/snapshot/versioned-source/build/utils/device/ssh.js:7:55)

Additional information may be available with the `--debug` flag.
For help, visit our support forums: https://forums.balena.io
For bug reports or feature requests, see: https://github.com/balena-io/balena-cli/issues/

ubuntu@ubuntu-desktop:~$ balena version -a
balena-cli version "12.2.2"
Node.js version "10.17.0"

Any idea?

This is frustrating... :/

pdcastro commented 4 years ago

(HTTP code 500) server error - Get https://registry.yokosotv.com/v2/: x509: certificate signed by unknown authority

I have myself just learned about the --add-certificate option for use with openBalena (hence not advising it earlier, sorry):

$ balena help preload
...
--add-certificate <add-certificate>
  Add the given certificate (in PEM format) to /etc/ssl/certs in the preloading container.
  The file name must end with '.crt' and must not be already contained in the preloader's
  /etc/ssl/certs folder. Can be repeated to add multiple certificates.

I gather that the ca.crt file mentioned in openBalena's getting started guide would be the one to be passed as an argument to the --add-certificate option of the balena preload command, but it may be safer to rename it something like open-balena.crt given the help advice above. See also:

And why does it say that 10.17.0 is installed on my PC?


root@ubuntu-desktop:/media# balena version -a
balena-cli version "12.2.2"
Node.js version "10.17.0"

root@ubuntu-desktop:/media# node -v v12.19.0


The CLI's standalone zip packages include Node.js built-in in the CLI executable, so it runs even if there is no Node.js installed on the PC. The Node.js version reported by `balena version -a` is the built-in version, which is the one used by the CLI. An installation of Node.js on the PC is only required for the advanced (not recommended) `npm` installation option ([install instructions](https://github.com/balena-io/balena-cli/blob/master/INSTALL.md)).

root@ubuntu-desktop:/media# balena version -a balena-cli version "12.2.2" Node.js version "10.17.0"

ubuntu@ubuntu-desktop:~$ sudo balena ssh 192.168.1.115 --debug [...] Part of the CLI could not be loaded. This typically means your CLI install is in a broken state.


Ah, I was not aware of this one. :-/ I have now checked that this `ssh` issue affects the standalone zip packages of CLI versions v12.1.15 to v12.3.1. It is possible to have multiple versions of the CLI (standalone zip package) downloaded to different folders, and run the ssh command with a CLI version other than v12.2.2, for example v12.1.14, v12.3.2 or later. By the way, for development devices on the local network, you can also use the `ssh` tool directly (instead of `balena ssh`) with a command line like:

ssh -p 22222 root@192.168.1.115



This works with devices running a development image of balenaOS. See [development vs production images](https://www.balena.io/docs/learn/welcome/production-plan/#development-images)

> This is frustrating... :/

Yes, it is, and I apologise. It is not the experience we want for openBalena users. openBalena is still labeled Beta in the Getting Started guide, and we are still working on the process of maintaining both openBalena and balenaCloud in sync. We have made progress by having balenaCloud build on top of openBalena, "dogfooding" as they say. But I'd say we had a misstep when we the balena SDK v14 added support for new balena API features (`/v6` HTTP endpoints made available in balenaCloud first) in a backwards-incompatible way, and the CLI adopted the new SDK without realising the impact on openBalena. The short-term fix, which we are working on now, is to update openBalena to support the incompatible balena SDK / API features, so the latest balenaCLI will work with openBalena as well. Long term, to prevent this happening again (while still allowing balenaCloud to add selected new features that may not be present in openBalena), we are discussing and prototyping a "product compiler" approach that may see separate branches of the balena SDK and CLI being automatically generated (and tested) for balenaCloud and openBalena, therefore having separate CLI releases for download for openBalena and balenaCloud. This is still under discussion and if we do go down this route, automation would be essential as we would not want to "manually" maintain multiple branches of SDK and CLI. In summary, we recognise the issues and frustration, and we are actively working towards both short-term and long-term solutions.
openedhardware commented 4 years ago

@pdcastro

Yeah, 11.7.10 works with --add-certificate param!

But 12.11.0 and 12.2.2 don't work though...

Thanks a lot! 🎉