live push tars all children rather than resolving what is necessary for build

[xginn8]

• balena CLI version: 11.9.0
• Operating system version: Arch Linux

When livepushing, the tarring step of the build (https://github.com/balena-io/balena-cli/blob/master/lib/utils/device/deploy.ts#L180) tars every file from the root directory, which includes many files that are not necessarily needed in the build. In my case (and in the case of typical BoB workflows), this includes many working copies of various backend services that are not actively being built. This tarring action is resource intensive and time consuming. Below is an example src directory within BoB used to push edits to various services:

~/balena/balena-on-balena master*
❯ tree src
src
├── api -> ../../api
├── builder -> ../../builder
├── device-monitor -> ../../device-monitor
├── monitor -> ../../balena-monitor
├── node-exporter -> ../../balena-node-exporter
├── proxy -> ../../proxy
├── ui -> ../../ui
└── vpn -> ../../vpn

In effect, this src folder exists as a local copy for when I want to build from source rather than use an image, and therefore can't be either .gitignored or .dockerignored. At the same time, most of these services are not built from source most of the time.

cc @hedss

This issue is related to https://github.com/balena-io/balena-cli/issues/1396

[pdcastro]

Thinking out loud, if automatically deciding what folders are "needed in the build" is a difficult problem, perhaps include / exclude patterns could be used as push command-line options, like rsync does.

Edit: Starting with CLI v12.28.0, livepush respects .dockerignore files (see balena help push for more details on how to use .dockerignore files).

[pdcastro]

this src folder exists as a local copy for when I want to build from source rather than use an image, and therefore can't be either .gitignored or .dockerignored

@xginn8, maybe .dockerignore can still help (livepush respects .dockerginore files starting with CLI v12.28.0.). I think I know what you mean about BoB and the src folder -- I have found myself adding a symlink to a clone of balena-builder when I was testing changes to the builder source code in the past.

We normally use the --multi-dockerignore (-m) option with BoB:

$ balena help push
...
  The --multi-dockerignore (-m) option may be used with microservices
  (multicontainer) applications that define a docker-compose.yml file. When
  this option is used, each service subdirectory (defined by the `build` or
  `build.context` service properties in the docker-compose.yml file) is
  filtered separately according to a .dockerignore file defined in the service
  subdirectory. If no .dockerignore file exists in a service subdirectory, then
  only the default .dockerignore patterns (see below) apply for that service
  subdirectory.

  When the --multi-dockerignore (-m) option is used, the .dockerignore file (if
  any) defined at the overall project root will be used to filter files and
  subdirectories other than service subdirectories. It will not have any effect
  on service subdirectories, whether or not a service subdirectory defines its
  own .dockerignore file. Multiple .dockerignore files are not merged or added
  together, and cannot override or extend other files. This behavior maximises
  compatibility with the standard docker-compose tool, while still allowing a
  root .dockerignore file (at the overall project root) to filter files and
  folders that are outside service subdirectories.

So I am thinking that, while using -m, a .dockerignore file at the project root (BoB repo root folder) could list the src folder:

.dockerignore file at the project root (BoB repo root folder):

src/**

This would mean that the src folder is in principle ignored, but subfolders of src would not be ignored if they are listed in the build or build.context service properties of the docker-compose.yml file. This is because the .dockerignore file at the BoB repo root folder does not apply to service folders listed in docker-compose.yml (they use their own service-specific .dockerignore files, or only the default patterns if a service-specific .dockerignore file does not exist -- see full balena help push).

This would not even require editing the .dockerignore files when you decide to switch between using images or building from source.

I have not tested it... But in my head it seems to work. :-)