Support `**` glob copying of files into image layer

[kriswuollett]

Add support for ** globs in the Dockerfile to copy files into the image that preserves the file tree. This issue appears often in various forums but I did not see an open nor closed issue for this in this repo.

Overview

The change I'd like to see is one of the following options:

• COPY command directly supports ** globs that preserves the file tree
• Add new rsync-like command that supports a subset of rsync functionality for this use case

Context

The Dockerfile and the build tools.

Use Case

Building a container image from the root of a monorepo and selecting a subset of files more intelligently than using a combination of .dockerignore and writing multiple lines like the following multiple times in a multi-stage node build:

# apps/my-app/Dockerfile.dockerignore
*
!lib/**
!apps/**

# apps/my-app/Dockerfile
# Fetch all dependencies first and cache them since they do not change often
COPY lib/my-package-a/package.json ./lib/my-package-a/
COPY lib/my-package-a/pnpm-lock.yaml* ./lib/my-package-a/
COPY lib/my-package-b/package.json ./lib/my-package-b/
COPY lib/my-package-b/pnpm-lock.yaml* ./lib/my-package-b/
COPY lib/my-package-c/package.json ./lib/my-package-c/
COPY lib/my-package-c/pnpm-lock.yaml* ./lib/my-package-c/

followed by:

# apps/my-app/Dockerfile
RUN pnpm i --frozen-lockfile

# ...

RUN npx turbo build --scope=my-app

Current Workaround

Copy and paste content of Dockerfile for each library that the app depends on.

Additional context

Mentions of the issue:

• https://forums.docker.com/t/glob-style-copy-in-dockerfiles/82472
• https://stackoverflow.com/questions/49939960/docker-copy-files-using-glob-pattern

[thaJeztah]

There's a pull request currently being reviewed in BuildKit that implements a --parents option; not exactly the same as what's proposed here, but can address some of these use-cases; https://github.com/moby/buildkit/pull/3001

[nicks]

I think we have to be careful to separate out: 1) A COPY syntax that preserves file trees at the destination 2) A pattern matching syntax that recursively selects stuff from large source file trees / monorepos

The first one totally makes sense. The second one sounds like it's a good idea, but historically has led to a ton of unintuitive performance issues and weird gotchas in other build systems. (e.g., I know the Bazel people put strict limits on ** to make it viable in monorepos)

[kriswuollett]

The request can be interpreted as an implementation or syntax option. However the underlying issue is selectively bulk copying files, which can happen a lot in various languages for things like lockfiles when desiring to cache as much layers as possible... And not having to change the Dockerfile whenever library dependencies change.

Other possibilities could include COPY preserving directory structure but specify files in an external file that has them in a list. That may be a simpler thing to do than code generating a Dockerfile which is the workaround for https://github.com/moby/moby/issues/735.