Include layers / volumes

[RickMoynihan]

Just a first stab on what this might look like, and the smallest thing to get me over the lack of such a feature.

Not sure what else you might have had in mind, perhaps some convenience syntaxes, for single file copying?

Currently it looks something like this:

               :include {"/some/dest-dir" ["./some/source-file" "./some-other/source-file-2"]}

[SevereOverfl0w]

This is a pragmatic solution, I think it's a good way for people who need this to get going.

My outstanding considerations for a stable API are:

• Should include layers run before or after the dependency/source layers? Or perhaps between dependencies and sources?
• Are there performance considerations to the ordering of layers? e.g. if you have files which rarely change, vs files which regularly change you might be able to better take advantage of caching
• Using a FileEntriesLayer, you can include multiple files in a single layer, is support needed for that? (Again, from a caching perspective).
• Do the order of layers matter, e.g. if you have {"/tmp/foo" ["/some/dir"] "/tmp/foo/bar" ["/some/file"]} does it work only if /tmp/foo is added prior to /tmp/foo/bar?

I haven't given this much thought, but one way to allow the user control over some of these decisions would be to pull out the dependency/source layers into functions that the user can supply when providing custom layers. If :layers is not provided, it would behave as it does currently.

:layers
;; any ordered collection
[??? ;; layer of rarely changing files, e.g. a binary file
 (pack/dependency-layer basis)
 ??? ;; layer of semi-regularly changing files, e.g. a config file
 (pack/source-files-layer basis)
 ??? ;; layer of always changing files, e.g. a build manifest containing a timestamp
]

Bonus points if there's a clear & safe way to support assoc/merge (or custom variants of) to layers. Assuming this actually makes sense around caching/layer sizes (it may not, it might be fine to just put a new layer after the existing one).

:layers
[
 ;; pack/merge might be necessary to enforce constraints around ordering/conflicts/etc.
 (pack/merge (pack/dependency-layer basis) (jars-provided-by-vendor))
 ;; preferred: using clojure built-ins because it's all just data
 (conj (pack/source-files-layer basis) (configs-layer))
]

We could also take some hints from tools.build and support convenience keywords:

:layers [:default (my-custom-layer)]
:layers [:dependency (my-custom-layer) :source]

[SevereOverfl0w]

A couple of things I didn't consider:

1. Ownership, permissions and modification times. These are particularly important for users wanting reproducibility.

2. Directories. These are a little annoying to work with if you only do files, where your proposed API is very convenient. Jib itself has a "addEntryRecursive", which takes callbacks for deriving the metadata if you want to override it. The Clojure-y solution isn't too bad, but I'd want to figure out how the destination path could be set cleanly. I don't love what I came up with here.

   (map
    (fn [f]
      [(.getPath src)
       ;; doing path resolution, relatively?
       ["/some/absolute/path"
        ;; relative:
        (.getPath src)]
   
       {:modification-time :inherit
        :ownership :inherit
        :permissions {:owner #{:read}}}])
    (tree-seq (io/file "foo")))

   I don't think we need to go with callbacks (yuk!) though. This is a pretty convenient API:

   (map
    (fn [[src dest attributes]]
      [src dest (assoc attributes :modification-time (Instant/now))])
    (directory "from" "/absolute/to"))  ;; (directory) just a shorthand for the above `file-seq` code.

This all is reminiscent of the vfs code.

[SevereOverfl0w]

I've done minimal testing but dominic/jib-layers2 provides an alternative API:

:layers [:libs :paths]
:layers [:libs {:entries (pack/copy-layer "/configs/" "./configs/prod.edn") :name "Config"} :paths]

The latter is an attempt at producing something equivalently convenient to the proposed API in this PR.

@RickMoynihan could you give my branch a spin and see if it works for your use-cases? Disclaimer: I've not actually run any of the code yet, so it might be horribly broken!

[SevereOverfl0w]

Fyi, I've cherry-picked the :volumes and :env support in.

[SevereOverfl0w]

My branch's API use looks like this for single files:

clj -X juxt.pack.cli.api/docker ... :layers '[:libs {:name "readme" :entries [{:src "README" :target "/app/README" :modification-time #inst "2020"}]} :paths]'

Folders are unusable from the CLI, which I'm not happy about. I'd like to rectify that before committing. Maybe make copy the default in some way.

[SevereOverfl0w]

I'm still lukewarm on the data API, and I'd like to spend some more time using it. For now, I'm going to only support the underlying FileEntriesLayer objects as well as the keywords. Then we can go from there!

[SevereOverfl0w]

I'm merging my branch without the data support for now. It isn't simpler than using a FileEntriesLayer, and it's worth holding out to find something that will be. I'll open a follow-up issue to collect use-cases.