Including information on how to build all images for application

Background

I'm working on a tool called atomicapp-builder [1], that should be able to rebuild a whole Nulecule app based on a given Nulecule file and cccp-index [2].

Terminology

While working on atomicapp-builder, I realized that Nulecule operates with two types of images, that don't really have names in Nulecule documentation (at least I didn't find the names :)). I'll call them meta images and binary images throughout this proposal:

meta image is an image created from repo that contains a Nulecule file and perhaps some provider files (like k8s config etc); meta image doesn't contain any actual source code of the application
binary image is an image that contains services needed to actually run the application, e.g. a database, a web server with application logic etc.
Problem statement

While current Nulecule specification, combined with a cccp-index, allows resolving what meta images need to be built for an application to work, there is no general way to tell what binary images need to be built and how. This is problematic for my use case, since atomicapp-builder should be able to build all images from scratch.

Proposal

I'd like to propose an addition of optional field called images for graph items. This field, if used, would provide information required to build binary images for the application. An example follows:

graph:
  - name: foobar
    params:
      - name: image
        description: The frontend part
        default: foobar:sometag
      - name: hostport
        description: The host TCP port as the external endpoint
        default: 80
      - name: publicip
        description: The IP address or addresses at which your app can be reached
    artifacts:
      kubernetes:
        - file://artifacts/k8s/foobar-controller.json
        - file://artifacts/k8s/foobar-service.json
    images:
      - name: foobar:sometag
        source: https://github.com/foo/bar/
        scm_type: git
        build_configs:
          stable:
            tag: v1.2.3
          latest:
            branch: master
        image_type: docker
        image_buildfile: some/dir/Dockerfile

I think the example is quite self-explanatory, so I'll just comment very briefly:

images is a list of json objects with following properties:
- name (required) - name of the image to build
- source (required) - SCM repository URL
- scm_type (optional) - type of SCM to use, defaults to git
- build_configs (optional) - a json object containing named build configurations; only 2 configs are allowed - stable and master
- every build config can contain either branch or tag; the default is branch: master
- by default, both configs point to master branch, e.g.

latest:
  branch: master
stable:
  branch: master

branch or tag (optional) - branch or tag to build from
image_type (optional) - type of containerization engine to build image for, defaults to docker
image_buildfile (optional) - path of build file of the image inside the repo, defaults to Dockerfile
Things to consider / Q&A
Q: What if user doesn't provide any/all instructions on how to build images? A: Since the field is optional, this is not a problem. Some tooling may however require providing list of all images that will be needed for the application; if an incomplete list is provided, this tooling may be unable to work with the application.
Q: Why have multiple build configs? A: So that there is a builtin way to specify how to build various versions of this application, e.g. the latest development version, the latest stable version, a specific released version, etc... The tooling used for building Nulecule apps shall expose options to select one of the build configs.
Q: Why not provide arbitrary number of configs? A: This crossed my mind, but I'm not sure how to sanely implement this in the build tooling. The problem is that the set of images to build is unknown prior to invoking the build, so a defined set of configurations makes sense (I can't specify configs for images that I don't know will be built). OTOH the problem is, that I may want to be building my app with latest config, but all its dependencies in stable config... I'm a bit torn here, another opinion would be appreciated!
Q: Isn't this too complicated? A: I don't know, you tell me :) If someone can think of a nicer/more concise way, I'll be happy to see it!

CC @arrfab @vpavlin @kbsingh

[1] https://github.com/bkabrda/atomicapp-builder/ [2] https://github.com/kbsingh/cccp-index/

First thought, this issue/RFE is a excellent description of a good practice how to extend the usage of the Nulecule Spec without touching the formal spec... We have exactly want people to extend the YAML file Nulecule and put all that they need in there.

@goern +1 :) @bkabrda Very nice, I like it..and I think I don't have any objections to accept it as you proposed it.

Q1: Exactly, it's good that it's optional and we will just expect images to already exist if they are not specified

Q2: Good idea. What about some mapping to the Nulecule appversion [1]? So that I don't have to duplicate the version information in my Nulecule file if the appversion is same to the version of the binary image?

build_config:
  stable:
    tag: $appversion

Q3: Two build_configs sound good to me. You probably don't need to rebuiled older versions of images.

Q4: No, it's pretty straighforward and I think it shouldn't be very hard to add support to f.e. DevAssistant to generate images section - or the other way around - generate Nulecule from the information about the binary images (which would be very cool)

Q5: Can we somehow infer the mapping from images to params:image? So that user does not have to specify the image name and tag twice? Is that useful or is it just a complication of the spec? (I guess once we have tooling in place developer shouldn't touch Nulecule file directly too often..)

[1] https://github.com/projectatomic/nulecule/blob/master/examples/helloapache/Nulecule#L7

Thanks for your replies. I would very much like this to be an optional part of the specification itself, since I think it's crucial to be able to enumerate list of images that the application needs to run - for build tools, CI systems etc. This not only ensures that all images can be rebuilt the right way, but also allows tools to verify that no unexpected images get used when application is run - hence improving trackability and enabling auditing etc.

Q5: Can we somehow infer the mapping from images to params:image? So that user does not have to specify the image name and tag twice? Is that useful or is it just a complication of the spec? (I guess once we have tooling in place developer shouldn't touch Nulecule file directly too often..)

Good point, we should allow people to do that. I would however propose to not do this right now, since it opens a whole new can of worms called "how do we reference one part of the file from a different part". If allowed, this should be carefully thought through and handled in a different issue.

One more thing that I'd like to add to this proposal is enumerating images that user doesn't want to rebuild/doesn't know their sources etc. Something like:

images:
  - name: spam/spam

IOW, this would express that user wants to use image spam/spam and it should be obtained by standard means for given image_type. (This would effectively make source optional, which is the only technical change to my original proposal.)

Just FYI I created a repo [1] that's supposed to hold the specification of this extension (I finally had to agree that this will be an extension after an offline discussion :)) I'd like to hear if you guys have some ideas on how the extensions to Nulecule should be handled. Does this seem ok? Do we want to create some kind of list of Nulecule extensions so that people know where to look if they're missing something in the spec? (Should new bug be opened for creating some sort of extension guidelines that would answer these questions?)

[1] https://github.com/bkabrda/nulecule-images/

@goern is our specification guru so he might weigh in here and comment on a textual representation.

@tradej can help with json schema

Tomáš, would it be possible to somehow validate extensions to the spec like this one? Any ideas?

projectatomic / nulecule