Complete config flow for a build

[sosiouxme]

This seemed the simplest way to create a doc annotated with code snippets side by side with the config that's feeding them, starting from where kojid starts the build. Examples are from my local osbs-box deployment.

[sosiouxme]

https://github.com/sosiouxme/osbs-client/issues/1#issuecomment-384400855 gives us in the osbs namespace the orchestrator BuildConfig which results in a Build which spawns a Pod like so:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    openshift.io/build.name: docker-hello-world-osbs-box-demo-f7644-1
    openshift.io/scc: privileged
  creationTimestamp: 2018-04-25T18:29:51Z
  labels:
    openshift.io/build.name: docker-hello-world-osbs-box-demo-f7644-1
  name: docker-hello-world-osbs-box-demo-f7644-1-build
  namespace: osbs
  ownerReferences: ...
spec:
  containers:
  - env:
    - name: BUILD
      value: <...json for own Build object...>
    - name: SOURCE_REPOSITORY
      value: git://github.com/sosiouxme/docker-hello-world
    - name: SOURCE_URI
      value: git://github.com/sosiouxme/docker-hello-world
    - name: SOURCE_REF
      value: origin/osbs-box-demo
    - name: ORIGIN_VERSION
      value: v3.6.0+c4dd4cf
    - name: OUTPUT_REGISTRY
    - name: OUTPUT_IMAGE
      value: kojiadmin/docker-hello-world:candidate-32571-20180425182951
    - name: REACTOR_CONFIG
      valueFrom:
        configMapKeyRef:
          key: config.yaml
          name: reactor-config-map
    - name: USER_PARAMS
      value: '{"arrangement_version": 6, "base_image": "fedora:latest", "build_image":
        "local_buildroot:latest", "build_json_dir": "/usr/share/osbs/", "build_type":
        "orchestrator", "component": "docker-hello-world", "customize_conf": "orchestrator_customize.json",
        "git_branch": "osbs-box-demo", "git_ref": "origin/osbs-box-demo", "git_uri":
        "git://github.com/sosiouxme/docker-hello-world", "image_tag": "kojiadmin/docker-hello-world:candidate-32571-20180425182951",
        "imagestream_name": "lucarval-docker-hello-world", "koji_target": "candidate",
        "koji_task_id": 6, "name": "docker-hello-world-osbs-box-demo-f7644", "platforms":
        ["x86_64"], "reactor_config_map": "reactor-config-map", "trigger_imagestreamtag":
        "fedora:latest", "user": "kojiadmin"}'
    - name: OPENSHIFT_CUSTOM_BUILD_BASE_IMAGE
      value: local_buildroot:latest
    - name: DOCKER_SOCKET
      value: /var/run/docker.sock
    image: local_buildroot:latest
...
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: docker-socket
    - mountPath: /var/run/secrets/atomic-reactor/client-config-secret
      name: client-config-secret-secret
      readOnly: true
    - mountPath: /var/run/secrets/atomic-reactor/kojisecret
      name: kojisecret-secret
      readOnly: true
    - mountPath: /var/run/secrets/atomic-reactor/v2-registry-dockercfg
      name: v2-registry-dockercfg-secret
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: builder-token-csj7w
      readOnly: true
...
  volumes:
  - hostPath:
      path: /var/run/docker.sock
    name: docker-socket
  - name: client-config-secret-secret
    secret:
      defaultMode: 420
      secretName: client-config-secret
  - name: kojisecret-secret
    secret:
      defaultMode: 420
      secretName: kojisecret
  - name: v2-registry-dockercfg-secret
    secret:
      defaultMode: 420
      secretName: v2-registry-dockercfg
  - name: builder-token-csj7w
    secret:
      defaultMode: 420
      secretName: builder-token-csj7w
status:

REACTOR_CONFIG here is passed config.yaml from reactor-config-map which was initialized in the namespace and (for me) looks like this:

---
version: 1

clusters:
  x86_64:
  - name: worker
    max_concurrent_builds: 4
    enabled: true

clusters_client_config_dir: /var/run/secrets/atomic-reactor/client-config-secret

koji:
  hub_url: https://172.17.0.1:8083/kojihub
  root_url: https://172.17.0.1:8083/kojifiles
  auth:
    ssl_certs_dir: /var/run/secrets/atomic-reactor/kojisecret

image_labels:
  vendor: "Red Hat, Inc."
  authoritative-source-url: registry.fedoraproject.org
  distribution-scope: public

openshift:
  url: https://172.17.0.1:8443/
  insecure: True
  build_json_dir: /usr/share/osbs/
  auth:
    enable: True

platform_descriptors:
- platform: x86_64
  architecture: amd64
  enable_v1: True

content_versions:
- v1
- v2

registries:
- url: https://172.17.0.1:5000/v2
  insecure: True
  auth:
    cfg_path: /var/run/secrets/atomic-reactor/v2-registry-dockercfg

source_registry:
  url: http://registry.fedoraproject.org

sources_command: rhpkg sources

required_secrets:
- kojisecret
- v2-registry-dockercfg

worker_token_secrets:
- client-config-secret

The image command is atomic-reactor --verbose inside-build

[sosiouxme]

Inside the orchestrator build

This invokes build_inside with input_method set to the script args default --input=auto: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L518-L521 which uses the auto pseudo input plugin to figure things out from the environment: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L543-L545 This results in OSv3InputPlugin being used for input. Its run method reads config from the env vars: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/input_osv3.py#L143-L147 ... though as far as I can see the config is only used internally, just stored as an attr on the plugin and never referenced outside it.

get_plugins_with_user_params generates an osbs client config from the container's /usr/share/osbs/ directory: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/input_osv3.py#L29-L37

ending up with modified contents of orchestrator_inner:6.json as the plugins

[sosiouxme]

Completing the plugin config

Now OSv3InputPlugin takes this plugin config, adds some git repo info, removes certain plugins that lack config (e.g. smtp with no email address), and returns that as its plugin output. https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/input_osv3.py#L154-L169

{
    "source": {
        "provider_params": {
            "git_commit": "origin/osbs-box-demo"
        }, 
        "uri": "git://github.com/sosiouxme/docker-hello-world", 
        "provider": "git"
    }, 
    "postbuild_plugins": [
        {
            "name": "fetch_worker_metadata"
        }, 
        {
            "name": "compare_components"
        }, 
        {
            "args": {
                "tag_suffixes": {
                    "unique": [
                        "candidate-32571-20180425182951"
                    ], 
                    "primary": [
                        "latest", 
                        "{version}", 
                        "{version}-{release}"
                    ]
                }
            }, 
            "name": "tag_from_config"
        }, 
        {
            "name": "group_manifests"
        }
    ], 
    "openshift_build_selflink": "/apis/build.openshift.io/v1/namespaces/osbs/builds/docker-hello-world-osbs-box-demo-f7644-1", 
    "prepublish_plugins": [], 
    "prebuild_plugins": [
        {
            "name": "reactor_config"
        }, 
        {
            "args": {
                "label_value": "true", 
                "label_key": "is_autorebuild"
            }, 
            "name": "check_and_set_rebuild"
        }, 
        {
            "required": false, 
            "name": "check_and_set_platforms", 
            "args": {
                "koji_target": "candidate"
            }
        }, 
        {
            "args": {
                "check_platforms": true
            }, 
            "name": "pull_base_image"
        }, 
        {
            "name": "bump_release"
        }, 
        {
            "name": "add_labels_in_dockerfile"
        }, 
        {
            "name": "koji_parent"
        }
    ], 
    "image": "kojiadmin/docker-hello-world:candidate-32571-20180425182951", 
    "exit_plugins": [
        {
            "required": false, 
            "name": "import_image", 
            "args": {
                "imagestream": "lucarval-docker-hello-world"
            }
        }, 
        {
            "name": "koji_import"
        }, 
        {
            "args": {
                "target": "candidate"
            }, 
            "name": "koji_tag_build"
        }, 
        {
            "name": "store_metadata_in_osv3"
        }, 
        {
            "name": "remove_built_image"
        }, 
        {
            "name": "remove_worker_metadata"
        }
    ], 
    "buildstep_plugins": [
        {
            "args": {
                "platforms": [
                    "x86_64"
                ], 
                "build_kwargs": {
                    "git_uri": "git://github.com/sosiouxme/docker-hello-world", 
                    "arrangement_version": 6, 
                    "target": "candidate", 
                    "reactor_config_map": "reactor-config-map", 
                    "koji_task_id": 6, 
                    "git_ref": "origin/osbs-box-demo", 
                    "user": "kojiadmin", 
                    "component": "docker-hello-world", 
                    "git_branch": "osbs-box-demo"
                }, 
                "config_kwargs": {
                    "build_image": "local_buildroot:latest"
                }
            }, 
            "name": "orchestrate_build"
        }
    ]
}

[sosiouxme]

Running the configured plugins

This is now fed into a DockerBuildWorkflow and runs: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L555-L556 https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L312-L338

DockerBuildWorkflow init turns the repo source into a Source (here a GitSource) that lazily checks out the repo when you try to get anything from it. https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L340 https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/source.py#L127-L141 https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/source.py#L75-L80

At this point we have the source and plugins can start looking for container.yaml inside it, e.g. when checking for autorebuild: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/pre_check_and_set_rebuild.py#L112-L120

[sosiouxme]

So DBW starts running the pre-build plugins: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L435-L439

The first is reactor_config which does its own read of the env var from the reactor config map: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/pre_reactor_config.py#L460-L479 The ReactorConfig object is stored in the workflow.plugin_workspace keyed by its own key, where it can be retrieved later by a module method that looks for it there: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/pre_reactor_config.py#L23-L25 as well as other module methods that leverage this. Other plugins import the module methods when they need to access this config, e.g.: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/build_orchestrate_build.py#L28-L29 The point of this scheme seems to be to read the config once and be able to retrieve it later, but always retrieve it as a deep copy so that plugins cannot accidentally overwrite values that other plugins look at afterward.

After the various pre-build plugins run, the build step plugins run: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/inner.py#L449-L453

The BuildStepPluginsRunner is where the docker_api buildstep plugin is set by default: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugin.py#L411-L419

Since we're in the orchestrator build, though, the orchestrate_build plugin is present in the plugin config so that is used.

[sosiouxme]

The orchestrate_build plugin sets up a ThreadPool to create and watch one worker build per platform (currently, on a platform-specific cluster): https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/build_orchestrate_build.py#L854-L855

Then it generates an osbs cluster client for each cluster that might build for each platform, and selects one per platform (based on priority and availability) to attempt the build (retries may select a different cluster). The osbs client is in a container and does not have /etc/osbs.conf to look at. I'm not ready yet to investigate this too deeply but it is pieced together according to cluster data from the reactor config: https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/build_orchestrate_build.py#L438-L449

With the osbs client config for a worker cluster in hand, the build is created. https://github.com/sosiouxme/atomic-reactor/blob/e96aaa5303bed115aee48c55d076a8b86a90ccf0/atomic_reactor/plugins/build_orchestrate_build.py#L604

As with the orchestrator build, a BuildConfig is created and a Build started with starts a pod like this:

apiVersion: v1
kind: Pod
metadata:
  [...]
spec:
  containers:
  - env:
    - name: BUILD
      value: "<... Build object ...>"
    - name: SOURCE_REPOSITORY
      value: git://github.com/sosiouxme/docker-hello-world
    - name: SOURCE_URI
      value: git://github.com/sosiouxme/docker-hello-world
    - name: SOURCE_REF
      value: origin/osbs-box-demo
    - name: ORIGIN_VERSION
      value: v3.6.0+c4dd4cf
    - name: OUTPUT_REGISTRY
    - name: OUTPUT_IMAGE
      value: kojiadmin/docker-hello-world:candidate-11114-20180425183022-x86_64
    - name: REACTOR_CONFIG
      value: |
        clusters:
          x86_64:
          - {enabled: true, max_concurrent_builds: 4, name: worker}
        clusters_client_config_dir: /var/run/secrets/atomic-reactor/client-config-secret
        content_versions: [v1, v2]
        image_labels: {authoritative-source-url: registry.fedoraproject.org, distribution-scope: public,
          vendor: 'Red Hat, Inc.'}
        koji:
          auth: {ssl_certs_dir: /var/run/secrets/atomic-reactor/kojisecret}
          hub_url: https://172.17.0.1:8083/kojihub
          root_url: https://172.17.0.1:8083/kojifiles
        openshift:
          auth: {enable: true}
          build_json_dir: /usr/share/osbs/
          insecure: true
          url: https://172.17.0.1:8443/
        platform_descriptors:
        - {architecture: amd64, enable_v1: true, platform: x86_64}
        registries:
        - auth: {cfg_path: /var/run/secrets/atomic-reactor/v2-registry-dockercfg}
          insecure: true
          url: https://172.17.0.1:5000/v2
        required_secrets: [kojisecret, v2-registry-dockercfg]
        source_registry: {url: 'http://registry.fedoraproject.org'}
        sources_command: rhpkg sources
        version: 1
    - name: USER_PARAMS
      value: '{"arrangement_version": 6, "base_image": "fedora:latest", "build_image":
        "local_buildroot:latest", "build_json_dir": "/usr/share/osbs/", "build_type":
        "worker", "component": "docker-hello-world", "customize_conf": "worker_customize.json",
        "git_branch": "osbs-box-demo", "git_ref": "origin/osbs-box-demo", "git_uri":
        "git://github.com/sosiouxme/docker-hello-world", "image_tag": "kojiadmin/docker-hello-world:candidate-11114-20180425183022-x86_64",
        "imagestream_name": "lucarval-docker-hello-world", "koji_target": "candidate",
        "koji_task_id": 6, "koji_upload_dir": "koji-upload/1524681021.170606.JTroFLPr",
        "name": "docker-hello-world-osbs-box-demo-f7644", "platform": "x86_64", "reactor_config_override":
        {"clusters": {"x86_64": [{"enabled": true, "max_concurrent_builds": 4, "name":
        "worker"}]}, "clusters_client_config_dir": "/var/run/secrets/atomic-reactor/client-config-secret",
        "content_versions": ["v1", "v2"], "image_labels": {"authoritative-source-url":
        "registry.fedoraproject.org", "distribution-scope": "public", "vendor": "Red
        Hat, Inc."}, "koji": {"auth": {"ssl_certs_dir": "/var/run/secrets/atomic-reactor/kojisecret"},
        "hub_url": "https://172.17.0.1:8083/kojihub", "root_url": "https://172.17.0.1:8083/kojifiles"},
        "openshift": {"auth": {"enable": true}, "build_json_dir": "/usr/share/osbs/",
        "insecure": true, "url": "https://172.17.0.1:8443/"}, "platform_descriptors":
        [{"architecture": "amd64", "enable_v1": true, "platform": "x86_64"}], "registries":
        [{"auth": {"cfg_path": "/var/run/secrets/atomic-reactor/v2-registry-dockercfg"},
        "insecure": true, "url": "https://172.17.0.1:5000/v2"}], "required_secrets":
        ["kojisecret", "v2-registry-dockercfg"], "source_registry": {"url": "http://registry.fedoraproject.org"},
        "sources_command": "rhpkg sources", "version": 1}, "release": "1", "trigger_imagestreamtag":
        "fedora:latest", "user": "kojiadmin"}'
    - name: OPENSHIFT_CUSTOM_BUILD_BASE_IMAGE
      value: local_buildroot:latest
    - name: DOCKER_SOCKET
      value: /var/run/docker.sock
    image: local_buildroot:latest
    imagePullPolicy: IfNotPresent
    name: custom-build
    resources: {}
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: FallbackToLogsOnError
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: docker-socket
    - mountPath: /var/run/secrets/atomic-reactor/kojisecret
      name: kojisecret-secret
      readOnly: true
    - mountPath: /var/run/secrets/atomic-reactor/v2-registry-dockercfg
      name: v2-registry-dockercfg-secret
      readOnly: true
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: builder-token-kgtnw
      readOnly: true
[...]
  volumes:
  - hostPath:
      path: /var/run/docker.sock
    name: docker-socket
  - name: kojisecret-secret
    secret:
      defaultMode: 420
      secretName: kojisecret
  - name: v2-registry-dockercfg-secret
    secret:
      defaultMode: 420
      secretName: v2-registry-dockercfg
  - name: builder-token-kgtnw
    secret:
      defaultMode: 420
      secretName: builder-token-kgtnw