search

sosiouxme / osbs-client

client for OpenShift Build Service
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

How osbs-client reads config and creates builds #1

Open sosiouxme opened 6 years ago

sosiouxme commented 6 years ago

osbs-client portion of https://github.com/sosiouxme/atomic-reactor/issues/1

sosiouxme commented 6 years ago

Initial creation of the build

koji-containerbuild-builder imports the osbs client in /usr/lib/koji-builder-plugins/builder_containerbuild.py and uses it to create a container build. It creates an osbs.conf.Configuration() as conf Configuration is created with its defaults: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/conf.py#L39-L55 https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/constants.py#L17-L18

Meaning /etc/osbs.conf which (in the environment of kojid) looks like:

[general]
build_json_dir = /usr/share/osbs/
verbose = 1
openshift_required_version = 1.0.6

[platform:x86_64]
architecture = amd64
enable_v1 = true

[default]
authoritative_registry = registry.fedoraproject.org
build_image = local_buildroot:latest
build_type = prod
distribution_scope = public
koji_certs_secret = kojisecret
koji_hub = https://172.17.0.1:8083/kojihub
koji_root = https://172.17.0.1:8083/kojifiles
namespace = osbs
openshift_url = https://172.17.0.1:8443/
registry_api_versions = v1,v2
registry_secret = v2-registry-dockercfg
registry_uri = https://172.17.0.1:5000/v2
source_registry_uri = http://registry.fedoraproject.org
sources_command = rhpkg sources
use_auth = true
use_kerberos = false
vendor = Red Hat, Inc.
verify_ssl = false
client_config_secret = client-config-secret
reactor_config_map = reactor-config-map
can_orchestrate = true
username = osbs
password = osbs

[scratch]
... similar ...

This is used to create osbs as osbs.api.OSBS(conf, conf): https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L89-L94

... which essentially just copies config values from the [default] section over to an OpenShift() instance: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/core.py#L49-L51

It then uses that to call osbs.create_orchestrator_build(orchestrator_create_build_args). https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L763-L774

Build args look like:

{'architecture': None,
 'component': 'docker-hello-world',
 'git_branch': 'osbs-box-demo',
 'git_ref': 'origin/osbs-box-demo',
 'git_uri': 'git://github.com/sosiouxme/docker-hello-world',
 'koji_task_id': 4,
 'platforms': ['x86_64'],
 'scratch': False,
 'target': 'candidate',
 'user': 'kojiadmin',
 'yum_repourls': []
}
sosiouxme commented 6 years ago

osbs.create_orchestrator_build sets defaults for conf templates based on constants: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L792-L798 https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/constants.py#L21-L29

... and creates a build: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L535-L549

This creates a BuildRequestV2: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/build_requestv2.py#L25-L29 which creates a BuildUserParams to store as user_params: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/user_params.py#L24 https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/spec.py#L109

A bunch of params are set on this and then it's passed on to be rendered into an OpenShift BuildConfig json: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L425-L426

Which is either created or updated in OpenShift, along with attendant ImageStream and triggers. It results in:

apiVersion: v1
kind: BuildConfig
metadata:
  creationTimestamp: 2018-04-25T18:29:51Z
  labels:
    git-branch: osbs-box-demo
    git-repo-name: docker-hello-world
    is_autorebuild: "true"
    koji-task-id: "6"
  name: docker-hello-world-osbs-box-demo-f7644
  namespace: osbs
  resourceVersion: "101482"
  selfLink: /oapi/v1/namespaces/osbs/buildconfigs/docker-hello-world-osbs-box-demo-f7644
  uid: a4786c3a-48b6-11e8-afa9-54e1addf5a42
spec:
  nodeSelector: null
  output:
    to:
      kind: DockerImage
      name: kojiadmin/docker-hello-world:candidate-32571-20180425182951
  postCommit: {}
  resources: {}
  runPolicy: Serial
  source:
    git:
      ref: origin/osbs-box-demo
      uri: git://github.com/sosiouxme/docker-hello-world
    type: Git
  strategy:
    customStrategy:
      env:
      - name: REACTOR_CONFIG
        valueFrom:
          configMapKeyRef:
            key: config.yaml
            name: reactor-config-map
      - name: USER_PARAMS
        value: '{"arrangement_version": 6, "base_image": "fedora:latest", "build_image":
          "local_buildroot:latest", "build_json_dir": "/usr/share/osbs/", "build_type":
          "orchestrator", "component": "docker-hello-world", "customize_conf": "orchestrator_customize.json",
          "git_branch": "osbs-box-demo", "git_ref": "origin/osbs-box-demo", "git_uri":
          "git://github.com/sosiouxme/docker-hello-world", "image_tag": "kojiadmin/docker-hello-world:candidate-32571-20180425182951",
          "imagestream_name": "lucarval-docker-hello-world", "koji_target": "candidate",
          "koji_task_id": 6, "name": "docker-hello-world-osbs-box-demo-f7644", "platforms":
          ["x86_64"], "reactor_config_map": "reactor-config-map", "trigger_imagestreamtag":
          "fedora:latest", "user": "kojiadmin"}'
      exposeDockerSocket: true
      from:
        kind: DockerImage
        name: local_buildroot:latest
      secrets:
      - mountPath: /var/run/secrets/atomic-reactor/client-config-secret
        secretSource:
          name: client-config-secret
      - mountPath: /var/run/secrets/atomic-reactor/kojisecret
        secretSource:
          name: kojisecret
      - mountPath: /var/run/secrets/atomic-reactor/v2-registry-dockercfg
        secretSource:
          name: v2-registry-dockercfg
    type: Custom
  triggers: []
status:
  lastVersion: 1

This results in the build pod running atomic-reactor.

sosiouxme commented 6 years ago

Within the build pod, atomic_reactor creates an OSBS with osbs.conf.Configuration(build_json_dir="/usr/share/osbs"). Since the default conf file /etc/osbs.conf is not present within the pod, this config doesn't read anything, just stores the extra kwarg. https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/conf.py#L54-L61

The resulting osbs.api.OSBS object basically stores an empty OpenShift client that presumably is unused. https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L90-L94

With atomic_reactor calling render_plugins_configuration(user_params) against this: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/api.py#L1227-L1231

where the user_params (the JSON from the USER_PARAMS env var above) are fed into a BuildUserParams: https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/user_params.py#L167-L170

which is used to create a PluginsConfiguration https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/plugins_configuration.py#L127-L137

which creates and stores a PluginsTemplate https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/plugins_configuration.py#L21-L22

with template_path="orchestrator_inner:6.json" and customize_conf_path="orchestrator_customize.json" https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/inputs/orchestrator_inner:6.json#L1-L116 https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/inputs/orchestrator_customize.json#L1-L7

Now the PluginsConfiguration gets render()ed which is where that plugin template is modified (plugins added, removed, args changed) and returned as json. https://github.com/sosiouxme/osbs-client/blob/9f8ea0530ed263f227a8d93b82c22867b0d5e8d8/osbs/build/plugins_configuration.py#L493

This JSON is passed back to OSv3InputPlugin.