Consider other types of "Source"

[mattmoor]

I'm opening this issue to track ideas for additional built-in "source" types for build, where today we only support Git / Custom.

[mattmoor]

Openshift supports a handful more types:

This is the openshift enum, where I've removed the overlapping options, and I've tweaked the comments to better reflect what these do out of context.

const (
    // An inline Dockerfile to build.
    BuildSourceDockerfile BuildSourceType = "Dockerfile"

    // A binary file to make available to the build portion based on various "strategies".
    BuildSourceBinary BuildSourceType = "Binary"

    // This containers the name of a Docker image to pull down and out of which to copy files.
    BuildSourceImage BuildSourceType = "Image"
)

@bparees it's not completely clear to me how folks typically use Binary?

[mattmoor]

Google Container Builder supports one more type natively.

This downloads and extracts a tarball from Google Cloud Storage.

  {
    "source": {
      "storageSource": {
        "bucket": "container-builder-examples",
        "object": "node-docker-example.tar.gz"
      }
    }
  }

However, since GCB supports multi-step builds many folks omit source and use their first step to bootstrap /workspace instead. e.g. appengine uses a form that stores multiple files in GCS and records which are relevant (and where they go) in a manifest file. A particular manifest is materialized as the first step of builds via a custom source step. We might consider adding some of these forms as more native types too.

[bparees]

@bparees it's not completely clear to me how folks typically use Binary?

Basically they typically use it when they have local source (either no git repo or they don't want to keep committing/pushing to a git repo because they're making debugging changes) that they want to rapidly iterate on... (make change in editor, save changes, run build).

This may help: https://docs.openshift.org/latest/dev_guide/dev_tutorials/binary_builds.html

[mattmoor]

@bparees Ok, that sounds a lot like the "storage source" and "manifest" systems we use in GCB. We use GCS for this, I am curious what storage medium you use to transmit the file?

[bparees]

I am curious what storage medium you use to transmit the file?

we stream it directly from the client machine to the running pod (might go through the api server? i'm not up to speed on the implementation details at the moment) and write it to the pod's filesystem. So i'm not sure if it would really be considered similar to your storage source approach or not.

This has the downside of meaning you cannot "rerun" a build since we don't have the input content preserved anywhere. We've discussed wanting to do this w/ some sort of object store for your binary content, but it hasn't gotten very high in the list.

[jchesterpivotal]

In concourse these are promoted into a separate concept -- Resources. There's a relatively simple contract, so it's easy for 3rd parties to add their own.

[knative-housekeeping-robot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale