search

concourse / pool-resource

atomically manages the state of the world (e.g. external environments)
Apache License 2.0
54 stars 36 forks source link

readme

pool-resource

a pool of locks (modeling semaphores)

Allows you to lock environments, pipeline flow, or other entities which have to be interacted with in a serial manner.

This resource is backed by a Git repository, so the configuration is largely the same as the git-resource.

Git Repository Structure

.
├── aws
│   ├── claimed
│   │   ├── .gitkeep
│   │   └── env-2
│   └── unclaimed
│       ├── .gitkeep
│       └── env-1
├── ping-pong-tables
│   ├── claimed
│   │   └── .gitkeep
│   └── unclaimed
│       ├── .gitkeep
│       └── north-table
└── vsphere
    ├── claimed
    │   ├── .gitkeep
    │   └── f3cb3823-a45a-49e8-ab41-e43268494205
    └── unclaimed
        ├── .gitkeep
        └── 83ed9977-3a10-4c49-a818-2d7a37693da7

This structure represents 3 pools of locks, aws, ping-pong-tables, and vsphere. The .gitkeep files are required to keep the unclaimed and claimed directories track-able by Git if there are no files in them.

You will need to mirror this structure in your own lock repository. In other words, initialize an empty repository and create one directory in the root of the repository for each pool of locks (e.g. aws). Inside of each lock pool directory, create one directory named claimed and one directory named unclaimed. Inside each claimed and unclaimed directory, create an empty filed named .gitkeep. Finally, create individual locks by making an empty file inside of the unclaimed directory (assuming you want your lock to be unclaimed by default) with the desired name of the lock (e.g. env-1).

Source Configuration

Example

Fetching a repo with only 100 commits of history:

- get: source-code
  params: {depth: 100}

Behavior

check: Check for changes to the pool.

The repository is cloned (or pulled if already present), and any commits made to the specified pool from the given version on are returned. If no version is given, the ref for HEAD is returned.

in: Fetch an acquired lock.

Outputs 2 files:

Parameters

out: Acquire, release, add, or remove a lock.

Performs one of the following actions to change the state of the pool.

Parameters

One of the following is required.

Example Concourse Configuration

The following example pipeline models acquiring, passing through, and releasing a lock based on the example git repository structure:

resources:
- name: aws-environments
  type: pool
  source:
    uri: git@github.com:concourse/locks.git
    branch: master
    pool: aws
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAtCS10/f7W7lkQaSgD/mVeaSOvSF9ql4hf/zfMwfVGgHWjj+W
      <Lots more text>
      DWiJL+OFeg9kawcUL6hQ8JeXPhlImG6RTUffma9+iGQyyBMCGd1l
      -----END RSA PRIVATE KEY-----

jobs:
- name: deploy-aws
  plan:
    - put: aws-environments
      params: {acquire: true}
    - task: deploy-aws
      file: my-scripts/deploy-aws.yml

- name: test-aws
  plan:
    - get: aws-environments
      passed: [deploy-aws]
    - task: test-aws
      file: my-scripts/test-aws.yml
    - put: aws-environments
      params: {release: aws-environments}

Managing multiple locks

The parameter for release is the name of the step whose lock to release. Normally this is just the same as the name of the resource, but if you have custom names for put (for example if you need to acquire multiple instances of the same pool), you would put that name instead. For example:

- name: test-multi-aws
  plan:
    - put: environment-1
      resource: aws-environments
      params: {acquire: true}
    - put: environment-2
      resource: aws-environments
      params: {acquire: true}
    - task: test-multi-aws
      file: my-scripts/test-multi-aws.yml
    - put: aws-environments
      params: {release: environment-1}
    - put: aws-environments
      params: {release: environment-2}

Claiming a specific lock from a pool, and releasing

The parameter for claim takes the name of a lock, and releasing the lock requires the name of the step which acquired the lock. For example:

- name: test-specific-aws
  plan:
    - put: specific-aws-env
      resource: aws-environments
      params: {claim: env-1}
    - put: aws-environments
      params: {release: specific-aws-env}

Configuring resource to proxy SSH commands through an HTTP proxy

resources:
- name: aws-environments
  type: pool
  source:
    uri: git@github.com:concourse/locks.git
    branch: master
    pool: aws
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      MIIEowIBAAKCAQEAtCS10/f7W7lkQaSgD/mVeaSOvSF9ql4hf/zfMwfVGgHWjj+W
      <Lots more text>
      DWiJL+OFeg9kawcUL6hQ8JeXPhlImG6RTUffma9+iGQyyBMCGd1l
      -----END RSA PRIVATE KEY-----
    https_tunnel:
      proxy_host: proxy-server.mycorp.com
      proxy_port: 3128
      proxy_user: myuser
      proxy_password: myverysecurepassword

Development

Prerequisites

Running the tests

The tests have been embedded with the Dockerfile; ensuring that the testing environment is consistent across any docker enabled platform. When the docker image builds, the test are run inside the docker container, on failure they will stop the build.

Run the tests with the following command:

docker build -t pool-resource --target tests --build-arg base_image=paketobuildpacks/run-jammy-base:latest .

Note about the integration tests

If you want to run the integration tests, a bit more work is required. You will require an actual git repo to which you can push and pull, configured for SSH access. To do this, add two files to integration-tests/ssh (note that names are important):

To set up or reset the contents of the repo, use the integration-tests/ssh/init-repo.sh script. The script clones the configured repository, (re-)creates the relevant directories, commits and pushes the changes. If you'd rather execute the commands yourself, view the script contents to understand the directory structure expected by the integration tests.

Then run the tests:

docker build -t pool-resource --build-arg base_image=concourse/golang-builder --target integrationtests .

Contributing

Please make all pull requests to the master branch and ensure tests pass locally.