Conductor volumes don't utilize top-level volumes directive

[marcusianlevine]

ISSUE TYPE

• Bug Report

container.yml

---

version: '2'

settings:
  conductor_base: "debian:jessie"
  conductor:
    base: "debian:jessie"
    environment:
      - COMPOSE_HTTP_TIMEOUT=300
      - "COMPOSE_PROJECT_NAME={{ compose_project_name }}"
     volumes:
       - temp-space:/tmp # Used to copy static content between containers
    project_name: "{{ compose_project_name }}"

defaults:
  compose_project_name: **_REDACTED_**
  PROJECT_NAME: analyticsdashboard
  DB_USER: "{{ PROJECT_NAME }}"
  DB_PASSWORD: **_REDACTED_**
  DB_NAME: "{{ PROJECT_NAME }}"
  DJANGO_ROOT: /django
  DJANGO_VENV: /venv
  DJANGO_USER: "django"
  DJANGO_PORT: 5000

services:
  ### DJANGO APP SERVED BY GUNICORN ###
  django:
    from: 'debian:jessie'
    roles:
      - &django_role
        role: marcusianlevine.django-container
        db_user: "{{ DB_USER }}"
        db_password: "{{ DB_PASSWORD }}"
        db_name: "{{ DB_NAME }}"
        django_port: "{{ DJANGO_PORT }}"
        project_name: "{{ PROJECT_NAME }}"
        var_files: 
          - secrets/defaults.yml
          - env_vars/base.yml
          - env_vars/rabbitmq.yml
          - env_vars/docker.yml
        script_templates:
          - scripts/gunicorn_start.j2
          - scripts/celery_start.j2
          - scripts/flower_start.j2
          - scripts/beat_start.j2
    environment: &django_env
      DATABASE_URL: 'pgsql://{{ DB_USER }}:{{ DB_PASSWORD }}@db:5432/{{ DB_NAME }}'
      DJANGO_ROOT: '{{ DJANGO_ROOT }}'
      DJANGO_VENV: '{{ DJANGO_VENV }}'
      PROJECT_NAME: '{{ PROJECT_NAME }}'
    expose:
      - '{{ DJANGO_PORT }}'
    working_dir: '{{ DJANGO_ROOT }}/{{ PROJECT_NAME }}'
    links:
      - postgresql:db
      - redis
    user: '{{ DJANGO_USER }}'
    command: ['/usr/bin/dumb-init', '/usr/bin/gunicorn_start']
    entrypoint: [/usr/bin/entrypoint.sh]
    volumes: 
      - 'logs:/{{ DJANGO_ROOT }}/{{ PROJECT_NAME }}/{{ PROJECT_NAME }}/logs'
    dev_overrides:
      environment: &debug_env
        <<: *django_env
        DEBUG: 'True'
      volumes:
        - '$PWD:{{ DJANGO_ROOT }}'
      command: [/usr/bin/dumb-init, /usr/bin/manage_django.sh, runserver, '0.0.0.0:{{ DJANGO_PORT }}']
      ports:
        - '{{ DJANGO_PORT }}:{{ DJANGO_PORT }}'

  ### CELERY WORKERS ###
  main_worker: &celery_worker
    from: "{{ compose_project_name }}-django"
    environment:
      <<: *django_env
    working_dir: '{{ DJANGO_ROOT }}/{{ PROJECT_NAME }}'
    links:
      - postgresql:db
      - rabbit
    user: '{{ DJANGO_USER }}'
    command: ['/usr/bin/dumb-init', '/usr/bin/celery_start']
    entrypoint: [/usr/bin/entrypoint.sh]
    dev_overrides:
      environment:
        <<: *django_env
        DEBUG: 'True'
      command: /bin/false
  tickets_worker:
    <<: *celery_worker
    command: ['/usr/bin/dumb-init', '/usr/bin/celery_start', 'tickets']
  calls_worker:
    <<: *celery_worker
    command: ['/usr/bin/dumb-init', '/usr/bin/celery_start', 'calls']
  beat: # only ever run one beat instance to avoid duplicate tasks
    <<: *celery_worker
    command: ['/usr/bin/dumb-init', '/usr/bin/beat_start']
  flower: # celery monitoring GUI
    <<: *celery_worker
    expose:
      - 5555
    command: ['/usr/bin/dumb-init', '/usr/bin/flower_start']

  ### CELERY BROKER (RABBITMQ) ### 
  rabbit:
    from: 'rabbitmq:3.6-management'
    environment:
      - RABBITMQ_DEFAULT_USER=admin
      - RABBITMQ_DEFAULT_PASS="{{ DB_PASSWORD }}"
    expose:
      - 5672  # useful for debugging
      - 15672  # rabbitmq management plugin

  ### DOCUMENTATION SERVER ###
  #TODO: write docs service, should just use nginx to serve statics

  ### NGINX PROXY SERVER ###
  nginx:
    from: 'debian:jessie'
    roles:
    - role: marcusianlevine.nginx-container
      ASSET_PATHS:
      - /tmp/django
      PROXY: yes
      PROXY_PASS: 'http://django:{{ DJANGO_PORT }}'
      PROXY_LOCATION: "/"
    expose:
      - 8000
    ports: []
    links:
      - django
    dev_overrides:
      command: /bin/false

  ### DATA STORES ###
  postgresql: # application database
    # Uses a pre-built postgresql image from Docker Hub 
    from: postgres:9.6
    environment:
      - 'POSTGRES_DB={{ DB_NAME }}'
      - 'POSTGRES_USER={{ DB_USER }}'
      - 'POSTGRES_PASS={{ DB_PASSWORD }}'
      - 'PGDATA=/var/lib/pgsql/data/userdata'
    volumes:
      - postgres-data:/var/lib/pgsql/data
    expose:
      - 5432

  redis: # in-memory cache
    from: 'redis:3.2'
volumes:
  postgres-data:
    docker: {}
  logs:
    docker: {}
  temp-space:
    docker: {}

OS / ENVIRONMENT

Ansible Container, version 0.9.2rc0
Darwin, ml-c02kc2qxffrp.local, 16.7.0, Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64, x86_64
2.7.11 |Anaconda 4.0.0 (x86_64)| (default, Dec  6 2015, 18:57:58)
[GCC 4.2.1 (Apple Inc. build 5577)] /Users/mlevine/anaconda/bin/python
{
  "ContainersPaused": 0,
  "Labels": null,
  "CgroupDriver": "cgroupfs",
  "ContainersRunning": 2,
  "ContainerdCommit": {
    "Expected": "cfb82a876ecc11b5ca0977d1733adbe58599088a",
    "ID": "cfb82a876ecc11b5ca0977d1733adbe58599088a"
  },
  "InitBinary": "docker-init",
  "NGoroutines": 84,
  "Swarm": {
    "ControlAvailable": false,
    "NodeID": "",
    "Error": "",
    "RemoteManagers": null,
    "LocalNodeState": "inactive",
    "NodeAddr": ""
  },
  "LoggingDriver": "json-file",
  "OSType": "linux",
  "HttpProxy": "",
  "Runtimes": {
    "runc": {
      "path": "docker-runc"
    }
  },
  "DriverStatus": [
    [
      "Backing Filesystem",
      "extfs"
    ],
    [
      "Supports d_type",
      "true"
    ],
    [
      "Native Overlay Diff",
      "true"
    ]
  ],
  "OperatingSystem": "Alpine Linux v3.5",
  "Containers": 12,
  "HttpsProxy": "",
  "BridgeNfIp6tables": true,
  "MemTotal": 2096168960,
  "SecurityOptions": [
    "name=seccomp,profile=default"
  ],
  "Driver": "overlay2",
  "IndexServerAddress": "https://index.docker.io/v1/",
  "ClusterStore": "",
  "InitCommit": {
    "Expected": "949e6fa",
    "ID": "949e6fa"
  },
  "Isolation": "",
  "SystemStatus": null,
  "OomKillDisable": true,
  "ClusterAdvertise": "",
  "SystemTime": "2017-07-21T20:22:06.5662711Z",
  "Name": "moby",
  "CPUSet": true,
  "RegistryConfig": {
    "AllowNondistributableArtifactsCIDRs": [],
    "Mirrors": [],
    "IndexConfigs": {
      "docker.io": {
        "Official": true,
        "Name": "docker.io",
        "Secure": true,
        "Mirrors": []
      }
    },
    "AllowNondistributableArtifactsHostnames": [],
    "InsecureRegistryCIDRs": [
      "127.0.0.0/8"
    ]
  },
  "DefaultRuntime": "runc",
  "ContainersStopped": 10,
  "NCPU": 2,
  "NFd": 72,
  "Architecture": "x86_64",
  "KernelMemory": true,
  "CpuCfsQuota": true,
  "Debug": true,
  "ID": "3ZDR:2D52:OZLT:QMTY:PPSG:I3NL:2SA5:KONO:Y646:MWOD:7XZJ:4H7E",
  "IPv4Forwarding": true,
  "KernelVersion": "4.9.36-moby",
  "BridgeNfIptables": true,
  "NoProxy": "",
  "LiveRestoreEnabled": false,
  "ServerVersion": "17.06.0-ce",
  "CpuCfsPeriod": true,
  "ExperimentalBuild": true,
  "MemoryLimit": true,
  "SwapLimit": true,
  "Plugins": {
    "Volume": [
      "local"
    ],
    "Network": [
      "bridge",
      "host",
      "ipvlan",
      "macvlan",
      "null",
      "overlay"
    ],
    "Authorization": null,
    "Log": [
      "awslogs",
      "fluentd",
      "gcplogs",
      "gelf",
      "journald",
      "json-file",
      "logentries",
      "splunk",
      "syslog"
    ]
  },
  "Images": 101,
  "DockerRootDir": "/var/lib/docker",
  "NEventsListener": 1,
  "CPUShares": true,
  "RuncCommit": {
    "Expected": "2d41c047c83e09a6d61d464906feb2a2f3c52aa4",
    "ID": "2d41c047c83e09a6d61d464906feb2a2f3c52aa4"
  }
}
{
  "KernelVersion": "4.9.36-moby",
  "Arch": "amd64",
  "BuildTime": "2017-06-23T21:51:55.152028673+00:00",
  "ApiVersion": "1.30",
  "Version": "17.06.0-ce",
  "MinAPIVersion": "1.12",
  "GitCommit": "02c1d87",
  "Os": "linux",
  "Experimental": true,
  "GoVersion": "go1.8.3"
}

SUMMARY

I am trying to write a container.yml that can be used to on a Jenkins CI server to build many iterations of the same project in parallel for a multibranch pipeline.

I'm running into a bunch of issues associated with running ansible-container build within many build processes on the same Docker host, the latest of which is the temp volume not having a unique name per run.

I tried to solve this using the top-level volumes directive to create the temp-space volume before mounting to the conductor, but it seems that the conductor creates its own separate temp-space volume without the project name prepended as is done by the volumes TLD

STEPS TO REPRODUCE

> ansible-container build

... long boring build runs ...

> docker volume ls

EXPECTED RESULTS

With the introduction of the volumes top-level directory, I would expect the conductor to reference the same project-scoped volumes that are created by the TLD

ACTUAL RESULTS

I find that the conductor creates its own temp-space volume with just the literal name, even though I defined temp-space in the volumes TLD

I tried to work around this with environment-provided container.yml variables, but it seems that volume names in the conductor section do not access global variables

local               testbuild6_logs
local               testbuild6_postgres-data
local               testbuild6_temp-space
...
local               temp-space

[marcusianlevine]

I was able to work around this issue, but I'm going to leave it open because it does seem that the conductor should share the top-level volumes namespace

Am I missing something? Is there a reason the conductor can't mount volumes defined in the top-level volumes directive?

[fish22]

Same issue here when building a maven project I use the maven docker container that creates a volume at /root/.m2 which I mount to a toplevel volume.

The idea is to reuse the same maven cache in /root/.m2 for future builds of the same project. However, I get new empty volume each build and have to redownload all maven depedencies during the build...

[chouseknecht]

Ansible Container doesn't create the top-level volumes. When it runs the conductor container, it builds up a docker run command, and hands it off to the docker python module. I believe in the end the docker daemon creates the volume, and handles the volume's state between container runs, restarts, etc.

I've run into the same problem, where on subsequent runs of the build command the named volume contains no data from the prior run. My fix has been to use a host volume, where a local file system path is mounted to the conductor.

[fish22]

@chouseknecht Well, from my test I have seen that this (At least for my particular case) is fixed in #576.

It appears that since the volumes were not being mounted during the build, if you build twice, (in my case for example) you will have an empty /root/.m2 each time. And also on the initial run.

After the fix above, the first build will create a named docker volume that will be reused each time this volume is needed (in build and run) until I remove it myself by hand.

[marcusianlevine]

@fish22 is right, my original issue is resolved by #576 as well, since I can just use a shared volume to transfer files instead of a conductor temp volume