v8: Render error when parameters file have more variables than needed

[Sirtea]

Description

I'm used to split a large list of services in several dockerapp files for maintainability. As some of data is used by several stacks (split, but a single big stack after all...) my approach was to have "the only one parameters file (tm)". This worked on version 0.6. In the new version 0.8 this no longer works.

Steps to reproduce the issue:

1. Create a dockerapp file (one should suffice for this time)

gerard@atlantis:~/workspace/example$ cat servers.dockerapp 
version: 0.1.0
name: servers
description: 
maintainers:
  - name: sirtea
    email: 

---
version: "3.6"
services:
  sftpserver:
    image: none/none
    environment:
      SFTP_USER: ${common.sftp_user}

---

common.sftp_user: FILLME
gerard@atlantis:~/workspace/example$ 

2. Create a parameters file

gerard@atlantis:~/workspace/example$ cat parameters.yml 
common.sftp_user: sirtea
sftpclient.whatever: "the sky is blue"
gerard@atlantis:~/workspace/example$ 

3. Render with docker-app v0.6 (this works)

gerard@atlantis:~/workspace/example$ docker-app render servers.dockerapp -f parameters.yml 
version: "3.6"
services:
  sftpserver:
    environment:
      SFTP_USER: sirtea
    image: none/none
gerard@atlantis:~/workspace/example$ 

4. Render again with v0.8 (this does not work)

gerard@atlantis:~/workspace/example$ ./docker-app08 render servers.dockerapp --parameters-file parameters.yml 
Error: parameter "sftpclient.whatever" is not defined in the bundle
gerard@atlantis:~/workspace/example$ 

Describe the results you received:

Render fails because excess of variables in the parameters file, that is shared by other dockerapp files. The "one docker-app = one parameters" approach is not useful; extracting the common part to a third file is unmaintainable as there are 17 dockerapp files and the common part depends on every subset of dockerapps, so the list of "common" parameters files gets quickly big.

Describe the results you expected:

Render whatever can be rendered and ignore the extra parameters.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

gerard@atlantis:~/workspace/example$ docker version
Client:
 Version:           18.09.6
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        481bc77
 Built:             Sat May  4 02:36:00 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.6
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       481bc77
  Built:            Sat May  4 01:59:36 2019
  OS/Arch:          linux/amd64
  Experimental:     false
gerard@atlantis:~/workspace/example$ 

Output of docker-app version:

gerard@atlantis:~/workspace/example$ docker-app version
Version:      v0.6.0
Git commit:   9f9c6680
Built:        Thu Oct  4 13:30:33 2018
OS/Arch:      linux/amd64
Experimental: off
Renderers:    none
gerard@atlantis:~/workspace/example$ ./docker-app08 version
Version:               v0.8.0
Git commit:            7eea32b7
Built:                 Tue Jun 11 20:53:26 2019
OS/Arch:               linux/amd64
Experimental:          off
Renderers:             none
Invocation Base Image: docker/cnab-app-base:v0.8.0
gerard@atlantis:~/workspace/example$ 

Output of docker info:

gerard@atlantis:~/workspace/example$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 4
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.0-9-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.57GiB
Name: atlantis
ID: TTBA:5DAK:VRH2:XGWT:UK5A:I6SV:XDM6:GRTG:GSGL:IBLH:JHQP:XGHW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
HTTP Proxy: <set but secret>
HTTPS Proxy: <set but secret>
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support
gerard@atlantis:~/workspace/example$ 

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical desktop linux machine with GUI and Docker. Problem relates to rendering and docker is not used at all on the example.

[kinghuang]

I've also run into this upgrading CI pipelines for docker-app 0.8.0. The GitLab CI job is doing things like --set image.repository=$CI_REGISTRY_IMAGE to make the project's Docker image repo available to apps, whether they use them or not, since the apps can't pick up environment variables as parameters (#360). This is now an error if the app doesn't have the parameter defined.

This makes it really hard to use a standardized CI job to install apps and pass them optional information from the CI environment that they might need.

[chris-crone]

Hi @Sirtea and @kinghuang,

Thanks for bringing attention to this! I definitely think that it's a valid use case and we should handle it. Do you think that we should warn users about ignored parameters like docker build does?

I'd also like to know how you're storing your common configurations? Are you using a git repo?

[Sirtea]

According to my use case, the warnings would be a lot for each deploy, so it would be quite messy; I still think that the warnings are useful so, if a user makes a typo in the var name it will notice. Thinking openly, there should be warnings, but I would appreciate being able to silence them either by parameter or simply with stderr redirection.

By the way, my usage is to store all dockerapp's related by service in a single repository (one by project or group of microservices). This project is used as a way to avoid variable repetition (DRY principle), as an easy way to deploy new environments, and as a way to split private information that shouldn't be in the repository (credentials and so on); the env-specific parameters file is not in the git repository.

[chris-crone]

Thinking openly, there should be warnings, but I would appreciate being able to silence them either by parameter or simply with stderr redirection.

This is exactly what we were thinking :)