Buildpack is smarter about where to find secrets.yml

[izgeri]

The buildpack is currently hardcoded to search the root directory for secrets.yml, and it errors if it doesn't find it.

The buildpack can be updated to accept an environment variable from the application's environment that configures the directory to search for secrets.yml in. In addition, it can be updated so that if the environment variable is not set, it searches in root and in other common directories (like BOOT-INF/classes/ for Java apps provided as JAR files - see below) used by applications for configuration YAML, so that the buildpack will work with applications in a variety of languages out of the box without requiring any specialized config.

AC:

• [ ] Buildpack is updated to accept an environment variable to configure the secrets.yml directory
• [ ] If secrets.yml directory is not set in the application environment, the buildpack is updated to look in BOOT-INF/classes/ as well as root for secrets.yml files. The ability to search directories for the secrets.yml file is written in such a way as to make it easy to add additional directories in future updates to the buildpack.
• [ ] Buildpack README is updated with information on configuring the secrets.yml file location
• [ ] Tile docs are updated with information on secrets.yml file location when using buildpack

Java apps

Java apps using the standard buildpack may not have the secrets.yml file in the root directory of the application in CF, breaking the expectations of the buildpack here.

We can update the buildpack to also check for the secrets.yml file in BOOT-INF/classes, which is where configuration files end up once built using Spring Boot v1.4+ if they are placed in src/main/resources pre-build (more info).

An example Java app manifest:

---
applications:
- name: sample-app
  memory: 512M
  instances: 1
  path: ./build/libs/sample-1.0.jar
  env:
    SPRING_PROFILES_ACTIVE: qa,cloud,default
  services:
  - sample-service

[izgeri]

Notes:

• We have an existing sample Java app deployed this way with a sample buildpack that searches ${BUILD_DIR} and ${BUILD_DIR}/BOOT-INF/classes to find secretless.yml when the .jar is deployed.

  • The secretless.yml is stored in src/app/resources in the app directory, but is in ${BUILD_DIR}/BOOT-INF/classes if you inspect the .jar file with jar tf target/hello-0.1.0.jar after running ./bin/build

• I've attempted to build a .jar file of the sample app above with secrets.yml in the root dir, and it is not included in the resulting .jar file.

• We may also want to take the opportunity to support the potential directory structure for war files as part of this update.

[izgeri]

From @dsbyrne -

the alternative would be to add an env variable to allow the end user to override the search path for secrets.yml

Not a bad idea to add to this work as well, since it will enable other types of applications that don't put secrets.yml in an expected location to continue using the buildpack as well.

[izgeri]

Note: we are still waiting for feedback from a customer with this workflow that this is a workable solution.

[rafis3]

I'm in favour of the environment variable approach, for the location of secrets.yml, because:

1. It's a better user experience, when we don't require rebuilding the applications.
2. We might encounter similar issues with other programming languages as well, which would require us to add more hard-coded lookup locations. An environment variable would be a generic solution that would always work.

[izgeri]

@RafiS3 can you clarify the first point? I'm not sure what you mean by:

It's a better user experience, when we don't require rebuilding the applications.

Also, we've spoken about this a bit offline but it's worth noting here as well. I agree that the env var is a great generic solution, and I've updated the issue to reflect that support for that should be implemented. In addition, I agree that we should support hardcoded values for several popular languages so that manually setting an env var is not required. Do you think that we should do that research before starting work on this issue, or is it sufficient to add Java support and env var config support now and add support for other languages in future efforts (which should be as simple as adding a new path to search)?

[rafis3]

@izgeri my point is that with some programming languages, the application artifact itself contains these default folders that we plan to support. So for example, if your application is a Java jar file, then the secrets.yml file would be built into the jar, because that's where BOOT-INF/classes/ is located. I think that we should recommend secrets.yml to be external to the application, so our users won't have to rebuilt it.

[TheSecMaven]

i think the issue we are having is related to this.

We have a manifest where we specify path: /path/to/jar.jar

this uploads the jar. the jar DOESNT contain secrets.yml. this fails, saying it can't find secrets.yml.

We update path to path:/path/to and this directory contains jar.jar, and secrets.yml (voila, conjur buildpack works, but PCF can't start the app for some reason, seemingly because it doesn't know what to do with the jar.

We don't want to have to put secrets.yml in our jar files because that means for any config change we have to rebuild our app, which is not consistent with the "build once deploy anywhere" strategy that we see people adopting across the industry.

Thoughts on this? reccomendations?

[izgeri]

@mkkeffeler I've moved this to its own issue so that we can help with it - please see #80