Compatibility with app system env variables

[andymoe]

Description

The VCAP_APPLICATION env does not maintain compatibility with traditional cloud foundry or pass alternatives environment variables such as CF_INSTANCE_INDEX documented here: https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html.

This hinders migrations of workloads from traditional cloud foundry to cf-for-k8s or kubecf and has specific impact on one of the rails applications I'm responsible for. Specifically, in my case, VCAP_APPLICATION.instance_index is relied on to run migrations from a single instance. In addition, because CF_INSTANCE_INDEX is also not available there is not a straightforward alternatives to the instance_index key in the VCAP_APPLICATION env var.

Other common tools that use these variables:

• Java-cfenv: https://github.com/pivotal-cf/java-cfenv
• Spring cloud connectors: https://github.com/spring-cloud/spring-cloud-connectors (Legacy)

Steps to reproduce

Push the following test app to cf-for-k8s

  var http = require('http');
http.createServer(function (request, response) {
   response.writeHead(200, {'Content-Type': 'text/plain'});
   response.end(`${JSON.stringify(process.env)}`);
}).listen(process.env.PORT);

Run the following curl command and grep to look for CF* environment variables: `curl my-test-app.example.com | jq . | grep CF`

and run a similar curl command and grep to look for the VCAP_APPLICATION variable

curl my-test-app.example.com | jq . | grep VCAP_APPLICATION

What was expected to happen

We'd expect compatibility with CF environment variables supported on traditional cloud foundry as described here under the app specific system variables section for running apps: https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#app-system-env

and also compatibility with the VCAP_APPLICATION environment variable detailed here: https://docs.cloudfoundry.org/devguide/deploy-apps/environment-variable.html#VCAP-APPLICATION

What actually happened

We only see the following limited set of CF_* environment variables and also a limited set under the VCAP_APPLICATION environment variable:

"CF_INSTANCE_INTERNAL_IP": "10.24.5.87",
"CF_INSTANCE_IP": "10.24.5.87",
"CF_INSTANCE_ADDR": "0.0.0.0:8080",
"CF_INSTANCE_PORT": "8080",
"CF_INSTANCE_PORTS": "[{\"external\":8080,\"internal\":8080}]",

 "VCAP_APPLICATION": "{\"cf_api\":\"my-test-app.example.com\",\"limits\":{\"fds\":16384,\"mem\":1024,\"disk\":1024},\"application_name\":\"test-env\",\"application_uris\":[\"my-test-app.example.com\"],\"name\":\"test-env\",\"space_name\":\"test\",\"space_id\":\"629ef6c5-4a40-401a-ab42-30d09b32d80b\",\"organization_id\":\"b95bc8ac-38b6-4cc8-bf72-8a6aea7fd4bb\",\"organization_name\":\"system\",\"uris\":[\"my-test-app.example.com\"],\"process_id\":\"3975af88-1740-4143-bf40-f6dd2fb6d599\",\"process_type\":\"web\",\"application_id\":\"3975af88-1740-4143-bf40-f6dd2fb6d599\",\"version\":\"dd410472-62f2-47db-868e-498bb6343159\",\"application_version\":\"dd410472-62f2-47db-868e-498bb6343159\"}"

Suggested fix (optional)

Maintain compatibility with traditional cloud foundry on VMs.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/172845120

The labels on this github issue will be updated when the story is started.

[georgethebeatle]

Hi @andymoe

We are using k8s StatefulSets In order to maintain a stable instance index of CF apps in the Kubernetes world. Unfortunately k8s does not provide the instance index to its clients. It encodes the instance number in the pod name. In fact CF apps can already access that name today if they lookup the POD_NAME env var. The name is in the format <app_name>-<space_name>-<uid>-<instance_index>, for example dora-space-42f40093cf-0. Would it be reasonable for you to use that env var and parse out the index number as a workaround?

[andymoe]

Hey @georgethebeatle, parsing out the index from the POD_NAME environment variable would be a workaround that would get our team unblocked. I do think the larger thrust of the issue is still valid that breaking compatibility in general will make it hard for folks with large cf-deployments to move apps to systems like cf-for-k8s or kubecf en masse. Perhaps there should be related issues for each environment variable or key missing from VCAP_APPLICATION

[hsiliev]

I think this would be of interest for everyone wanting to migrate from BOSH CF to K8s distros (KubeCF, cf-for-k8s).

Do the Eirini team see this as core functionality or something that can extracted as extension?

[julz]

@andymoe - agreed, we've spun out https://www.pivotaltracker.com/story/show/173013296 to address env variable compatibility

[emalm]

As an update, the Eirini story to support CF_INSTANCE_INDEX specifically is at https://www.pivotaltracker.com/story/show/173150663. Could we get that prioritized? This env var is part of the CF "downwards" API, such as it is, and we are aware of existing categories of CF apps whose instances behave differently based on whether or not the instance index is 0. (/cc @zrob as I think he has some more specifics on this that I can't recall at the moment.)