Nginx app won't start due to mkdir permission error

[danail-branekov]

Describe the bug

After bumping CF4K8S to v1.1.0 we are observing the following error when pushing the eirinidotcf to CF via this script:

2020/12/17 10:36:07 [emerg] 6#0: mkdir() "/client_body_temp" failed (13: Permission denied)

We had a look at the command which cloud controller requests for that application and it is

/bin/sh
-c
/cnb/lifecycle/launcher nginx -p $PWD -c "/workspace/nginx.conf"

We managed to confirm that the value of $PWD is the root directory (/). We tried changing the directory to /workspace via changing the command above manually and this seems to "solve" the issue.

We downgraded CF4K8S to 1.0.0 and ran the very same push script. This time the application started successfully. However, when looking at the stateful s et in kubernetes, we saw that no command has been requested by cloud controller, i.e. the application apparently runs its entrypoint.

To Reproduce*

1. Create a K8S cluster and deploy cf-for-k8s v1.1.0
2. Push eirinidotcf as per the script above
3. See the pod crashing

Expected behavior

The application starts and can be curled on the app URL given by the CF CLI

Additional context

cf-for-k8s SHA

Release v1.1.0

Deploy instructions

Our CI uses this script to deploy CF, the values from cluster state are generated by this script

Cluster information

IAAS: GKE

CLI versions

1. ytt --version: 0.30.0
2. kapp --version: 0.35.0
3. kubectl version:

Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:42:56Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

4. cf version: 7.2.0+be4a5ce2b.2020-12-10

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/176205431

The labels on this github issue will be updated when the story is started.

[Birdrock]

I rolled Eirini and CF API back to previous versions while keeping the rest of the components on v1.1.0. Doing so allowed the app to come up and could be curled. It'll take some discussion with the Eirini team and digging through CF API to debug.

[braunsonm]

For anyone else running into this as the author stated you can get around this by specifying a startup command during your push (or in the manifest) like so:

cf push <app> -c '/cnb/lifecycle/launcher nginx -p /workspace -c "/workspace/nginx.conf"'

[jamespollard8]

@danail-branekov @braunsonm We just tested this on head of develop and it's working fine now. (Specifically, we tested the eirinidotcf script here)

We believe the fix came through https://github.com/cloudfoundry/cloud_controller_ng/pull/2052

Eirini folks, you should be able to revert your commit: https://github.com/cloudfoundry-incubator/eirini-ci/commit/8d4339f255931a14998accf5808deba94f92e62e

Please let us know if you're still seeing issues here -- don't hesitate to re-open the issue

[braunsonm]

To clarify @jamespollard8 this is not fixed in any tags? Won't be able to verify the fix until the next release then

[jamespollard8]

To clarify @jamespollard8 this is not fixed in any tags? Won't be able to verify the fix until the next release then

Right. We (mostly @Birdrock) have been working hard to get this new cf-for-k8s release out but we're having a real hard time getting the necessary cf-api bump through. Releasing is a top priority for us and we hope to finish it up soon.

[jamespollard8]

This was included in cf-for-k8s v2.0.0