Container app startup fails if it's created into any other project/namespace than "default"

[mikahe]

General information

OS: Linux Ubuntu 22.04.1 and 20.04.4
Hypervisor: KVM 
Did you run crc setup before starting it? (Yes)
Running CRC on: Baremetal-Server

CRC version: 2.6.0+d606e64

[# Put crc version output here](version: 2.6.0+d606e64)

CRC status

DEBU CRC version: 2.6.0+d606e64
DEBU OpenShift version: 4.10.22
DEBU Podman version: 4.1.0
DEBU Running 'crc status'
DEBU Checking file: /home/mika/.crc/machines/crc/.crc-exist DEBU Checking file: /home/mika/.crc/machines/crc/.crc-exist DEBU Found binary path at /home/mika/.crc/bin/crc-driver-libvirt DEBU Launching plugin server for driver libvirt
DEBU Plugin server listening at address 127.0.0.1:43495 DEBU () Calling .GetVersion
DEBU Using API Version 1
DEBU () Calling .SetConfigRaw
DEBU () Calling .GetMachineName
DEBU (crc) Calling .GetBundleName
DEBU (crc) Calling .GetState
DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="Getting current state..." DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="Fetching VM..." DEBU (crc) Calling .GetIP
DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="GetIP called for crc" DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="Getting current state..." DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="IP address: 192.168.130.11" DEBU (crc) Calling .GetIP
DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="GetIP called for crc" DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="Getting current state..." DEBU (crc) DBG | time="2022-08-26T11:12:05+03:00" level=debug msg="IP address: 192.168.130.11" DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1 DEBU Using ssh private keys: [/home/mika/.crc/machines/crc/id_ecdsa /home/mika/.crc/cache/crc_libvirt_4.10.22_amd64/id_ecdsa_crc] DEBU SSH command results: err: , output: 32737570816 26869628928 /sysroot DEBU Making call to close driver server
DEBU (crc) Calling .Close
DEBU Successfully made call to close driver server DEBU Making call to close connection to plugin binary CRC VM: Running OpenShift: Running (v4.10.22) Podman:
Disk Usage: 26.87GB of 32.74GB (Inside the CRC VM) Cache Usage: 17.09GB Cache Directory: /home/mika/.crc/cache

Steps to reproduce

Create container app in new project/namespace some other than "default", e.g. "demo" or whatever. Use either web console or cli: https://docs.openshift.com/container-platform/4.10/applications/projects/working-with-projects.html

Expected: Using custom project name/namespace container app works like virtual machines do Actual: Container app startup is successful only using "default" project Logs: E.g. using nginx: 2022/08/26 07:28:40 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

[praveenkumar]

@mikahe I tried with crc 2.7.1 and able to create new project and also deploy the new app.

$ oc new-project demo
Now using project "demo" on server "https://api.crc.testing:6443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

    kubectl create deployment hello-node --image=k8s.gcr.io/serve_hostname

$ oc project
Using project "demo" on server "https://api.crc.testing:6443".

$  oc new-app rails-postgresql-example
--> Deploying template "openshift/rails-postgresql-example" to project demo

     Rails + PostgreSQL (Ephemeral)
     ---------
     An example Rails application with a PostgreSQL database. For more information about using this template, including OpenShift considerations, see https://github.com/sclorg/rails-ex/blob/master/README.md.

     WARNING: Any data stored will be lost upon pod destruction. Only use this template for testing.
[...]

$ oc get pods -n demo
NAME                               READY   STATUS      RESTARTS   AGE
hello-node-c9c694789-b8m5d         1/1     Running     0          47s
postgresql-1-8tsvj                 1/1     Running     0          4m22s
postgresql-1-deploy                0/1     Completed   0          4m34s
rails-postgresql-example-1-build   1/1     Running     0          4m35s

[mikahe]

Hi praveenkumar, thanks for the info! Hopefully crc 2.7.1 will resolve it then. I'm not using it yet, since the Virtualization operator is not compatible with 4.11, so stuck in 2.6.0 currently.

$ oc new-project demo Now using project "demo" on server "https://api.crc.testing:6443".

You can add applications to this project with the 'new-app' command. For example, try:

oc new-app rails-postgresql-example

to build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application:

kubectl create deployment hello-node --image=k8s.gcr.io/e2e-test-images/agnhost:2.33 -- /agnhost serve-hostname

$ oc project Using project "demo" on server "https://api.crc.testing:6443".

$ oc new-app docker.io/nginx --> Found container image 2b7d643 (6 days old) from docker.io for "docker.io/nginx"

* An image stream tag will be created as "nginx:latest" that will track this image

--> Creating resources ... imagestream.image.openshift.io "nginx" created deployment.apps "nginx" created service "nginx" created --> Success Application is not exposed. You can expose services to the outside world by executing one or more of the commands below: 'oc expose service/nginx' Run 'oc status' to view your app. $ oc status In project demo on server https://api.crc.testing:6443

svc/nginx - 10.217.5.47:80 deployment/nginx deploys istag/nginx:latest deployment #2 running for 7 seconds - 0/1 pods (warning: 1 restarts) deployment #1 deployed 8 seconds ago - 0/1 pods growing to 1

Errors:

• pod/nginx-f746d5475-7xvg8 is crash-looping

1 error, 1 info identified, use 'oc status --suggest' to see details. $ oc logs pod/nginx-f746d5475-7xvg8 /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: can not modify /etc/nginx/conf.d/default.conf (read-only file system?) /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2022/08/29 12:44:12 [warn] 1#1: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2 nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /etc/nginx/nginx.conf:2 2022/08/29 12:44:12 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied) nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

$ oc version Client Version: 4.10.22 Server Version: 4.10.22 Kubernetes Version: v1.23.5+3afdacb

$ oc project default Now using project "default" on server "https://api.crc.testing:6443".

$ oc new-app docker.io/nginx --> Found container image 2b7d643 (6 days old) from docker.io for "docker.io/nginx"

* An image stream tag will be created as "nginx:latest" that will track this image

--> Creating resources ... deployment.apps "nginx" created service "nginx" created --> Success Application is not exposed. You can expose services to the outside world by executing one or more of the commands below: 'oc expose service/nginx' Run 'oc status' to view your app.

$ oc get pods NAME READY STATUS RESTARTS AGE nginx-f746d5475-bd4kq 1/1 Running 0 9s

$ oc logs pod/nginx-f746d5475-bd4kq /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh /docker-entrypoint.sh: Configuration complete; ready for start up 2022/08/29 12:51:24 [notice] 1#1: using the "epoll" event method 2022/08/29 12:51:24 [notice] 1#1: nginx/1.23.1 2022/08/29 12:51:24 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 2022/08/29 12:51:24 [notice] 1#1: OS: Linux 4.18.0-305.49.1.el8_4.x86_64 2022/08/29 12:51:24 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2022/08/29 12:51:24 [notice] 1#1: start worker processes 2022/08/29 12:51:24 [notice] 1#1: start worker process 31 2022/08/29 12:51:24 [notice] 1#1: start worker process 32 2022/08/29 12:51:24 [notice] 1#1: start worker process 33 2022/08/29 12:51:24 [notice] 1#1: start worker process 34 2022/08/29 12:51:24 [notice] 1#1: start worker process 35

$ oc get pods NAME READY STATUS RESTARTS AGE nginx-f746d5475-bd4kq 1/1 Running 0 2m48s

[cfergeau]

I just tested this, and it's more likely a generic openshift issue than something crc specific. I could reproduce it, you need to be logged in as kubeadmin in order to create the docker.io/nginx app in the default namespace. When in that namespace, oc debug nginx-... gives me a root prompt, while if I do the same in a different namespace (eg test), then I get a user prompt. And the failure is related to an error creating a directory in /var/cache/... which is owned by root. So it could be some issue with the permissions you get when creating an app in a non-default namespace, or it could be the image itself which is not 'openshift-ready', I don't know if some adjustments may be needed to it. But I don't think there's anything crc specific in there.