Not able to mount on config on /app folder

rajdhandus commented 4 years ago

Steps to reproduce the issue

Create a sample aspnet core app and containerize the app
Create a ConfigMap and Deployment
Try to mount ConfigMap on /app folder as a part of the Kubernetes Deployment

Expected behavior

Container should eventually start running

Actual behavior

Container enters CrashBackOffLoop and show exit code 145

Additional information (e.g. issue happens only occasionally)

If mount location is changed to /app1 or anything else other than /app - the container works

Used this sample app - https://dotnet.microsoft.com/learn/aspnet/microservice-tutorial/run-docker for this..

Dockerfile contents

FROM mcr.microsoft.com/dotnet/core/sdk:3.1 AS build
WORKDIR /src
COPY myMicroservice.csproj .
RUN dotnet restore
COPY . .
RUN dotnet publish -c release -o /app

FROM mcr.microsoft.com/dotnet/core/aspnet:3.1.0-alpine3.10
WORKDIR /app
COPY --from=build /app .
ENTRYPOINT ["dotnet", "myMicroservice.dll"]

Output of `docker version`

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:37 2019
 OS/Arch:           windows/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of `docker info`

Client:
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc., v0.8.0)
  buildx: Build with BuildKit (Docker Inc., v0.3.1-tp-docker)

Server:
 Containers: 72
  Running: 6
  Paused: 0
  Stopped: 66
 Images: 263
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.19.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.943GiB
 Name: docker-desktop
 ID: KW5Q:KVDX:4J6O:NFNG:DUZM:YJG6:ZL3I:H7PW:3KE6:LFTU:OVBX:PPC3
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 83
  Goroutines: 108
  System Time: 2020-01-02T17:40:47.455122245Z
  EventsListeners: 5
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

app.yaml.txt

MichaelSimons commented 4 years ago

@rajdhandus - Can you provide details on what you mean by the following?

Try to mount ConfigMap on /app folder as a part of the Kubernetes Deployment

Are you literally utilizing a volume mount with a destination directory of /app? Can you provide a concrete set of steps that reproduce the problem?

rajdhandus commented 4 years ago

@MichaelSimons - Thanks for your response..I had attached the Kubernetes Yaml file with the original post. I am also quoting the config that works and the config that doesn't work here in this post... As you can compare and see - the only real difference is the mounthPath

pva.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: skylight

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: skylight
data:
  appsettings.json: |-
      {
          "Logging": {
            "LogLevel": {
              "Default": "Information",
              "Microsoft": "Warning",
              "Microsoft.Hosting.Lifetime": "Information"
            }
          },
          "powervirtualagent": {
            "botId": "981fb769-8d74-4904-933f-3d55da0b45ea",
            "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
            "contentVersion": "e7a3cfdd-80fc-e911-a98d-000d3a30dc0a",
            "tokenUrl": "https://powerva.microsoft.com/api/botmanagement/v1/directline/directlinetoken",
            "baseDirectlineUrl": "https://directline.botframework.com/v3/directline"
          },
          "AllowedHosts": "*",
          "ApplicationInsightInstrumentationKey": "56b950a8-49a3-4d59-ba12-c362d87a859e"
      }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: css-dcs-skylight-powervirtualagent
  namespace: skylight
  labels:
    app: css-dcs-skylight-powervirtualagent
spec:
  replicas: 3
  selector:
    matchLabels:
        app: css-dcs-skylight-powervirtualagent
  template:
    metadata:
      name: css-dcs-skylight-powervirtualagent
      labels:
        app: css-dcs-skylight-powervirtualagent
    spec:
      volumes:
        - name: config-volume
          configMap:
            name: special-config
      containers:
      - name: css-dcs-skylight-powervirtualagent
        image: slacrcoredev.azurecr.io/css-dcs-skylight-powervirtualagent:4350
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config
        ports:
            - containerPort: 80
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: css-dcs-skylight-powervirtualagent
  name: css-dcs-skylight-powervirtualagent
  namespace: skylight
spec:
  type: ClusterIP
  ports:
    - port: 80
  selector:
    app: css-dcs-skylight-powervirtualagent

pva1.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: skylight1

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
  namespace: skylight1
data:
  appsettings.json: |-
      {
          "Logging": {
            "LogLevel": {
              "Default": "Information",
              "Microsoft": "Warning",
              "Microsoft.Hosting.Lifetime": "Information"
            }
          },
          "powervirtualagent": {
            "botId": "981fb769-8d74-4904-933f-3d55da0b45ea",
            "tenantId": "72f988bf-86f1-41af-91ab-2d7cd011db47",
            "contentVersion": "e7a3cfdd-80fc-e911-a98d-000d3a30dc0a",
            "tokenUrl": "https://powerva.microsoft.com/api/botmanagement/v1/directline/directlinetoken",
            "baseDirectlineUrl": "https://directline.botframework.com/v3/directline"
          },
          "AllowedHosts": "*",
          "ApplicationInsightInstrumentationKey": "56b950a8-49a3-4d59-ba12-c362d87a859e"
      }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: css-dcs-skylight-powervirtualagent
  namespace: skylight1
  labels:
    app: css-dcs-skylight-powervirtualagent
spec:
  replicas: 3
  selector:
    matchLabels:
        app: css-dcs-skylight-powervirtualagent
  template:
    metadata:
      name: css-dcs-skylight-powervirtualagent
      labels:
        app: css-dcs-skylight-powervirtualagent
    spec:
      volumes:
        - name: config-volume
          configMap:
            name: special-config
      containers:
      - name: css-dcs-skylight-powervirtualagent
        image: slacrcoredev.azurecr.io/css-dcs-skylight-powervirtualagent:4350
        volumeMounts:
        - name: config-volume
          mountPath: /app
        ports:
            - containerPort: 80
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
        readinessProbe:
          httpGet:
            path: /health
            port: 80
          initialDelaySeconds: 3
          periodSeconds: 3
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: css-dcs-skylight-powervirtualagent
  name: css-dcs-skylight-powervirtualagent
  namespace: skylight1
spec:
  type: ClusterIP
  ports:
    - port: 80
  selector:
    app: css-dcs-skylight-powervirtualagent

Kubectl Deployment Commands

PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl apply -f .\pva.yaml
namespace/skylight created
configmap/special-config created
deployment.apps/css-dcs-skylight-powervirtualagent created
service/css-dcs-skylight-powervirtualagent created
PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl get pods -n skylight
NAME                                                 READY   STATUS    RESTARTS   AGE
css-dcs-skylight-powervirtualagent-f7ddcb4cf-c2lqs   0/1     Running   0          3s
css-dcs-skylight-powervirtualagent-f7ddcb4cf-djc5z   0/1     Running   0          3s
css-dcs-skylight-powervirtualagent-f7ddcb4cf-tkxvj   0/1     Running   0          3s
PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl apply -f .\pva.yaml
namespace/skylight1 created
configmap/special-config created
deployment.apps/css-dcs-skylight-powervirtualagent created
service/css-dcs-skylight-powervirtualagent created
PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl get pods -n skylight1
NAME                                                  READY   STATUS   RESTARTS   AGE
css-dcs-skylight-powervirtualagent-8657c8f975-dhk2f   0/1     Error    0          3s
css-dcs-skylight-powervirtualagent-8657c8f975-q2l8t   0/1     Error    0          3s
css-dcs-skylight-powervirtualagent-8657c8f975-ts4pd   0/1     Error    0          3s
PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl get pods -n skylight1
NAME                                                  READY   STATUS             RESTARTS   AGE
css-dcs-skylight-powervirtualagent-8657c8f975-dhk2f   0/1     Error              1          4s
css-dcs-skylight-powervirtualagent-8657c8f975-q2l8t   0/1     CrashLoopBackOff   1          4s
css-dcs-skylight-powervirtualagent-8657c8f975-ts4pd   0/1     Error              1          4s
PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl get pods -n skylight1
NAME                                                  READY   STATUS             RESTARTS   AGE
css-dcs-skylight-powervirtualagent-8657c8f975-dhk2f   0/1     CrashLoopBackOff   1          6s
css-dcs-skylight-powervirtualagent-8657c8f975-q2l8t   0/1     CrashLoopBackOff   1          6s
css-dcs-skylight-powervirtualagent-8657c8f975-ts4pd   0/1     CrashLoopBackOff   1          6s
PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl get pods -n skylight1
NAME                                                  READY   STATUS             RESTARTS   AGE
css-dcs-skylight-powervirtualagent-8657c8f975-dhk2f   0/1     CrashLoopBackOff   5          5m44s
css-dcs-skylight-powervirtualagent-8657c8f975-q2l8t   0/1     CrashLoopBackOff   5          5m44s
css-dcs-skylight-powervirtualagent-8657c8f975-ts4pd   0/1     CrashLoopBackOff   5          5m44s

PS C:\Users\rapurush\source\repos\CSS-DCS-Skylight-PowerVirtualAgent\PowerVirtualAgent.Host\k8s> kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"1da9875156ba0ad48e7d09a5d00e41489507f592", GitTreeState:"clean", BuildDate:"2019-11-14T05:19:20Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

MichaelSimons commented 4 years ago

@rajdhandus - Because you are mounting the config to the /app folder in your image, isn't that overwriting the content of the /app folder from the image (e.g. your binaries)? Have you tried this locally and inspected the file layout of your running container?

MichaelSimons commented 4 years ago

@rajdhandus - did you get a chance to look at my last question?

rajdhandus commented 4 years ago

@MichaelSimons - shown below is how the /app folder looks when i run the docker container in my local docker engine..

PS C:\Users\rapurush\source\repos\myMicroservice> docker run -it slacrcoredev.azurecr.io/my-microservice:v2
/app #
/app # ls -lrt
total 224
-rwxr-xr-x    1 root     root           192 Jan  1 01:22 appsettings.json
-rwxr-xr-x    1 root     root           162 Jan  1 01:22 appsettings.Development.json
-rw-r--r--    1 root     root           490 Jan  2 17:31 web.config
-rw-r--r--    1 root     root           213 Jan  2 17:31 myMicroservice.runtimeconfig.json
-rw-r--r--    1 root     root          1828 Jan  2 17:31 myMicroservice.pdb
-rw-r--r--    1 root     root          9216 Jan  2 17:31 myMicroservice.dll
-rw-r--r--    1 root     root        103147 Jan  2 17:31 myMicroservice.deps.json
-rwxr-xr-x    1 root     root         86584 Jan  2 17:31 myMicroservice

When I deploy the same to the Kubernetes Cluster - I would ideally like to overwrite the appsettings.json with the ConfigMap mount content. However, even if I have a different filename - I still see an issue with the Kubernetes deployment.. The containers do not run successfully inside the Kubernetes Cluster(AKS & Docker 4 desktop)...

Should we not mount anything in the same folder where the binaries are located?

MichaelSimons commented 4 years ago

@rajdhandus - if you volume mount a folder to an existing folder you overwrite the original directory. The following demonstrates this.

$ docker run -it --rm --entrypoint ls mcr.microsoft.com/dotnet/core/samples:dotnetapp /app
dotnetapp            dotnetapp.dll  dotnetapp.runtimeconfig.json
dotnetapp.deps.json  dotnetapp.pdb
$ docker run -it --rm --entrypoint ls -v d:\repos\dotnet-docker:/app mcr.microsoft.com/dotnet/core/samples:dotnetapp /app
2.1               README.md               eng
3.0               README.runtime-deps.md  manifest.json
3.1               README.runtime.md       manifest.samples.json
CONTRIBUTING.md   README.samples.md       samples
ISSUE_TEMPLATE    README.sdk.md           tests
LICENSE           build-and-test.ps1
README.aspnet.md  documentation

You can however volume mount specific files as demonstrated below. This may work well for your scenario.

Adding a file:

$ docker run -it --rm --entrypoint ls -v d:\repos\dotnet-docker\README.md:/app/README.md mcr.microsoft.com/dotnet/core/samples:dotnetapp /app
README.md  dotnetapp.deps.json  dotnetapp.pdb
dotnetapp  dotnetapp.dll        dotnetapp.runtimeconfig.json

Replacing a file:

$ docker run -it --rm --entrypoint ls -v d:\repos\dotnet-docker\README.md:/app/dotnetapp.dll mcr.microsoft.com/dotnet/core/samples:dotnetapp /app
dotnetapp            dotnetapp.dll  dotnetapp.runtimeconfig.json
dotnetapp.deps.json  dotnetapp.pdb
$ docker run -it --rm --entrypoint cat -v d:\repos\dotnet-docker\README.md:/app/dotnetapp.dll mcr.microsoft.com/dotnet/core/samples:dotnetapp /app/dotnetapp.dll
# Featured Repos

* [dotnet/core/sdk](https://hub.docker.com/_/microsoft-dotnet-core-sdk/): .NET Core SDK
* [dotnet/core/aspnet](https://hub.docker.com/_/microsoft-dotnet-core-aspnet/): ASP.NET Core Runtime
* [dotnet/core/runtime](https://hub.docker.com/_/microsoft-dotnet-core-runtime/): .NET Core Runtime
* [dotnet/core/runtime-deps](https://hub.docker.com/_/microsoft-dotnet-core-runtime-deps/): .NET Core Runtime Dependencies
* [dotnet/core/samples](https://hub.docker.com/_/microsoft-dotnet-core-samples/): .NET Core Samples

# About .NET Core
...

MichaelSimons commented 4 years ago

Closing as there doesn't appear to be an issue with the .NET Core dockerfiles. This is question regarding the use of volume mounts. A recommendation was made on using file volume mounts.

dotnet / dotnet-docker