search

hpe-storage / python-hpedockerplugin

HPE Native Docker Plugin
Apache License 2.0
36 stars 64 forks source link

Ansible installer with encryption : After installation, doryD is in ContainerCreating state. #763

Closed sandesh-desai closed 4 years ago

sandesh-desai commented 4 years ago

Testbed Details:

[root@master-node ansible_3par_docker_plugin]# kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"7015f71e75f670eb9e7ebd4b5749639d42e20079", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} [root@master-node ansible_3par_docker_plugin]# docker version Client: Version: 18.09.6 API version: 1.39 Go version: go1.10.8 Git commit: 1578dca Built: Sat May 4 02:09:15 2019 OS/Arch: linux/amd64 Experimental: false

Server: Docker Engine - Enterprise Engine: Version: 18.09.6 API version: 1.39 (minimum version 1.12) Go version: go1.10.8 Git commit: 1578dca Built: Sat May 4 02:03:09 2019 OS/Arch: linux/amd64 Experimental: false [root@master-node ansible_3par_docker_plugin]# cat /etc/os-release NAME="Red Hat Enterprise Linux Server" VERSION="7.6 (Maipo)" ID="rhel" ID_LIKE="fedora" VARIANT="Server" VARIANT_ID="server" VERSION_ID="7.6" PRETTY_NAME="Red Hat Enterprise Linux" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.6 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.6"

10.50.9.11 - master 10.50.9.15/17 - workers

Steps Performed:

1.Created the properties file ansible-vault create plugin_configuration_properties.yml Create a new vault password. 2.Edit the properties file ansible-vault edit plugin_configuration_properties.yml and save 3.Ran the install script ansible-playbook -i /root/hosts install_hpe_3par_volume_driver.yml --ask-vault-pass 4.Password will be prompted give the same password which was give while creating 5.After installation of plugin Expected Result: doryD should be in Running state. Actual Result: doryD is in ContainerCreating state.

Attaching the output and logs: Doryd issue output.txt

MSGLOG_5thdec.txt

sneharai4 commented 4 years ago

I can see that flannel pod is down. Also, doryd is not able come up as Running due to memory issue.

error seen on doryd after running installer:

Events:
  Type     Reason                  Age                         From                   Message
  ----     ------                  ----                        ----                   -------
  Warning  FailedCreatePodSandBox  7m10s (x294129 over 3d23h)  kubelet, worker-node1  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "kube-storage-controller-doryd-7dd487b446-ngsnj": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:281: applying cgroup configuration for process caused \"mkdir /sys/fs/cgroup/memory/kubepods/besteffort/pod85a8ffb2-3259-42c4-89d3-bed4ff573164/eadb8fcc643eaf8448c8a23867f879a761d24df1459ff03baead3732d8f0b6c2: cannot allocate memory\"": unknown
  Normal   SandboxChanged          2m10s (x294412 over 3d23h)  kubelet, worker-node1  Pod sandbox changed, it will be killed and re-created.

I manually deleted the deployment and created it again. I still see the memory allocation issue.

$ kubectl delete -f dep-kube-storage-controller-k8s113.yaml
$ kubectl apply -f dep-kube-storage-controller-k8s113.yaml
$ kubectl describe pod kube-storage-controller-doryd-7dd487b446-dfwv8  -n kube-system

Events:
  Type     Reason                    Age                            From                   Message
  ----     ------                    ----                           ----                   -------
  Normal   Scheduled                 23s                            default-scheduler      Successfully assigned kube-system/kube-storage-controller-doryd-7dd487b446-dfwv8 to worker-node1
  Warning  FailedCreatePodContainer  <invalid> (x2 over <invalid>)  kubelet, worker-node1  unable to ensure pod container exists: failed to create container for [kubepods besteffort pod295487d8-67a7-4600-b02c-156e64d74c36] : mkdir /sys/fs/cgroup/memory/kubepods/besteffort/pod295487d8-67a7-4600-b02c-156e64d74c36: cannot allocate memory
amitk1977 commented 4 years ago

Verified on OpenShift 3.11 and K8s 1.15. Issue is fixed and verified.

sandesh-desai commented 4 years ago

Fresh Installation works fine. Attaching the resolved Logs:

PASSED-K8S-1.15-MM-INSTALL-3.3.1-ENCRYPTION-VAULT-PASS.txt