search

ibm-cloud-architecture / terraform-icp-vmware

Terraform recipe to deploy IBM Cloud Private on VMware
10 stars 15 forks source link

ICP 2.1.0.3 + 3.1 - No space left when image from NFS #26

Open jesusmah opened 6 years ago

jesusmah commented 6 years ago

Trying to install ICP 2.1.03 providing the ICP image from the CSP's NFS server fails at:

$ cat terraform_1971711340.sh 
#!/bin/sh
echo "Loading image ibmcom/icp-inception:2.1.0.3-ee"
/tmp/icp-bootmaster-scripts/load-image.sh ibmcom/icp-inception:2.1.0.3-ee /tmp/null "nfs:172.16.0.12:/storage/icp/2.1.0.3/ibm-cloud-private-x86_64-2.1.0.3.tar.gz"

with the following message in the loadimage.txt file: Error processing tar file(exit status 1): write /usr/local/bin/envoy: no space left on device

Space left on the VM:

virtuser@jesus-icp-2103-master01:/tmp$ df -H
Filesystem                                   Size  Used Avail Use% Mounted on
udev                                         8.4G     0  8.4G   0% /dev
tmpfs                                        1.7G  9.2M  1.7G   1% /run
/dev/sda1                                     52G   28G   23G  56% /
tmpfs                                        8.5G     0  8.5G   0% /dev/shm
tmpfs                                        5.3M     0  5.3M   0% /run/lock
tmpfs                                        8.5G     0  8.5G   0% /sys/fs/cgroup
/dev/sdc1                                     53G   55M   50G   1% /opt/ibm/cfc
172.16.247.81:/storage/jesus/sedev/registry  528G   11G  491G   3% /var/lib/registry
172.16.247.81:/storage/jesus/sedev/audit     528G   11G  491G   3% /var/lib/icp/audit
172.16.0.12:/storage/icp/2.1.0.3              17T  118G   16T   1% /opt/ibm/cluster/images
tmpfs                                        1.7G     0  1.7G   0% /run/user/1000

However, some docker images where loaded onto the system:

$ docker images
REPOSITORY                                TAG                 IMAGE ID            CREATED             SIZE
ibmcom/icp-inception                      2.1.0.3-ee          a4f59956b139        4 months ago        644MB
ibmcom/icp-vip-manager                    1.0                 2e41ff51026f        4 months ago        19.2MB
ibmcom/icp-catalog-ui                     2.1.0               f072b729fbb6        4 months ago        150MB
ibmcom/icp-platform-ui                    2.2.0               fe5dcee6fa2a        4 months ago        201MB
ibmcom/zipkin                             2.6.0               9f735543340c        5 months ago        219MB
ibmcom/icp-identity-manager               1.0.0               33fb3041cbd0        5 months ago        207MB
ibmcom/k8s-dns-sidecar                    1.14.4              9a1c6e466be1        5 months ago        46.7MB
ibmcom/k8s-dns-kube-dns                   1.14.4              386ed8c54576        5 months ago        54.2MB
ibmcom/k8s-dns-dnsmasq-nanny              1.14.4              399bf079e2d6        5 months ago        46.3MB
ibmcom/rescheduler                        v0.5.2              f5e5c67370c9        5 months ago        73.2MB
ibmcom/kube-mgmt                          0.4                 9c0259795ca0        5 months ago        48.1MB
ibmcom/icp-mongodb-migration              2.1.0.3             583d58c6c162        5 months ago        480MB
ibmcom/logging-pki-init                   1.3.0               22a76ded471c        5 months ago        223MB
ibmcom/icp-helm-api                       1.0.0               4cc42d7cb86b        5 months ago        431MB
ibmcom/iam-policy-decision                1.0.0               32f40de42177        5 months ago        599MB
ibmcom/sas-api                            1.2.1               7ee25909a0d9        5 months ago        664MB
ibmcom/sas-mgmt                           1.2.1               e948275b72d9        5 months ago        653MB
ibmcom/icp-image-manager                  2.2.2               af47c2ebda24        5 months ago        25.6MB
ibmcom/metering-data-manager              1.0.3               d1595e7a955e        5 months ago        80.1MB
ibmcom/metering-ui                        1.0.3               4a3c20ed1c2b        5 months ago        432MB
ibmcom/icp-platform-api                   2.2.0               37003752449a        5 months ago        172MB
ibmcom/icp-router                         2.2.0               4f72c5f909bd        5 months ago        46.5MB
ibmcom/icp-management-ingress             2.2.0               4b5ac3c6d7c5        5 months ago        74.4MB
ibmcom/iam-policy-administration          1.0.0               b0694d21fbdb        5 months ago        748MB
ibmcom/icp-identity-provider              1.0.0               0f59c941d44d        5 months ago        204MB
ibmcom/icp-platform-auth                  1.0.0               a10aba353a9b        5 months ago        955MB
ibmcom/tiller                             v2.7.3-icp          554f8c347a0b        5 months ago        75.9MB
ibmcom/icp-mongodb-install                0.5                 93d91ed55bc7        5 months ago        16.3MB
ibmcom/live-crawler                       1.2.1               36a489c8592a        5 months ago        1.01GB
ibmcom/reg-crawler                        1.2.1               58ef95b4908b        5 months ago        1.04GB
ibmcom/statsd                             0.7.2               58280bd2d544        5 months ago        1.08GB
ibmcom/k8szk                              v2                  bbc1bc99adcd        5 months ago        550MB
ibmcom/usncrawler                         1.2.1               d33344d81010        5 months ago        183MB
ibmcom/metering-server                    1.0.3               ff3fcc211e33        5 months ago        706MB
ibmcom/mariadb                            10.2.14             2404aa3e9672        5 months ago        396MB
ibmcom/notification-dispatcher            1.2.1               593cd129b03d        5 months ago        1.2GB
ibmcom/secure-config-annotator            1.2.1               29dc1cf99fa1        5 months ago        1.34GB
ibmcom/rootkit-annotator                  1.2.1               5286ed8501e4        5 months ago        480MB
ibmcom/icp-helm-rudder                    1.0.0               c73a8d56613a        5 months ago        28.7MB
ibmcom/password-annotator                 1.2.1               01b0af727c62        5 months ago        1.2GB
ibmcom/compliance-annotator               1.2.1               9609722ac3fb        5 months ago        1.29GB
ibmcom/vulnerability-annotator            1.2.1               4b87abcbeaf3        5 months ago        1.27GB
ibmcom/config-parser                      1.2.1               3263296595c6        5 months ago        1.23GB
ibmcom/py-generic-indexer                 1.2.1               ecc934dfee5f        5 months ago        1.27GB
ibmcom/py-config-indexer                  1.2.1               370200a6e594        5 months ago        1.27GB
ibmcom/iam-token-service                  1.0.0               c880185f4bd1        5 months ago        1.01GB
ibmcom/service-catalog-service-catalog    v0.1.11-icp         78f69feba697        5 months ago        131MB
ibmcom/metering-reader                    1.0.3               b947c2190af9        5 months ago        218MB
ibmcom/nginx-ingress-controller           0.13.0              cacf73a34d3f        5 months ago        329MB
ibmcom/kubernetes                         v1.8.3              53cf8bb74a1e        5 months ago        552MB
ibmcom/hyperkube                          v1.10.0-ee          6d154c49abf7        5 months ago        684MB
ibmcom/k8s-prometheus-adapter             v0.2.1              4a3ba29699fa        5 months ago        52.1MB
ibmcom/istio-kubectl                      v1.10.0             c698aa197774        5 months ago        63.1MB
ibmcom/coredns                            1.0.3               cf59d5df812d        5 months ago        42MB
ibmcom/calico-upgrade                     v1.0.3              fd636cd20f6c        5 months ago        35.1MB
ibmcom/searchguard-init                   1.0.0               ff2077c498cf        6 months ago        139MB
ibmcom/icp-platform-deploy                2.2.0               e8129ada35f8        6 months ago        196MB
ibmcom/istio-grafana                      0.7.1               0e265670d2bc        6 months ago        306MB
ibmcom/istio-ca                           0.7.1               5c82fea42c78        6 months ago        42.7MB
ibmcom/istio-mixer                        0.7.1               251b0d3d2b93        6 months ago        54MB
ibmcom/istio-servicegraph                 0.7.1               570fc9572491        6 months ago        10.6MB
ibmcom/istio-sidecar_injector             0.7.1               1db980725edb        6 months ago        33.4MB
ibmcom/istio-proxy_init                   0.7.1               79ced81390a3        6 months ago        115MB
ibmcom/istio-pilot                        0.7.1               6223c9364df7        6 months ago        51.9MB
ibmcom/elasticsearch-plugin-searchguard   1.0.0               0c0a436dd529        6 months ago        16.9MB
ibmcom/calico-node                        v3.0.4              5361c5a52912        6 months ago        278MB
ibmcom/icp-cert-gen                       1.0.0               ac1db4a3e46d        6 months ago        79.3MB
ibmcom/calico-ctl                         v2.0.2              5437ae7b00dc        6 months ago        42.8MB
ibmcom/calico-kube-controllers            v2.0.2              0754e1c707e7        6 months ago        55.1MB
ibmcom/calico-cni                         v2.0.3              cef0252b1749        6 months ago        69.1MB
ibmcom/indices-cleaner                    0.2                 cd62be0eb54d        6 months ago        62.6MB
ibmcom/icp-mongodb                        3.6.0               5b1317f8158f        7 months ago        366MB
ibmcom/unified-router                     2.2.0               63efa05248d3        8 months ago        48.7MB
ibmcom/va-elasticsearch-curator           5.4.1               9f274a701e60        8 months ago        72.6MB
ibmcom/kube-state-metrics                 v1.2.0              6fc90bfda615        8 months ago        35.6MB
ibmcom/alertmanager                       v0.13.0             6f3ac9ce31ba        9 months ago        31.9MB
ibmcom/etcd                               v3.2.14             94bc3af972c9        9 months ago        37.2MB
ibmcom/registry                           2.6.2               d1fd7d86a825        9 months ago        33.3MB
ibmcom/icp-initcontainer                  1.0.0               77144d8c6bdc        9 months ago        3.97MB
ibmcom/elasticsearch-exporter             1.0.2               d767dc6ccfb2        9 months ago        9.2MB
ibmcom/metrics-server                     v0.2.1              9801395070f3        9 months ago        42.5MB
ibmcom/grafana                            4.6.3               5d1d85717268        10 months ago       284MB
ibmcom/icp-datastore                      1.1.0               786a7ea5fc0b        10 months ago       1.72GB
ibmcom/opa                                0.5.13              0e807e1d469c        10 months ago       13.4MB
ibmcom/node-exporter                      v0.15.2             ff5ecdcfc4a2        10 months ago       22.8MB
ibmcom/calico-routereflector              v0.5.0              63526fadb3f0        10 months ago       292MB
ibmcom/logstash-liveness-probe            0.1.8               2a484870ce6c        10 months ago       78.9MB
ibmcom/gluster                            3.12.1              0a644a8465a6        10 months ago       347MB
ibmcom/prom-statsd-exporter               v0.5.0              b733ba1f2a9d        10 months ago       14.1MB
ibmcom/prometheus                         v2.0.0              67141fa03496        11 months ago       80.2MB
ibmcom/keepalived                         1.2.24              21adeb592386        11 months ago       4.68MB
ibmcom/ucarp                              1.5.2               d6f29a895ed9        13 months ago       7.67MB
ibmcom/curl                               3.6                 f73fee23ac74        14 months ago       5.35MB
ibmcom/kibana                             5.5.1               7bf53f710f44        14 months ago       629MB
ibmcom/filebeat                           5.5.1               af80995a4fcf        14 months ago       271MB
ibmcom/logstash                           5.5.1               e8f5c2ec30f2        14 months ago       577MB
ibmcom/elasticsearch                      5.5.1               74ef44f69db6        14 months ago       544MB
ibmcom/heketi                             5                   5851537645df        14 months ago       343MB
ibmcom/heapster                           v1.4.0              749531a6d2cf        15 months ago       73.4MB
ibmcom/configmap-reload                   v0.1                b70d7dba98e6        2 years ago         4.78MB
ibmcom/defaultbackend                     1.2                 4ac4774c2764        2 years ago         6.94MB
ibmcom/collectd-exporter                  0.3.1               8853888e7b44        2 years ago         13.8MB
ibmcom/pause                              3.0                 99e59f495ffa        2 years ago         747kB
jesusmah commented 6 years ago

This also happened on an ICP 3.1 installation. The reason seems to be that there is only 50GB allocated to the root filesystem.

jesusmah commented 6 years ago

Latest Ubuntu template called ubu_1604_31 comes with a big enough disk to be able to handle the ICP installation tarball file. Anyway, the problem was that for the 2.1.0.3 vmware terraform code we were not partitioning the /dev/sdb disk that docker uses (because the storage driver was Overlay2 rather than devicemapper) ending up on the no disk space situation. The vmware terraform 3.1 code in the latest PR https://github.com/ibm-cloud-architecture/terraform-icp-vmware/pull/31 fixes the problem. However the problem still remains for 2.1.0.3 code...