search

datopian / ckan-cloud-helm

CKAN on Kubernetes (k8s) - Helm charts
https://tech.datopian.com/
MIT License
20 stars 10 forks source link

create_instance - ckan stuck on init phase #5

Closed akariv closed 5 years ago

akariv commented 5 years ago

create-instance log:

Error from server (NotFound): namespaces "demo-instance-61" not found
Error: getting deployed release "ckan-cloud-demo-instance-61": release: "ckan-cloud-demo-instance-61" not found
Creating instance: demo-instance-61
Setting CNAME from Cloudflare record cloud-demo-instance-61.ckan.io to hostname aef5bb582e64a11e89b330a0ee23ee90-659176021.us-west-2.elb.amazonaws.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   614    0   614    0     0   3336      0 --:--:-- --:--:-- --:--:--  3318
100 14604    0 14604    0     0  79369      0 --:--:-- --:--:-- --:--:-- 78940
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   135    0   135    0     0    798      0 --:--:-- --:--:-- --:--:--   798
Creating record name cloud-demo-instance-61.ckan.io in zone name ckan.io
zone id = 2e6c9eda70bfeb2d673f205cbcc64a56
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   683    0   526  100   157   1691    504 --:--:-- --:--:-- --:--:--  2196
Great Success!
Created DNS record cloud-demo-instance-61.ckan.io with hostname aef5bb582e64a11e89b330a0ee23ee90-659176021.us-west-2.elb.amazonaws.com
configmap "etc-traefik" deleted
configmap/etc-traefik created
deployment.extensions/traefik patched
Waiting for deployment "traefik" rollout to finish: 1 old replicas are pending termination...
Error from server (NotFound): namespaces "demo-instance-61" not found
Creating namespace: demo-instance-61
namespace/demo-instance-61 created
erviceaccount/ckan-demo-instance-61-operator created
role.rbac.authorization.k8s.io/ckan-demo-instance-61-operator-role created
rolebinding.rbac.authorization.k8s.io/ckan-demo-instance-61-operator-rolebinding created
initializing centralized infrastructure
Verifying ckan-infra secret on namespace demo-instance-61
Error from server (NotFound): secrets "ckan-infra" not found
creating ckan-infra secret
ecret/ckan-infra created
Verifying solrcloud collection demo-instance-61 on solrcloud pod solr-574bc96dfc-mm9rm in namespace ckan-cloud
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   203  100   203    0     0  34855      0 --:--:-- --:--:-- --:--:-- 40600
creating solrcloud collection

Connecting to ZooKeeper at localhost:9983 ...
INFO  - 2018-12-09 18:08:27.110; org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider; Cluster at localhost:9983 ready
Re-using existing configuration directory ckan_default

Creating new collection 'demo-instance-61' using command:
http://localhost:8983/solr/admin/collections?action=CREATE&name=demo-instance-61&numShards=1&replicationFactor=1&maxShardsPerNode=1&collection.configName=ckan_default

{
  "responseHeader":{
    "status":0,
    "QTime":2518},
  "success":{"192.168.208.226:8983_solr":{
      "responseHeader":{
        "status":0,
        "QTime":1396},
      "core":"demo-instance-61_shard1_replica1"}}}
centralized infrastructure initialized successfully
Deploying CKAN instance:
Initializing ckan-cloud Helm repo https://raw.githubusercontent.com/ViderumGlobal/ckan-cloud-helm/master/charts_repository
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
"ckan-cloud" has been added to your repositories
Using latest stable ckan chart
[debug] Created tunnel using local port: '36173'

[debug] SERVER: "127.0.0.1:36173"

[debug] Fetched ckan-cloud/ckan to /root/.helm/cache/archive/ckan-v0.0.6.tgz

Release "ckan-cloud-demo-instance-61" does not exist. Installing it now.
[debug] CHART PATH: /root/.helm/cache/archive/ckan-v0.0.6.tgz

NAME:   ckan-cloud-demo-instance-61
REVISION: 1
RELEASED: Sun Dec  9 18:08:31 2018
CHART: ckan-v0.0.6
USER-SUPPLIED VALUES:
authAnon_create_dataset: "false"
authCreate_dataset_if_not_in_organization: "true"
authCreate_unowned_dataset: "false"
authCreate_user_via_api: "false"
authCreate_user_via_web: "true"
authRoles_that_cascade_to_sub_groups: admin
authUser_create_groups: "false"
authUser_create_organizations: "false"
authUser_delete_groups: "false"
authUser_delete_organizations: "false"
centralizedSecretName: ckan-infra
ckanAdminEmail: adam.kariv@gmail.com
ckanGunicornWorkers: 2
ckanImage: viderum/ckan-cloud-docker:ckan-latest-datagov-theme
ckanJobsDbResources: '{"requests": {"cpu": "20m", "memory": "50Mi"}, "limits": {"memory":
  "200Mi"}}'
ckanJobsDbTerminationGracePeriodSeconds: 30
ckanJobsResources: '{"requests": {"cpu": "150m", "memory": "600Mi"}, "limits": {"memory":
  "1500Mi"}}'
ckanJobsTerminationGracePeriodSeconds: 30
ckanOperatorImage: viderum/ckan-cloud-docker:cca-operator-4e29e20e71364c7fc161b612dc71291a27090bff
ckanPlugins: stats text_view image_view recline_view datastore xloader harvest ckan_harvester
  datajson_harvest datajson archiver report dcat dcat_rdf_harvester dcat_json_harvester
  dcat_json_interface structured_data geodatagov geodatagov_miscs z3950_harvester
  arcgis_harvester geodatagov_geoportal_harvester waf_harvester_collection geodatagov_csw_harvester
  geodatagov_doc_harvester geodatagov_waf_harvester spatial_metadata spatial_query
ckanResources: '{"requests": {"cpu": "400m", "memory": "1200Mi"}, "limits": {"memory":
  "2Gi"}}'
ckanStorageClassName: cca-ckan
dataStoreDbPersistentDiskSizeGB: 20
datastoreDbResources: '{"requests": {"cpu": "100m", "memory": "300Mi"}, "limits":
  {"memory": "600Mi"}}'
datastoreDbTerminationGracePeriodSeconds: 30
dbPersistentDiskSizeGB: 20
dbResources: '{"requests": {"cpu": "100m", "memory": "300Mi"}, "limits": {"memory":
  "600Mi"}}'
dbTerminationGracePeriodSeconds: 30
disableJobs: true
displayTimezone: America/New_York
domain: cloud-demo-instance-61.ckan.io
extraCkanConfig: |-
  ## datajson settings
  ckanext.datajson.inventory_links_enabled = True
  ckanext.datajson.url_enabled = False
  ## Datagov settings
  ckanext.geodatagov.dynamic_menu.url_default = http://www.data.gov/app/plugins/datagov-custom/wp_download_links.php
  ckanext.geodatagov.dynamic_menu.url =
  ckanext.geodatagov.bureau_csv.url_default = https://project-open-data.cio.gov/data/omb_bureau_codes.csv
  ckanext.geodatagov.bureau_csv.url = https://project-open-data.cio.gov/data/omb_bureau_codes.csv
  ## Harvest settings
  ckan.harvest.mq.type = redis
  ckan.harvest.mq.hostname = redis
  ckan.harvest.mq.redis_db = 9
favIcon: /base/images/ckan.ico
id: demo-instance-61
jobsDbPersistentDiskSizeGB: 5
kind: datapusher
nginxReplicas: 1
nginxResources: '{"requests": {"cpu": "60m", "memory": "300Mi"}, "limits": {"memory":
  "800Mi"}}'
noProbes: true
provisioningKindTitle: DataPusher / Catalog
redisResources: '{"requests": {"cpu": "15m", "memory": "100Mi"}, "limits": {"memory":
  "300Mi"}}'
registerSubdomain: cloud-demo-instance-61
replicas: 1
iteDescription: Staging environment
iteLogo: /base/images/ckan-logo.png
iteTitle: demo-instance-61 title
iteUrl: https://cloud-demo-instance-61.ckan.io
olrPersistentDiskSizeGB: 10
olrResources: '{"requests": {"cpu": "60m", "memory": "600Mi"}, "limits": {"memory":
  "1Gi"}}'
torageClassName: cca-storage
terminationGracePeriodSeconds: 30
useCentralizedInfra: true
usePersistentVolumes: true
withSansSSL: true

COMPUTED VALUES:
authAnon_create_dataset: "false"
authCreate_dataset_if_not_in_organization: "true"
authCreate_unowned_dataset: "false"
authCreate_user_via_api: "false"
authCreate_user_via_web: "true"
authRoles_that_cascade_to_sub_groups: admin
authUser_create_groups: "false"
authUser_create_organizations: "false"
authUser_delete_groups: "false"
authUser_delete_organizations: "false"
centralizedSecretName: ckan-infra
ckanAdminEmail: adam.kariv@gmail.com
ckanGunicornWorkers: 2
ckanImage: viderum/ckan-cloud-docker:ckan-latest-datagov-theme
ckanJobsDbResources: '{"requests": {"cpu": "20m", "memory": "50Mi"}, "limits": {"memory":
  "200Mi"}}'
ckanJobsDbTerminationGracePeriodSeconds: 30
ckanJobsResources: '{"requests": {"cpu": "150m", "memory": "600Mi"}, "limits": {"memory":
  "1500Mi"}}'
ckanJobsTerminationGracePeriodSeconds: 30
ckanOperatorImage: viderum/ckan-cloud-docker:cca-operator-4e29e20e71364c7fc161b612dc71291a27090bff
ckanPlugins: stats text_view image_view recline_view datastore xloader harvest ckan_harvester
  datajson_harvest datajson archiver report dcat dcat_rdf_harvester dcat_json_harvester
  dcat_json_interface structured_data geodatagov geodatagov_miscs z3950_harvester
  arcgis_harvester geodatagov_geoportal_harvester waf_harvester_collection geodatagov_csw_harvester
  geodatagov_doc_harvester geodatagov_waf_harvester spatial_metadata spatial_query
ckanResources: '{"requests": {"cpu": "400m", "memory": "1200Mi"}, "limits": {"memory":
  "2Gi"}}'
ckanSecretName: ckan-secrets
ckanStorageClassName: cca-ckan
dataStoreDbPersistentDiskSizeGB: 20
datastoreDbResources: '{"requests": {"cpu": "100m", "memory": "300Mi"}, "limits":
  {"memory": "600Mi"}}'
datastoreDbTerminationGracePeriodSeconds: 30
dbImage: viderum/ckan-cloud-docker:db-v0.0.6
dbPersistentDiskSizeGB: 20
dbResources: '{"requests": {"cpu": "100m", "memory": "300Mi"}, "limits": {"memory":
  "600Mi"}}'
dbTerminationGracePeriodSeconds: 30
disableJobs: true
displayTimezone: America/New_York
domain: cloud-demo-instance-61.ckan.io
envVarsSecretName: ckan-env-vars
extraCkanConfig: |-
  ## datajson settings
  ckanext.datajson.inventory_links_enabled = True
  ckanext.datajson.url_enabled = False
  ## Datagov settings
  ckanext.geodatagov.dynamic_menu.url_default = http://www.data.gov/app/plugins/datagov-custom/wp_download_links.php
  ckanext.geodatagov.dynamic_menu.url =
  ckanext.geodatagov.bureau_csv.url_default = https://project-open-data.cio.gov/data/omb_bureau_codes.csv
  ckanext.geodatagov.bureau_csv.url = https://project-open-data.cio.gov/data/omb_bureau_codes.csv
  ## Harvest settings
  ckan.harvest.mq.type = redis
  ckan.harvest.mq.hostname = redis
  ckan.harvest.mq.redis_db = 9
favIcon: /base/images/ckan.ico
id: demo-instance-61
jobsDbPersistentDiskSizeGB: 5
kind: datapusher
nginxImage: viderum/ckan-cloud-docker:nginx-v0.0.6
nginxReplicas: 1
nginxResources: '{"requests": {"cpu": "60m", "memory": "300Mi"}, "limits": {"memory":
  "800Mi"}}'
noProbes: true
provisioningKindTitle: DataPusher / Catalog
redisResources: '{"requests": {"cpu": "15m", "memory": "100Mi"}, "limits": {"memory":
  "300Mi"}}'
registerSubdomain: cloud-demo-instance-61
replicas: 1
iteDescription: Staging environment
iteLogo: /base/images/ckan-logo.png
iteTitle: demo-instance-61 title
iteUrl: https://cloud-demo-instance-61.ckan.io
olrImage: viderum/ckan-cloud-docker:solr-v0.0.6
olrPersistentDiskSizeGB: 10
olrResources: '{"requests": {"cpu": "60m", "memory": "600Mi"}, "limits": {"memory":
  "1Gi"}}'
olrcloudImage: viderum/ckan-cloud-docker:solrcloud-v0.0.6
torageClassName: cca-storage
terminationGracePeriodSeconds: 30
useCentralizedInfra: true
usePersistentVolumes: true
withSansSSL: true

HOOKS:
MANIFEST:

---
# Source: ckan/templates/ckan-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ckan
data:
  who.ini: |
    [plugin:auth_tkt]
    use = ckan.lib.auth_tkt:make_plugin

    [plugin:friendlyform]
    use = repoze.who.plugins.friendlyform:FriendlyFormPlugin
    login_form_url= /user/login
    login_handler_path = /login_generic
    logout_handler_path = /user/logout
    rememberer_name = auth_tkt
    post_login_url = /user/logged_in
    post_logout_url = /user/logged_out
    charset = utf-8

    [general]
    request_classifier = repoze.who.classifiers:default_request_classifier
    challenge_decider = repoze.who.classifiers:default_challenge_decider

    [identifiers]
    plugins =
        friendlyform;browser
        auth_tkt

    [authenticators]
    plugins =
        auth_tkt
        ckan.lib.authenticator:UsernamePasswordAuthenticator

    [challengers]
    plugins =
        friendlyform;browser
  production.ini.template: |
    [DEFAULT]
    debug = false

    [server:main]
    use = egg:Paste#http
    host = 0.0.0.0
    port = 5000

    [app:main]
    use = egg:ckan
    full_stack = true
    cache_dir = /tmp/%%(ckan.site_id)s/
    beaker.session.key = ckan

    beaker.session.secret = {{BEAKER_SESSION_SECRET}}
    app_instance_uuid = {{APP_INSTANCE_UUID}}

    who.config_file = %%(here)s/who.ini
    who.log_level = warning
    who.log_file = %%(cache_dir)s/who_log.ini
    # Session timeout (user logged out after period of inactivity, in seconds).
    # Inactive by default, so the session doesn't expire.
    # who.timeout = 86400

    ## Database Settings
    sqlalchemy.url = {{SQLALCHEMY_URL}}

    ckan.datastore.write_url = {{CKAN_DATASTORE_WRITE_URL}}
    ckan.datastore.read_url = {{CKAN_DATASTORE_READ_URL}}

    # PostgreSQL' full-text search parameters
    ckan.datastore.default_fts_lang = english
    ckan.datastore.default_fts_index_method = gist

    ## Site Settings

    ckan.site_url = https://cloud-demo-instance-61.ckan.io
    #ckan.use_pylons_response_cleanup_middleware = true

    ## Authorization Settings

    ckan.auth.anon_create_dataset = false
    ckan.auth.create_unowned_dataset = false
    ckan.auth.create_dataset_if_not_in_organization = true
    ckan.auth.user_create_groups = false
    ckan.auth.user_create_organizations = false
    ckan.auth.user_delete_groups = false
    ckan.auth.user_delete_organizations = false
    ckan.auth.create_user_via_api = false
    ckan.auth.create_user_via_web = true
    ckan.auth.roles_that_cascade_to_sub_groups = admin

    ## Search Settings

    ckan.site_id = default
    solr_url = {{SOLR_URL}}

    ## Redis Settings

    # URL to your Redis instance, including the database to be used.
    ckan.redis.url = {{CKAN_REDIS_URL}}

    ## CORS Settings

    # If cors.origin_allow_all is true, all origins are allowed.
    # If false, the cors.origin_whitelist is used.
    # ckan.cors.origin_allow_all = true
    # cors.origin_whitelist is a space separated list of allowed domains.
    # ckan.cors.origin_whitelist = http://example1.com http://example2.com

    ## Plugins Settings

    #       Add ``resource_proxy`` to enable resorce proxying and get around the
    #       same origin policy
    ckan.plugins = stats text_view image_view recline_view datastore xloader harvest ckan_harvester datajson_harvest datajson archiver report dcat dcat_rdf_harvester dcat_json_harvester dcat_json_interface structured_data geodatagov geodatagov_miscs z3950_harvester arcgis_harvester geodatagov_geoportal_harvester waf_harvester_collection geodatagov_csw_harvester geodatagov_doc_harvester geodatagov_waf_harvester spatial_metadata spatial_query

    # Define which views should be created by default
    # (plugins must be loaded in ckan.plugins)
    ckan.views.default_views = image_view text_view recline_view

    # Customize which text formats the text_view plugin will show
    #ckan.preview.json_formats = json
    #ckan.preview.xml_formats = xml rdf rdf+xml owl+xml atom rss
    #ckan.preview.text_formats = text plain text/plain

    # Customize which image formats the image_view plugin will show
    #ckan.preview.image_formats = png jpeg jpg gif

    ## Front-End Settings

    # Uncomment following configuration to enable using of Bootstrap 2
    #ckan.base_public_folder = public-bs2
    #ckan.base_templates_folder = templates-bs2

    ckan.site_title = demo-instance-61 title
    ckan.site_logo = /base/images/ckan-logo.png
    ckan.site_description = Staging environment
    ckan.favicon = /base/images/ckan.ico
    ckan.gravatar_default = identicon
    ckan.preview.direct = png jpg gif
    ckan.preview.loadable = html htm rdf+xml owl+xml xml n3 n-triples turtle plain atom csv tsv rss txt json
    ckan.display_timezone = America/New_York

    # package_hide_extras = for_search_index_only
    #package_edit_return_url = http://another.frontend/dataset/<NAME>
    #package_new_return_url = http://another.frontend/dataset/<NAME>
    #ckan.recaptcha.publickey =
    #ckan.recaptcha.privatekey =
    #licenses_group_url = http://licenses.opendefinition.org/licenses/groups/ckan.json
    # ckan.template_footer_end =

    ## Internationalisation Settings
    #    ckan.locale_default = 
    #    ckan.locale_order = 
    #    ckan.locales_offered = 
    #    ckan.locales_filtered_out = 

    ## Feeds Settings

    ckan.feeds.authority_name =
    ckan.feeds.date =
    ckan.feeds.author_name =
    ckan.feeds.author_link =

    ## Storage Settings

    ckan.storage_path = /var/lib/ckan
    ckan.max_resource_size = 100
    ckan.max_image_size = 5

    ## xloader settings

    ckanext.xloader.jobs_db.uri = postgresql://postgres:123456@jobs-db/postgres

    # Resource Proxy settings
    # Preview size limit, default: 1MB
    #ckan.resource_proxy.max_file_size = 1048576
    # Size of chunks to read/write.
    #ckan.resource_proxy.chunk_size = 4096

    ## Activity Streams Settings

    #ckan.activity_streams_enabled = true
    #ckan.activity_list_limit = 31
    #ckan.activity_streams_email_notifications = true
    #ckan.email_notifications_since = 2 days
    ckan.hide_activity_from_users = %%(ckan.site_id)s

    ## Email settings

    #email_to = errors@example.com
    #error_email_from = ckan-errors@example.com
    smtp.server = {{SMTP_SERVER}}
    smtp.starttls = {{SMTP_STARTTLS}}
    smtp.user = {{SMTP_USER}}
    smtp.password = {{SMTP_PASSWORD}}
    smtp.mail_from = {{SMTP_MAIL_FROM}}

    ## datajson settings
    ckanext.datajson.inventory_links_enabled = True
    ckanext.datajson.url_enabled = False
    ## Datagov settings
    ckanext.geodatagov.dynamic_menu.url_default = http://www.data.gov/app/plugins/datagov-custom/wp_download_links.php
    ckanext.geodatagov.dynamic_menu.url =
    ckanext.geodatagov.bureau_csv.url_default = https://project-open-data.cio.gov/data/omb_bureau_codes.csv
    ckanext.geodatagov.bureau_csv.url = https://project-open-data.cio.gov/data/omb_bureau_codes.csv
    ## Harvest settings
    ckan.harvest.mq.type = redis
    ckan.harvest.mq.hostname = redis
    ckan.harvest.mq.redis_db = 9

    ## Logging configuration
    [loggers]
    keys = root, ckan, ckanext

    [handlers]
    keys = console

    [formatters]
    keys = generic

    [logger_root]
    level = WARNING
    handlers = console

    [logger_ckan]
    level = INFO
    handlers = console
    qualname = ckan
    propagate = 0

    [logger_ckanext]
    level = DEBUG
    handlers = console
    qualname = ckanext
    propagate = 0

    [handler_console]
    class = StreamHandler
    args = (sys.stderr,)
    level = NOTSET
    formatter = generic

    [formatter_generic]
    format = %%(asctime)s %%(levelname)-5.5s [%%(name)s] %%(message)s
  ckan_init.sh.template: |
    #!/usr/bin/env bash
    echo Waiting for ckan infrastructure...
    while sleep 2; do
        psql -d {{SQLALCHEMY_URL}} -c "select 1" -t | cat &&\
        psql -d {{CKAN_DATASTORE_WRITE_URL}} -c "select 1" -t | cat &&\
        wget -qO - {{SOLR_URL}}/schema/version &&\
        redis-cli -h redis ping &&\
        exit 0
        echo .
    done
    exit 1
---
# Source: ckan/templates/ckan-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-instance-61-ckan
pec:
  storageClassName: cca-ckan
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Mi
---
# Source: ckan/templates/jobs-db-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: demo-instance-61-jobs-db
pec:
  storageClassName: cca-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5G
---
# Source: ckan/templates/ckan-service.yaml
apiVersion: v1
kind: Service
metadata: {name: ckan}
pec:
  ports:
  - {name: '5000', port: 5000}
  selector: {app: ckan}
---
# Source: ckan/templates/jobs-db-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: jobs-db
pec:
  selector:
    app: jobs-db
  ports:
  - name: "5432"
    port: 5432
---
# Source: ckan/templates/nginx-service.yaml
apiVersion: v1
kind: Service
metadata: {name: nginx}
pec:
  ports:
  - {name: '8080', port: 8080}
  selector: {app: nginx}
---
# Source: ckan/templates/redis-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: redis
pec:
  selector:
    app: redis
  ports:
  - name: "6379"
    port: 6379
---
# Source: ckan/templates/ckan-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata: {name: ckan}
pec:
  replicas: 1
  revisionHistoryLimit: 2
  template:
    metadata:
      labels: {app: ckan}
      annotations:
        checksum/configmap: 8396b577265e947df4219ef61a3b41f3a0b688037a92ef9756c4bea5e09beca9
    spec:

      terminationGracePeriodSeconds: 30

      serviceAccountName: ckan-demo-instance-61-operator
      initContainers:
      - name: secrets
        image: "viderum/ckan-cloud-docker:cca-operator-4e29e20e71364c7fc161b612dc71291a27090bff"
        command:
        - bash
        - "-c"
        - |
          ./cca-operator.sh initialize-ckan-env-vars "ckan-env-vars" &&\
          ./cca-operator.sh initialize-ckan-secrets "ckan-env-vars" \
                                                    "ckan-secrets" &&\
          ./cca-operator.sh get-ckan-secrets "ckan-secrets" /etc/ckan-conf/secrets/secrets.sh &&\
          echo ckan secrets initialized successfully && exit 0
          echo failed to initialize, waiting 1 hour before exiting && sleep 3600 && exit 1

        env:
        - name: CKAN_CLOUD_INSTANCE_ID

          value: "demo-instance-61"

        - name: CKAN_CLOUD_POSTGRES_HOST
          valueFrom:
            secretKeyRef:
              name: "ckan-infra"
              key: "POSTGRES_HOST"
        - name: CKAN_CLOUD_POSTGRES_USER
          valueFrom:
            secretKeyRef:
              name: "ckan-infra"
              key: "POSTGRES_USER"
        - name: PGPASSWORD
          valueFrom:
            secretKeyRef:
              name: "ckan-infra"
              key: "POSTGRES_PASSWORD"

        - name: CKAN_CLOUD_SOLR_HOST
          value: "solr.ckan-cloud"
        - name: CKAN_CLOUD_SOLR_PORT
          value: "8983"

        volumeMounts:
        - name: ckan-conf-secrets
          mountPath: /etc/ckan-conf/secrets
      - name: files
        image: alpine
        command:
        - sh
        - "-c"
        - |
          chown -R 900:900 /var/lib/ckan && chmod -R ug+w /var/lib/ckan && chown -R 900:900 /etc/ckan-conf/secrets &&\
          ls -lah /var/lib/ckan && ls -lah /etc/ckan-conf/secrets
        securityContext:
          runAsUser: 0
        volumeMounts:
        - {mountPath: /var/lib/ckan, name: ckan-data, subPath: demo-instance-61-ckan}
        - name: ckan-conf-secrets
          mountPath: /etc/ckan-conf/secrets

      containers:
      - name: ckan
        image: "viderum/ckan-cloud-docker:ckan-latest-datagov-theme"
        env:
        - name: CKAN_STORAGE_PATH
          value: /var/lib/ckan/data
        - name: CKAN_K8S_SECRETS
          value: /etc/ckan-conf/secrets/secrets.sh
        - name: CKAN_K8S_TEMPLATES
          value: /etc/ckan-conf/templates
        - name: GUNICORN_WORKERS
          value: "2"
        ports:
        - containerPort: 5000

        resources: {"requests": {"cpu": "400m", "memory": "1200Mi"}, "limits": {"memory": "2Gi"}}

        volumeMounts:
        - {mountPath: /var/lib/ckan, name: ckan-data, subPath: demo-instance-61-ckan}
        - name: ckan-conf-secrets
          mountPath: /etc/ckan-conf/secrets
          readOnly: true
        - name: ckan-conf-templates
          mountPath: /etc/ckan-conf/templates
          readOnly: true

      volumes:
      - name: ckan-conf-secrets
        emptyDir: {}
      - name: ckan-conf-templates
        configMap:
          name: ckan
      - name: ckan-data

        persistentVolumeClaim:
          claimName: demo-instance-61-ckan
---
# Source: ckan/templates/jobs-db-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata: {name: jobs-db}
pec:
  replicas: 1
  revisionHistoryLimit: 2
  strategy:
    type: Recreate
  template:
    metadata:
      labels: {app: jobs-db}
    spec:

      terminationGracePeriodSeconds: 30

      containers:
      - name: postgres
        image: postgres
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_PASSWORD
          value: "123456"

        resources: {"requests": {"cpu": "20m", "memory": "50Mi"}, "limits": {"memory": "200Mi"}}

        volumeMounts:
        - mountPath: /var/lib/postgresql/data
          name: ckan-data
          subPath: ckan-jobs-db
      volumes:
      - name: ckan-data

        persistentVolumeClaim:
          claimName: demo-instance-61-jobs-db
---
# Source: ckan/templates/nginx-deployment.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx
pec:
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      terminationGracePeriodSeconds: 1
      containers:
      - name: nginx
        image: "viderum/ckan-cloud-docker:nginx-v0.0.6"
        ports:
        - containerPort: 8080

        resources: {"requests": {"cpu": "60m", "memory": "300Mi"}, "limits": {"memory": "800Mi"}}
---
# Source: ckan/templates/redis-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: redis
pec:
  replicas: 1
  revisionHistoryLimit: 2
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:alpine
        ports:
        - containerPort: 6379

        resources: {"requests": {"cpu": "15m", "memory": "100Mi"}, "limits": {"memory": "300Mi"}}
Release "ckan-cloud-demo-instance-61" does not exist. Installing it now.
NAME:   ckan-cloud-demo-instance-61
LAST DEPLOYED: Sun Dec  9 18:08:32 2018
NAMESPACE: demo-instance-61
STATUS: DEPLOYED

RESOURCES:
==> v1beta1/Deployment
NAME     AGE
ckan     1s
jobs-db  1s
nginx    1s
redis    1s

==> v1/Pod(related)

NAME                      READY  STATUS             RESTARTS  AGE
ckan-5b4c94b657-pp484     0/1    Pending            0         1s
jobs-db-765dd49b5d-tb2c6  0/1    Pending            0         1s
nginx-65f7bf4c6-r4w88     0/1    ContainerCreating  0         1s
redis-7ccd7b468b-j9ggn    0/1    ContainerCreating  0         1s

==> v1/ConfigMap

NAME  AGE
ckan  1s

==> v1/PersistentVolumeClaim
demo-instance-61-ckan     1s
demo-instance-61-jobs-db  1s

==> v1/Service
ckan     1s
jobs-db  1s
nginx    1s
redis    1s

ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS              RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Pending             0          1s
jobs-db-765dd49b5d-tb2c6   0/1     ContainerCreating   0          1s
nginx-65f7bf4c6-r4w88      0/1     ContainerCreating   0          1s
redis-7ccd7b468b-j9ggn     0/1     ContainerCreating   0          1s
...3s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS              RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Pending             0          5s
jobs-db-765dd49b5d-tb2c6   0/1     ContainerCreating   0          5s
nginx-65f7bf4c6-r4w88      1/1     Running             0          5s
redis-7ccd7b468b-j9ggn     1/1     Running             0          5s
...6s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS              RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Pending             0          9s
jobs-db-765dd49b5d-tb2c6   0/1     ContainerCreating   0          9s
nginx-65f7bf4c6-r4w88      1/1     Running             0          9s
redis-7ccd7b468b-j9ggn     1/1     Running             0          9s
...9s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS              RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Pending             0          12s
jobs-db-765dd49b5d-tb2c6   0/1     ContainerCreating   0          12s
nginx-65f7bf4c6-r4w88      1/1     Running             0          12s
redis-7ccd7b468b-j9ggn     1/1     Running             0          12s
...12s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS              RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2            0          16s
jobs-db-765dd49b5d-tb2c6   0/1     ContainerCreating   0          16s
nginx-65f7bf4c6-r4w88      1/1     Running             0          16s
redis-7ccd7b468b-j9ggn     1/1     Running             0          16s
...15s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS              RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2            0          20s
jobs-db-765dd49b5d-tb2c6   0/1     ContainerCreating   0          20s
nginx-65f7bf4c6-r4w88      1/1     Running             0          20s
redis-7ccd7b468b-j9ggn     1/1     Running             0          20s
...18s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          23s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          23s
nginx-65f7bf4c6-r4w88      1/1     Running    0          23s
redis-7ccd7b468b-j9ggn     1/1     Running    0          23s
...21s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          27s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          27s
nginx-65f7bf4c6-r4w88      1/1     Running    0          27s
redis-7ccd7b468b-j9ggn     1/1     Running    0          27s
...24s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          30s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          30s
nginx-65f7bf4c6-r4w88      1/1     Running    0          30s
redis-7ccd7b468b-j9ggn     1/1     Running    0          30s
...27s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          34s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          34s
nginx-65f7bf4c6-r4w88      1/1     Running    0          34s
redis-7ccd7b468b-j9ggn     1/1     Running    0          34s
...30s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          38s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          38s
nginx-65f7bf4c6-r4w88      1/1     Running    0          38s
redis-7ccd7b468b-j9ggn     1/1     Running    0          38s
...33s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          41s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          41s
nginx-65f7bf4c6-r4w88      1/1     Running    0          41s
redis-7ccd7b468b-j9ggn     1/1     Running    0          41s
...36s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          45s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          45s
nginx-65f7bf4c6-r4w88      1/1     Running    0          45s
redis-7ccd7b468b-j9ggn     1/1     Running    0          45s
...39s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          49s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          49s
nginx-65f7bf4c6-r4w88      1/1     Running    0          49s
redis-7ccd7b468b-j9ggn     1/1     Running    0          49s
...42s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          53s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          53s
nginx-65f7bf4c6-r4w88      1/1     Running    0          53s
redis-7ccd7b468b-j9ggn     1/1     Running    0          53s
...45s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          56s
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          56s
nginx-65f7bf4c6-r4w88      1/1     Running    0          56s
redis-7ccd7b468b-j9ggn     1/1     Running    0          56s
...48s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...51s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...54s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...57s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...60s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...63s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...66s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...69s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...72s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...75s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...78s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...81s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...84s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...87s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m
...90s
ckan-5b4c94b657-pp484: Pending
NAME                       READY   STATUS     RESTARTS   AGE
ckan-5b4c94b657-pp484      0/1     Init:0/2   0          1m
jobs-db-765dd49b5d-tb2c6   1/1     Running    0          1m
nginx-65f7bf4c6-r4w88      1/1     Running    0          1m
redis-7ccd7b468b-j9ggn     1/1     Running    0          1m

Inspecting the ckan pod:

bash-4.3# kubectl -n demo-instance-61 describe pods ckan-5b4c94b657-pp484
Name:           ckan-5b4c94b657-pp484
Namespace:      demo-instance-61
Node:           ip-192-168-64-173.us-west-2.compute.internal/192.168.64.173
Start Time:     Sun, 09 Dec 2018 18:08:47 +0000
Labels:         app=ckan
                pod-template-hash=1607506213
Annotations:    checksum/configmap: 8396b577265e947df4219ef61a3b41f3a0b688037a92ef9756c4bea5e09beca9
Status:         Pending
IP:             192.168.103.252
Controlled By:  ReplicaSet/ckan-5b4c94b657
Init Containers:
  secrets:
    Container ID:  docker://0098f7b1859f076e04c278607c983a6aea2d07472d56e3317cc772afbac30016
    Image:         viderum/ckan-cloud-docker:cca-operator-4e29e20e71364c7fc161b612dc71291a27090bff
    Image ID:      docker-pullable://viderum/ckan-cloud-docker@sha256:f2d7d2edc6c3508823d49074d2f444733dee2f731803b125d0fce611f40acbbd
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -c
      ./cca-operator.sh initialize-ckan-env-vars "ckan-env-vars" &&\
      ./cca-operator.sh initialize-ckan-secrets "ckan-env-vars" \
                                                "ckan-secrets" &&\
      ./cca-operator.sh get-ckan-secrets "ckan-secrets" /etc/ckan-conf/secrets/secrets.sh &&\
      echo ckan secrets initialized successfully && exit 0
      echo failed to initialize, waiting 1 hour before exiting && sleep 3600 && exit 1

    State:          Running
      Started:      Sun, 09 Dec 2018 18:08:50 +0000
    Ready:          False
    Restart Count:  0
    Environment:
      CKAN_CLOUD_INSTANCE_ID:    demo-instance-61
      CKAN_CLOUD_POSTGRES_HOST:  <set to the key 'POSTGRES_HOST' in secret 'ckan-infra'>      Optional: false
      CKAN_CLOUD_POSTGRES_USER:  <set to the key 'POSTGRES_USER' in secret 'ckan-infra'>      Optional: false
      PGPASSWORD:                <set to the key 'POSTGRES_PASSWORD' in secret 'ckan-infra'>  Optional: false
      CKAN_CLOUD_SOLR_HOST:      solr.ckan-cloud
      CKAN_CLOUD_SOLR_PORT:      8983
    Mounts:
      /etc/ckan-conf/secrets from ckan-conf-secrets (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from ckan-demo-instance-61-operator-token-j6fnv (ro)
  files:
    Container ID:
    Image:         alpine
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      chown -R 900:900 /var/lib/ckan && chmod -R ug+w /var/lib/ckan && chown -R 900:900 /etc/ckan-conf/secrets &&\
      ls -lah /var/lib/ckan && ls -lah /etc/ckan-conf/secrets

    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ckan-conf/secrets from ckan-conf-secrets (rw)
      /var/lib/ckan from ckan-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from ckan-demo-instance-61-operator-token-j6fnv (ro)
Containers:
  ckan:
    Container ID:
    Image:          viderum/ckan-cloud-docker:ckan-latest-datagov-theme
    Image ID:
    Port:           5000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  2Gi
    Requests:
      cpu:     400m
      memory:  1200Mi
    Environment:
      CKAN_STORAGE_PATH:   /var/lib/ckan/data
      CKAN_K8S_SECRETS:    /etc/ckan-conf/secrets/secrets.sh
      CKAN_K8S_TEMPLATES:  /etc/ckan-conf/templates
      GUNICORN_WORKERS:    2
    Mounts:
      /etc/ckan-conf/secrets from ckan-conf-secrets (ro)
      /etc/ckan-conf/templates from ckan-conf-templates (ro)
      /var/lib/ckan from ckan-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from ckan-demo-instance-61-operator-token-j6fnv (ro)
Conditions:
  Type           Status
  Initialized    False
  Ready          False
  PodScheduled   True
Volumes:
  ckan-conf-secrets:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  ckan-conf-templates:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ckan
    Optional:  false
  ckan-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  demo-instance-61-ckan
    ReadOnly:   false
  ckan-demo-instance-61-operator-token-j6fnv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ckan-demo-instance-61-operator-token-j6fnv
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                   From                                                   Message
  ----     ------                 ----                  ----                                                   -------
  Warning  FailedScheduling       3m59s (x5 over 4m6s)  default-scheduler                                      pod has unbound PersistentVolumeClaims (repeated 4 times)
  Normal   Scheduled              3m51s                 default-scheduler                                      Successfully assigned ckan-5b4c94b657-pp484 to ip-192-168-64-173.us-west-2.compute.internal
  Normal   SuccessfulMountVolume  3m51s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  MountVolume.SetUp succeeded for volume "ckan-conf-secrets"
  Normal   SuccessfulMountVolume  3m51s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  MountVolume.SetUp succeeded for volume "ckan-conf-templates"
  Normal   SuccessfulMountVolume  3m51s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  MountVolume.SetUp succeeded for volume "ckan-demo-instance-61-operator-token-j6fnv"
  Normal   SuccessfulMountVolume  3m51s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  MountVolume.SetUp succeeded for volume "pvc-70504f22-fbdd-11e8-8366-0618ac193f1a"
  Normal   Pulling                3m50s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  pulling image "viderum/ckan-cloud-docker:cca-operator-4e29e20e71364c7fc161b612dc71291a27090bff"
  Normal   Pulled                 3m48s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  Successfully pulled image "viderum/ckan-cloud-docker:cca-operator-4e29e20e71364c7fc161b612dc71291a27090bff"
  Normal   Created                3m48s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  Created container
  Normal   Started                3m48s                 kubelet, ip-192-168-64-173.us-west-2.compute.internal  Started container
OriHoch commented 5 years ago

the relevant error is in the secrets initContainer logs - 

Initializing datastore DB demo-instance-61-datastore on ckan-cloud.cokcn6iyrwpx.us-west-2.rds.amazonaws.com
CREATE ROLE
ALTER DATABASE
WARNING:  no privileges were granted for "spatial_ref_sys"
WARNING:  no privileges were granted for "geography_columns"
WARNING:  no privileges were granted for "geometry_columns"
WARNING:  no privileges were granted for "raster_columns"
WARNING:  no privileges were granted for "raster_overviews"
ALTER DEFAULT PRIVILEGES
CREATE VIEW
GRANT
CREATE FUNCTION
ERROR:  permission denied for relation spatial_ref_sys
CONTEXT:  SQL statement "CREATE TRIGGER zfulltext BEFORE INSERT OR UPDATE ON spatial_ref_sys FOR EACH ROW EXECUTE PROCEDURE populate_full_text_trigger();"
PL/pgSQL function inline_code_block line 3 at EXECUTE
Datastore DB initialization failed
failed to initialize, waiting 1 hour before exiting
OriHoch commented 5 years ago

I'll have a look tomorrow

OriHoch commented 5 years ago

there are some problems with the way we are waiting for CKAN instance to be ready:

  1. the initContainer restarts on failure (after 1 hour) - after restart it detects the secrets already created and then completes successfully, hiding the error and allowing CKAN to start
  2. we are checking CKAN ready status in create instance script to continue but it could be reported as ready and fail a few seconds later on init scripts

to solve this problems I'm implementing a cca-operator instance-health command which will report accurate status and health for the instance, will use it in create-instance script

akariv commented 5 years ago

Any idea why initContainer failed in the first place?

On Mon, Dec 10, 2018 at 10:19 AM Ori Hoch notifications@github.com wrote:

there are some problems with the way we are waiting for CKAN instance to be ready:

  1. the initContainer restarts on failure (after 1 hour) - after restart it detects the secrets already created and then completes successfully, hiding the error and allowing CKAN to start
  2. we are checking CKAN ready status in create instance script to continue but it could be reported as ready and fail a few seconds later on init scripts

to solve this problems I'm implementing a cca-operator instance-health command which will report accurate status and health for the instance, will use it in create-instance script

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ViderumGlobal/ckan-cloud-helm/issues/5#issuecomment-445728573, or mute the thread https://github.com/notifications/unsubscribe-auth/AAQMdSEs45kYTqte9_3DwIN_0I4pnMi1ks5u3hkLgaJpZM4ZKP0X .

OriHoch commented 5 years ago

i was lucky enough to get it before the 1 hour ended.. so the log I pasted above is the first error