search

StackStorm / stackstorm-k8s

K8s Helm Chart that codifies StackStorm (aka "IFTTT for Ops" https://stackstorm.com/) Highly Availability fleet as a simple to use reproducible infrastructure-as-code app
https://helm.stackstorm.com/
Apache License 2.0
105 stars 107 forks source link

Problem with nfs deploy on job job-st2-register-content #238

Closed elsopapa closed 3 years ago

elsopapa commented 3 years ago

I have a problem deploying with helm when using nfs , the job fail with "Job has reached the specified backoff limit"

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    helm.sh/hook: post-install, post-upgrade, post-rollback
    helm.sh/hook-delete-policy: before-hook-creation
    helm.sh/hook-weight: "7"
  creationTimestamp: "2021-09-03T12:10:00Z"
  labels:
    app: st2-register-content
    chart: stackstorm-ha-0.60.0
    heritage: Helm
    release: stackstorm
    tier: backend
    vendor: stackstorm
  managedFields:
  - apiVersion: batch/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:helm.sh/hook: {}
          f:helm.sh/hook-delete-policy: {}
          f:helm.sh/hook-weight: {}
        f:labels:
          .: {}
          f:app: {}
          f:chart: {}
          f:heritage: {}
          f:release: {}
          f:tier: {}
          f:vendor: {}
      f:spec:
        f:backoffLimit: {}
        f:completions: {}
        f:parallelism: {}
        f:template:
          f:metadata:
            f:annotations:
              .: {}
              f:checksum/config: {}
              f:checksum/packs: {}
            f:labels:
              .: {}
              f:app: {}
              f:chart: {}
              f:heritage: {}
              f:release: {}
              f:tier: {}
              f:vendor: {}
            f:name: {}
          f:spec:
            f:containers:
              k:{"name":"st2-register-content"}:
                .: {}
                f:command: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
                f:volumeMounts:
                  .: {}
                  k:{"mountPath":"/etc/st2/st2.docker.conf"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                    f:subPath: {}
                  k:{"mountPath":"/etc/st2/st2.user.conf"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                    f:subPath: {}
                  k:{"mountPath":"/opt/stackstorm/configs-helm/"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                  k:{"mountPath":"/opt/stackstorm/configs/"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                  k:{"mountPath":"/opt/stackstorm/packs"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                  k:{"mountPath":"/opt/stackstorm/virtualenvs"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
            f:dnsPolicy: {}
            f:initContainers:
              .: {}
              k:{"name":"st2-pack-configs-from-helm"}:
                .: {}
                f:command: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
                f:volumeMounts:
                  .: {}
                  k:{"mountPath":"/opt/stackstorm/configs"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                  k:{"mountPath":"/opt/stackstorm/configs-shared"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
              k:{"name":"st2-system-packs"}:
                .: {}
                f:command: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
                f:volumeMounts:
                  .: {}
                  k:{"mountPath":"/opt/stackstorm/packs-shared"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
                  k:{"mountPath":"/opt/stackstorm/virtualenvs-shared"}:
                    .: {}
                    f:mountPath: {}
                    f:name: {}
              k:{"name":"wait-for-db"}:
                .: {}
                f:command: {}
                f:image: {}
                f:imagePullPolicy: {}
                f:name: {}
                f:resources: {}
                f:terminationMessagePath: {}
                f:terminationMessagePolicy: {}
            f:restartPolicy: {}
            f:schedulerName: {}
            f:securityContext: {}
            f:terminationGracePeriodSeconds: {}
            f:volumes:
              .: {}
              k:{"name":"st2-config-vol"}:
                .: {}
                f:configMap:
                  .: {}
                  f:defaultMode: {}
                  f:name: {}
                f:name: {}
              k:{"name":"st2-pack-configs-from-helm-vol"}:
                .: {}
                f:configMap:
                  .: {}
                  f:defaultMode: {}
                  f:name: {}
                f:name: {}
              k:{"name":"st2-pack-configs-vol"}:
                .: {}
                f:name: {}
                f:nfs:
                  .: {}
                  f:path: {}
                  f:server: {}
              k:{"name":"st2-packs-vol"}:
                .: {}
                f:name: {}
                f:nfs:
                  .: {}
                  f:path: {}
                  f:server: {}
              k:{"name":"st2-virtualenvs-vol"}:
                .: {}
                f:name: {}
                f:nfs:
                  .: {}
                  f:path: {}
                  f:server: {}
    manager: Go-http-client
    operation: Update
    time: "2021-09-03T12:10:00Z"
  - apiVersion: batch/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          .: {}
          k:{"type":"Failed"}:
            .: {}
            f:lastProbeTime: {}
            f:lastTransitionTime: {}
            f:message: {}
            f:reason: {}
            f:status: {}
            f:type: {}
        f:failed: {}
        f:startTime: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-09-03T12:15:55Z"
  name: stackstorm-job-st2-register-content
  namespace: stackstorm
  resourceVersion: "4746671"
  uid: a23c66f8-0de1-4300-b65f-6ef5434f1e9c
spec:
  backoffLimit: 6
  completions: 1
  parallelism: 1
  selector:
    matchLabels:
      controller-uid: a23c66f8-0de1-4300-b65f-6ef5434f1e9c
  template:
    metadata:
      annotations:
        checksum/config: 8a6b70b380ae893f99f0805552e399be85b45aecca06691e53ac6c3704db2f7f
        checksum/packs: e83e05073687f92640d08a9ba75e7bfcf310af7d86a6e2f5bae3890c0d216ae6
      creationTimestamp: null
      labels:
        app: st2-register-content
        chart: stackstorm-ha-0.60.0
        controller-uid: a23c66f8-0de1-4300-b65f-6ef5434f1e9c
        heritage: Helm
        job-name: stackstorm-job-st2-register-content
        release: stackstorm
        tier: backend
        vendor: stackstorm
      name: job-st2-register-content
    spec:
      containers:
      - command:
        - st2-register-content
        - --config-file=/etc/st2/st2.conf
        - --config-file=/etc/st2/st2.docker.conf
        - --config-file=/etc/st2/st2.user.conf
        - --register-all
        - --register-fail-on-failure
        image: stackstorm/st2actionrunner:3.5dev
        imagePullPolicy: IfNotPresent
        name: st2-register-content
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/st2/st2.docker.conf
          name: st2-config-vol
          subPath: st2.docker.conf
        - mountPath: /etc/st2/st2.user.conf
          name: st2-config-vol
          subPath: st2.user.conf
        - mountPath: /opt/stackstorm/packs
          name: st2-packs-vol
        - mountPath: /opt/stackstorm/virtualenvs
          name: st2-virtualenvs-vol
        - mountPath: /opt/stackstorm/configs/
          name: st2-pack-configs-vol
        - mountPath: /opt/stackstorm/configs-helm/
          name: st2-pack-configs-from-helm-vol
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sh
        - -c
        - |
          until nc -z -w 2 stackstorm-mongodb-headless 27017 && echo mongodb ok;
            do
              echo 'Waiting for MongoDB Connection...'
              sleep 2;
          done
        image: busybox:1.28
        imagePullPolicy: IfNotPresent
        name: wait-for-db
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      - command:
        - sh
        - -ec
        - |
          /bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
          /bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared
        image: stackstorm/st2actionrunner:3.5dev
        imagePullPolicy: IfNotPresent
        name: st2-system-packs
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/stackstorm/packs-shared
          name: st2-packs-vol
        - mountPath: /opt/stackstorm/virtualenvs-shared
          name: st2-virtualenvs-vol
      - command:
        - sh
        - -ec
        - |
          /bin/cp -aR /opt/stackstorm/configs/. /opt/stackstorm/configs-shared
        image: stackstorm/st2actionrunner:3.5dev
        imagePullPolicy: IfNotPresent
        name: st2-pack-configs-from-helm
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /opt/stackstorm/configs-shared
          name: st2-pack-configs-vol
        - mountPath: /opt/stackstorm/configs
          name: st2-pack-configs-from-helm-vol
      restartPolicy: OnFailure
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          name: stackstorm-st2-config
        name: st2-config-vol
      - name: st2-packs-vol
        nfs:
          path: /nfs/stackstorm/packs
          server: bcnfs03.xx.xx.com.ar
      - name: st2-virtualenvs-vol
        nfs:
          path: /nfs/stackstorm/virtualenv
          server: bcnfs03.xx.xx.com.ar
      - name: st2-pack-configs-vol
        nfs:
          path: /nfs/stackstorm/config
          server: bcnfs03.xxx.xxx.com.ar
      - configMap:
          defaultMode: 420
          name: stackstorm-st2-pack-configs
        name: st2-pack-configs-from-helm-vol
status:
  conditions:
  - lastProbeTime: "2021-09-03T12:15:55Z"
    lastTransitionTime: "2021-09-03T12:15:55Z"
    message: Job has reached the specified backoff limit
    reason: BackoffLimitExceeded
    status: "True"
    type: Failed
  failed: 1
  startTime: "2021-09-03T12:10:00Z"

File in nfs

[root@BCNFS03 stackstorm]# find .
.
./config
./packs
./packs/core
./packs/core/CHANGES.md
./packs/core/README.md
./packs/core/icon.png
./packs/core/pack.yaml
./packs/core/requirements-tests.txt
./packs/core/actions
./packs/core/actions/README.md
./packs/core/actions/announcement.yaml
./packs/core/actions/ask.yaml
./packs/core/actions/echo.yaml
./packs/core/actions/error.yaml
./packs/core/actions/generate_uuid.py
./packs/core/actions/http.yaml
./packs/core/actions/inject_trigger.py
./packs/core/actions/inject_trigger.yaml
./packs/core/actions/local.yaml
./packs/core/actions/local_sudo.yaml
./packs/core/actions/noop.yaml
./packs/core/actions/pause.py
./packs/core/actions/pause.yaml
./packs/core/actions/remote.yaml
./packs/core/actions/remote_sudo.yaml
./packs/core/actions/send_mail
./packs/core/actions/send_mail/send_mail
./packs/core/actions/sendmail.yaml
./packs/core/actions/uuid.yaml
./packs/core/actions/winrm_cmd.yaml
./packs/core/actions/winrm_ps_cmd.yaml
./packs/core/sensors
./packs/core/sensors/README.md
./packs/core/tests
./packs/core/tests/test_action_inject_trigger.py
./packs/core/tests/test_action_sendmail.py
./packs/core/tests/test_action_uuid.py
./packs/default
./packs/default/icon.png
./packs/default/pack.yaml
./packs/default/actions
./packs/default/actions/README.md
./packs/default/actions/workflows
./packs/default/actions/workflows/README.md
./packs/default/rules
./packs/default/rules/README.md
./packs/default/sensors
./packs/default/sensors/README.md
./packs/linux
./packs/linux/README.md
./packs/linux/icon.png
./packs/linux/pack.yaml
./packs/linux/requirements.txt
./packs/linux/actions
./packs/linux/actions/check_loadavg.yaml
./packs/linux/actions/check_processes.yaml
./packs/linux/actions/cp.yaml
./packs/linux/actions/diag_loadavg.yaml
./packs/linux/actions/dig.py
./packs/linux/actions/dig.yaml
./packs/linux/actions/file_touch.yaml
./packs/linux/actions/lsof.yaml
./packs/linux/actions/lsof_pids.yaml
./packs/linux/actions/mv.yaml
./packs/linux/actions/netstat.yaml
./packs/linux/actions/netstat_grep.yaml
./packs/linux/actions/pkill.yaml
./packs/linux/actions/rm.yaml
./packs/linux/actions/rsync.yaml
./packs/linux/actions/scp.yaml
./packs/linux/actions/service.py
./packs/linux/actions/service.yaml
./packs/linux/actions/traceroute.sh
./packs/linux/actions/traceroute.yaml
./packs/linux/actions/vmstat.yaml
./packs/linux/actions/wait_for_ssh.py
./packs/linux/actions/wait_for_ssh.yaml
./packs/linux/actions/checks
./packs/linux/actions/checks/check_loadavg.py
./packs/linux/actions/checks/check_processes.py
./packs/linux/actions/lib
./packs/linux/actions/lib/__init__.py
./packs/linux/actions/workflows
./packs/linux/actions/workflows/diag_loadavg.yaml
./packs/linux/sensors
./packs/linux/sensors/README.md
./packs/linux/sensors/file_watch_sensor.py
./packs/linux/sensors/file_watch_sensor.yaml
./packs/linux/tests
./packs/linux/tests/test_action_dig.py
./packs/packs
./packs/packs/config.schema.yaml
./packs/packs/icon.png
./packs/packs/pack.yaml
./packs/packs/actions
./packs/packs/actions/delete.yaml
./packs/packs/actions/download.yaml
./packs/packs/actions/get.yaml
./packs/packs/actions/get_config.py
./packs/packs/actions/get_config.yaml
./packs/packs/actions/get_pack_dependencies.yaml
./packs/packs/actions/get_pack_warnings.yaml
./packs/packs/actions/install.meta.yaml
./packs/packs/actions/load.yaml
./packs/packs/actions/restart_component.yaml
./packs/packs/actions/search.yaml
./packs/packs/actions/setup_virtualenv.yaml
./packs/packs/actions/show.yaml
./packs/packs/actions/uninstall.meta.yaml
./packs/packs/actions/unload.yaml
./packs/packs/actions/update_virtualenv.yaml
./packs/packs/actions/virtualenv_prerun.yaml
./packs/packs/actions/pack_mgmt
./packs/packs/actions/pack_mgmt/__init__.py
./packs/packs/actions/pack_mgmt/delete.py
./packs/packs/actions/pack_mgmt/download.py
./packs/packs/actions/pack_mgmt/get_installed.py
./packs/packs/actions/pack_mgmt/get_pack_dependencies.py
./packs/packs/actions/pack_mgmt/get_pack_warnings.py
./packs/packs/actions/pack_mgmt/register.py
./packs/packs/actions/pack_mgmt/search.py
./packs/packs/actions/pack_mgmt/setup_virtualenv.py
./packs/packs/actions/pack_mgmt/show_remote.py
./packs/packs/actions/pack_mgmt/unload.py
./packs/packs/actions/pack_mgmt/virtualenv_setup_prerun.py
./packs/packs/actions/workflows
./packs/packs/actions/workflows/install.yaml
./packs/packs/actions/workflows/uninstall.yaml
./packs/packs/aliases
./packs/packs/aliases/pack_get.yaml
./packs/packs/aliases/pack_install.yaml
./packs/packs/aliases/pack_search.yaml
./packs/packs/aliases/pack_show.yaml
./packs/packs/tests
./packs/packs/tests/test_action_aliases.py
./packs/packs/tests/test_action_download.py
./packs/packs/tests/test_action_unload.py
./packs/packs/tests/test_get_pack_dependencies.py
./packs/packs/tests/test_get_pack_warnings.py
./packs/packs/tests/test_virtualenv_setup_prerun.py
./packs/packs/tests/fixtures
./packs/packs/tests/fixtures/stackstorm-test
./packs/packs/tests/fixtures/stackstorm-test/pack.yaml
./packs/packs/tests/fixtures/stackstorm-test2
./packs/packs/tests/fixtures/stackstorm-test2/pack.yaml
./packs/packs/tests/fixtures/stackstorm-test3
./packs/packs/tests/fixtures/stackstorm-test3/pack.yaml
./packs/packs/tests/fixtures/stackstorm-test4
./packs/packs/tests/fixtures/stackstorm-test4/pack.yaml
./packs/chatops
./packs/chatops/CHANGES.md
./packs/chatops/README.md
./packs/chatops/icon.png
./packs/chatops/pack.yaml
./packs/chatops/actions
./packs/chatops/actions/format_execution_result.py
./packs/chatops/actions/format_execution_result.yaml
./packs/chatops/actions/match.py
./packs/chatops/actions/match.yaml
./packs/chatops/actions/match_and_execute.py
./packs/chatops/actions/match_and_execute.yaml
./packs/chatops/actions/post_message.yaml
./packs/chatops/actions/post_result.yaml
./packs/chatops/actions/run.yaml
./packs/chatops/actions/templates
./packs/chatops/actions/templates/default.j2
./packs/chatops/actions/workflows
./packs/chatops/actions/workflows/post_result.yaml
./packs/chatops/actions/workflows/run.yaml
./packs/chatops/rules
./packs/chatops/rules/notify_errbot.yaml
./packs/chatops/rules/notify_hubot.yaml
./packs/chatops/tests
./packs/chatops/tests/fixtures
./packs/chatops/tests/fixtures/http_execution.json
./packs/chatops/tests/fixtures/local_cmd_execution.json
./packs/chatops/tests/fixtures/python_action_execution.json
./packs/chatops/tests/fixtures/remote_cmd_execution.json
./packs/chatops/tests/__init__.py
./packs/chatops/tests/test_format_result.py
./virtualenv
elsopapa commented 3 years ago 
pi", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: #011stream:stderr#011docker:{"container_id"=>"56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd"}#011kubernetes:{"container_name"=>"st2-system-packs", "namespace_name"=>"stackstorm", "pod_name"=>"stackstorm-job-st2-register-content-zwzxn", "pod_id"=>"086da343-f935-4086-94ea-ed61bc2d1150", "host"=>"bckbs01", "labels"=>{"controller-uid"=>"296a2039-b9e2-4016-8e5e-9be00e904d96", "job-name"=>"stackstorm-job-st2-register-content"}, "master_url"=>"https://10.43.0.1:443/api", "namespace_id"=>"31a6fdb9-5060-4b3d-adc6-41bfd834a664"}#011tag:cluster.var.log.containers.stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log#011log_type:k8s_normal_container
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: log:2021-09-03 13:24:30 +0000 [info]: #0 following tail of /var/log/containers/stackstorm-job-st2-register-content-zwzxn_stackstorm_st2-system-packs-56dbdb8a59554206f1531dc17ea1410f7677379ef3b459a09ae8073c233f67fd.log
Sep  3 13:24:57 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:24:34.789381    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:24:58 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:24:48.789527    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:26:02 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:24:59.789383    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:26:02 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:25:11.789311    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:26:03 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:25:25.790264    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:26:03 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:25:37.789456    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:26:03 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:25:49.790479    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
Sep  3 13:26:03 rancher-logging-fluentd-linux-vtqrp cluster: log:E0903 13:26:00.789366    1788 pod_workers.go:191] Error syncing pod 086da343-f935-4086-94ea-ed61bc2d1150 ("stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 2m40s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-zwzxn_stackstorm(086da343-f935-4086-94ea-ed61bc2d1150)"
elsopapa commented 3 years ago 
I0903 14:01:59.621102  455086 request.go:1123] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"stackstorm-job-st2-register-content-r676s","generateName":"stackstorm-job-st2-register-content-","namespace":"st2v4","uid":"63a2a444-9354-4cfb-82ba-ba657c15f339","resourceVersion":"4804607","creationTimestamp":"2021-09-03T16:55:07Z","labels":{"app":"st2-register-content","chart":"stackstorm-ha-0.60.0","controller-uid":"2e17fb12-1aff-49a4-8390-b8ac7067df0e","heritage":"Helm","job-name":"stackstorm-job-st2-register-content","release":"stackstorm","tier":"backend","vendor":"stackstorm"},"annotations":{"checksum/config":"8a6b70b380ae893f99f0805552e399be85b45aecca06691e53ac6c3704db2f7f","checksum/packs":"e83e05073687f92640d08a9ba75e7bfcf310af7d86a6e2f5bae3890c0d216ae6","cni.projectcalico.org/podIP":"10.42.0.217/32","cni.projectcalico.org/podIPs":"10.42.0.217/32"},"ownerReferences":[{"apiVersion":"batch/v1","kind":"Job","name":"stackstorm-job-st2-register-content","uid":"2e17fb12-1aff-49a4-8390-b8ac7067df0e","controller":true,"blockOwnerDeletion":true}],"managedFi [truncated 10913 chars]
I0903 14:01:59.626915  455086 round_trippers.go:432] GET https://bckbs01.bue299.comafi.com.ar/k8s/clusters/c-tght4/api/v1/namespaces/st2v4/pods/stackstorm-job-st2-register-content-r676s/log?container=st2-register-content
I0903 14:01:59.626935  455086 round_trippers.go:438] Request Headers:
I0903 14:01:59.626941  455086 round_trippers.go:442]     Accept: application/json, */*
I0903 14:01:59.626946  455086 round_trippers.go:442]     User-Agent: kubectl/v1.21.4 (linux/amd64) kubernetes/3cce4a8
I0903 14:01:59.626953  455086 round_trippers.go:442]     Authorization: Bearer <masked>
I0903 14:01:59.694086  455086 round_trippers.go:457] Response Status: 400 Bad Request in 67 milliseconds
I0903 14:01:59.694105  455086 round_trippers.go:460] Response Headers:
I0903 14:01:59.694112  455086 round_trippers.go:463]     Audit-Id: 02952eff-37d2-4ff4-aac8-767a8d02f721
I0903 14:01:59.694117  455086 round_trippers.go:463]     Cache-Control: no-cache, private
I0903 14:01:59.694121  455086 round_trippers.go:463]     Content-Length: 239
I0903 14:01:59.694126  455086 round_trippers.go:463]     Content-Type: application/json
I0903 14:01:59.694130  455086 round_trippers.go:463]     Date: Fri, 03 Sep 2021 17:01:59 GMT
I0903 14:01:59.694134  455086 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0903 14:01:59.694182  455086 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"st2-register-content\" in pod \"stackstorm-job-st2-register-content-r676s\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}
I0903 14:01:59.694581  455086 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "container \"st2-register-content\" in pod \"stackstorm-job-st2-register-content-r676s\" is waiting to start: PodInitializing",
  "reason": "BadRequest",
  "code": 400
}]
F0903 14:01:59.694615  455086 helpers.go:115] Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-r676s" is waiting to start: PodInitializing
goroutine 1 [running]:
cognifloyd commented 3 years ago

I'm happy to look into this, but I have some work projects I need to deal with right now. I'll check this issue out this weekend.

elsopapa commented 3 years ago

Ir only happens when i use shared Volumes, pv or nfs.

cognifloyd commented 3 years ago

Would you post your values for st2.packs.images and st2.packs.volumes?

On Sat, Sep 4, 2021, 06:34 elsopapa @.***> wrote:

Ir only happens when i use shared Volumes, pv or nfs.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/StackStorm/stackstorm-ha/issues/238#issuecomment-912958826, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL4QPRC3X7Y42K4QHF5ZITUAH74TANCNFSM5DLVHRKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

elsopapa commented 3 years ago

Default image from the repo (now im triyng with pv and got the same error)

# Default values for StackStorm HA cluster
# This is a YAML-formatted file.

##
## Docker image settings, applied to all StackStorm pods
##
image:
  # Image pull policy
  pullPolicy: IfNotPresent
  # st2 image repository. Set this to override the default ("stackstorm")
  # docker image repository ("docker.stackstorm.com"). Applies to all st2 containers except
  # st2chatops and st2packs (which have their own override). This also does not impact
  # dependencies such as mongo or redis, which have their own helm chart settings.
  repository: ""
  # st2 image tag - defaults to AppVersion.
  # Note that Helm templating is supported in this block!
  tag: "{{ .Chart.AppVersion }}"
  # Image pull secret.
  # May be required for public docker hub due to rate limiting or any private repository.
  # See: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  #pullSecret: "your-pull-secret"

##
## Service Account
##
serviceAccount:
  # Whether the Chart should create the service account or not
  create: true
  # Used to define service account annotations
  serviceAccountAnnotations: {}
  # Used to override service account name
  serviceAccountName:
  # Fallback image pull secret.
  # If a pod does not have pull secrets, k8s will use the service account's pull secrets.
  # See: https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/#serviceaccount-admission-controller
  #pullSecret: "your-pull-secret"

##
## StackStorm shared variables
##
st2:
  # Username, used to login to StackStorm system
  username: st2admin
  # Password, used to login to StackStorm system
  # If set, st2.password always overrides any existing password.
  # If not set, the password is auto-generated on install and preserved across upgrades.
  password: xxxxxxx
  # ST2 crypto key for the  K/V datastore.
  # See https://docs.stackstorm.com/datastore.html#securing-secrets-admin-only for more info.
  # Warning! Replace with your own generated key!
  #datastore_crypto_key: {"hmacKey": {"hmacKeyString": "", "size": 256}, "size": 256, "aesKeyString": "", "mode": "CBC"}
  # SSH private key for the 'stanley' system user ('system_user.ssh_key_file' in st2.conf)
  # If set, st2.ssh_key always overrides any existing ssh_key.
  # If not set, the ssh_key is auto-generated on install and preserved across upgrades.
  #ssh_key: |-
  #  -----BEGIN RSA PRIVATE KEY-----
  #  ...
  #  -----END RSA PRIVATE KEY-----

  # Custom StackStorm config (st2.user.conf) which will apply settings on top of default st2.conf
  config: |
    [api]
    allow_origin = '*'

  # This mirrors the [system_user] section of st2.conf, but makes the values available for helm templating.
  # If you change the user, you must provide a customized st2actionrunner image that includes your user.
  system_user:
    user: stanley
    # templating is allowed for this key
    ssh_key_file: "/home/{{ .Values.st2.system_user.user }}/.ssh/stanley_rsa"

  # Custom pack configs and image settings.
  #
  # By default, system packs are available. By default, however, `st2 pack install` cannot be run in the k8s cluster,
  # so you will need to bake additional packs into an 'st2packs' image. Please see github.com/stackstorm/stackstorm-ha/README.md
  # for details on how to build this image.
  # To change this default, and use persistent/shared/writable storage that is available in your cluster, you need to
  # enable st2.packs.volumes below, adding volume definitions customized for use your cluster's storage provider.
  packs:
    # Custom StackStorm pack configs. Each record creates a file in '/opt/stackstorm/configs/'
    # https://docs.stackstorm.com/reference/pack_configs.html#configuration-file
    # NOTE: This takes precedence over the contents of st2.packs.volumes.configs (if defined) on helm upgrade.
    configs:
      core.yaml: |
        ---
        # example core pack config yaml

    # Custom packs images settings.
    #
    # For each given st2packs container you can define repository, name, tag and pullPolicy for this image below.
    # Multiple pack images can help when dealing with frequent updates by only rebuilding smaller images for desired packs
    # E.g. having all desired StackStorm-Exchange packs in one image and several custom packs in additional images
    images: []
      #- repository: index.docker.io/stackstorm
      #  name: st2packs
      #  tag: example
      #  pullPolicy: IfNotPresent
      # Optional name of the imagePullSecret if your custom packs image is hosted by a private Docker registry
      #  pullSecret: st2packs-auth

    # Custom packs volumes definitions.
    #
    # Use this to have StackStorm use persistent/shared/writable storage configured previously in your cluster.
    # The choice of storage solution is cluster-dependent (it changes besed on where the cluster is hosted
    # and which storage solutions are available in your cluster).
    #
    # To use this, set enabled to true, and add cluster-specific volume definitions for at least packs and virtualenvs below.
    # Please consult the documentation for your cluster's storage solution.
    # Some generic examples are listed under st2.packs.volumes.packs below.
    volumes:
      enabled: true
      packs: 
        # mounted to /opt/stackstorm/packs
        # packs volume definition is required if st2.packs.volumes is enabled

        # example using persistentVolumeClaim:
        persistentVolumeClaim:
         claimName: pvc-st2-packs

        # nfs:
        #  server: "bcnfs03.bue299.comafi.com.ar"
        #  path: /nfs/stackstorm/packs

        # example using a flexVolume + rook-ceph
        #flexVolume:
        #  driver: ceph.rook.io/rook
        #  fsType: ceph
        #  options:
        #    fsName: fs1
        #    clusterNamespace: rook-ceph
        #    path: /st2/packs

      virtualenvs: 
        # mounted to /opt/stackstorm/virtualenvs
        # virtualenvs volume definition is required if st2.packs.volumes is enabled
        # see the examples under st2.packs.volumes.packs
         persistentVolumeClaim:
          claimName: pvc-st2-virtualenvs

      configs: 
        # mounted to /opt/stackstorm/configs
        # configs volume definition is optional, but only used if st2.packs.volumes is enabled
        # Anything in `st2.packs.configs` will be added to this volume automatically on helm install/upgrade.
        # see the examples under st2.packs.volumes.packs
        persistentVolumeClaim:
         claimName: pvc-st2-configs

    # https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
    # It is possible to run st2sensorcontainer in HA mode by running one process on each compute instance.
    # Each sensor node needs to be provided with proper partition information to share work with other sensor
    # nodes so that the same sensor does not run on different nodes.
    # Defaults come from st2sensorcontainer (see below), with per-sensor overrides defined here.
    sensors:
      # Specify default container that executes all sensors.
      # To partition sensors with one sensor per node, override st2.packs.sensors.
      # NOTE: Do not modify this file.
      - name:
        livenessProbe: {}
        readinessProbe: {}
        annotations: {}
        resources:
          requests:
            memory: "100Mi"
            cpu: "50m"
        # Override default image settings (for now, only tag can be overridden)
        image: {}
          ## Note that Helm templating is supported in this block!
          #tag: "{{ .Values.image.tag }}"
        # Additional advanced settings to control pod/deployment placement
        affinity: {}
        nodeSelector: {}
        tolerations: []
        serviceAccount:
          attach: false
        # note: postStartScript is not valid here. Use st2sensorcontainer.postStartScript instead.
  # Import data into StackStorm's Key/Value datastore (https://docs.stackstorm.com/datastore.html)
  keyvalue:
    #- name: st2_version
    #  scope: st2kv.system
    #  secret: false
    #  encrypted: false
    #  value: "2.9"
    #- name: release_name
    #  scope: st2kv.system
    #  secret: false
    #  encrypted: false
    #  # Note that Helm templating is supported in this block!
    #  value: "{{ .Release.Name }}"
  # Import a list of ST2 API Keys (https://docs.stackstorm.com/authentication.html#api-key-migration)
  apikeys:
    #- created_at: '2018-12-15T00:21:48.507388Z'
    #  enabled: true
    #  id: 5c14491c6cb8de1a9207e3a2
    #  key_hash: 56928c2d9637ce44338e9564d4b939df8b258410db23b5a80f8ad69d58e648b574f35f9293c3a76bde263738be9aa8379a81553cd55513ad672540b7b0ec0cac
    #  metadata: {"comment": "Example unsecure ST2 API key from K8s HA Helm values.yaml"}
    #  uid: api_key:56928c2d9637ce44338e9564d4b939df8b258410db23b5a80f8ad69d58e648b574f35f9293c3a76bde263738be9aa8379a81553cd55513ad672540b7b0ec0cac
    #  user: st2admin
  # StackStorm Role Based Access Control settings (https://docs.stackstorm.com/rbac.html)
  rbac:
    enabled: false
    # Custom StackStorm RBAC roles, shipped in '/opt/stackstorm/rbac/roles/'
    # See https://docs.stackstorm.com/rbac.html#defining-roles-and-permission-grants
    roles:
      sample.yaml: |
        # sample RBAC role file, see https://docs.stackstorm.com/rbac.html#defining-roles-and-permission-grants
        ---
        name: "sample"
        description: "Example Role which contains no permission grants and serves for demonstration purposes"

    # Custom StackStorm RBAC role assignments, shipped in '/opt/stackstorm/rbac/assignments/'
    # See: https://docs.stackstorm.com/rbac.html#defining-user-role-assignments
    assignments:
      st2admin.yaml: |
        ---
        username: st2admin
        roles:
          - system_admin
      stanley.yaml: |
        ---
        username: stanley
        roles:
          - admin

    # StackStorm RBAC LDAP groups-to-roles mapping rules, shipped in '/opt/stackstorm/rbac/mappings/'
    # See RBAC Roles Based on LDAP Groups: https://docs.stackstorm.com/rbac.html#automatically-granting-roles-based-on-ldap-group-membership
    mappings:
      #stormers.yaml: |
      #  ---
      #  group: "CN=stormers,OU=groups,DC=stackstorm,DC=net"
      #  description: "Automatically grant admin role to all stormers group members."
      #  roles:
      #    - "admin"

##
## StackStorm HA Ingress
##
ingress:
  # As recommended, ingress is disabled by default.
  enabled: false
  # Annotations are used to configure the ingress controller
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  # Map hosts to paths
  hosts: []
  # - host: hostname.domain.tld
  #   # Map paths to services
  #   paths:
  #     - path: /
  #       serviceName: service
  #       servicePort: port
  # Secure the Ingress by specifying a secret that contains a TLS private key and certificate
  tls: []
  # - secretName: chart-example-tls
  #   hosts:
  #     - chart-example.test

##
## NOTE: there used to be a secrets.st2 section here. These values have moved into st2.* above. Please update your values.
##

##
## StackStorm HA Cluster pod settings for each individual service/component.
##
# Many st2web instances, placed behind a load balancer that serve web app and proxify requests to st2auth, st2api, st2stream.
st2web:
  # Minimum 2 replicas are recommended to run st2web in HA mode
  replicas: 2
  # Tested resource consumption based on multiple requests to st2web within nginx
  # Please adjust based on your conscious choice
  resources:
    requests:
      memory: "25Mi"
      cpu: "50m"
    limits:
      memory: "100Mi"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # TODO: Add Ingress setting as a way to expose service to public (#6).
  # ingress:
  service:
    # type can be one of "ClusterIP", "NodePort", "LoadBalancer" or "ExternalName"
    type: "NodePort"
    # The hostname associated with st2web service (externalName, added to external DNS, etc.)
    hostname: ""
    # For more information regarding annotations, see
    # https://kubernetes.io/docs/concepts/services-networking/service/#ssl-support-on-aws
    annotations: {}
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # User-defined st2web config with custom settings to replace default config.js
  # See https://github.com/StackStorm/st2web#connecting-to-st2-server for more info
  # config: |
  #  // see https://github.com/StackStorm/st2web/blob/master/config.js
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2auth
# Multiple st2auth processes can be behind a load balancer in an active-active configuration.
st2auth:
  replicas: 2
  resources:
    requests:
      memory: "85Mi"
      cpu: "50m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2api
# Multiple st2api process can be behind a load balancer in an active-active configuration.
st2api:
  replicas: 2
  resources:
    requests:
      memory: "150Mi"
      cpu: "25m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2stream
# Multiple st2stream process can be behind a load balancer in an active-active configuration.
st2stream:
  replicas: 2
  resources:
    requests:
      memory: "100Mi"
      cpu: "50m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2rulesengine
# Multiple st2rulesengine processes can run in active-active with only connections to MongoDB and RabbitMQ. All these will share the TriggerInstance load and naturally pick up more work if one or more of the processes becomes unavailable.
st2rulesengine:
  replicas: 2
  resources:
    requests:
      memory: "75Mi"
      cpu: "25m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2timersengine
# Only single replica is created via K8s Deployment as timersengine can't work in active-active mode at the moment and it relies on K8s failover/reschedule capabilities to address cases of process failure.
st2timersengine:
  resources:
    requests:
      memory: "75Mi"
      cpu: "10m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2workflowengine
# Multiple st2workflowengine processes can run in active-active mode and will share the load and pick up more work if one or more of the processes become available.
st2workflowengine:
  replicas: 2
  resources:
    requests:
      memory: "200Mi"
      cpu: "100m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2scheduler
# TODO: Description TBD
st2scheduler:
  replicas: 2
  resources:
    requests:
      memory: "75Mi"
      cpu: "50m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2notifier
# st2notifier runs in active-active mode and requires for that coordination backend like Redis or Zookeeper
st2notifier:
  replicas: 2
  resources:
    requests:
      memory: "75Mi"
      cpu: "50m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""
# https://docs.stackstorm.com/reference/ha.html#st2actionrunner
# Multiple st2actionrunner processes can run in active-active with only connections to MongoDB and RabbitMQ. Work gets naturally
# distributed across runners via RabbitMQ. Adding more st2actionrunner processes increases the ability of StackStorm to execute actions.
st2actionrunner:
  replicas: 5
  resources:
    requests:
      memory: "200Mi"
      cpu: "75m"
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  annotations: {}
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  # Allow the injection of hostAliases (https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/#adding-additional-entries-with-hostaliases)
  # records in the st2actionrunner containers to handle edge case in DNS accessibility/topology
  hostAliases: []
  # - hostnames:
  #     - foo
  #   ip: 1.1.1.1
  #  - hostnames:
  #      - bar
  #   ip: 8.8.8.8
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""

# https://docs.stackstorm.com/reference/ha.html#st2sensorcontainer
# Please see st2.packs.sensors for each sensor instance's config.
# This contains default settings for all sensor pods.
st2sensorcontainer:
  resources:
    requests:
      memory: "100Mi"
      cpu: "50m"
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  livenessProbe: {}
  readinessProbe: {}
  annotations: {}
  # Additional advanced settings to control pod/deployment placement
  affinity: {}
  nodeSelector: {}
  tolerations: []
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""

# The st2client deployment/pod simplifies ad-hoc administration.
# st2client is a special purpose actionrunner pod, but you can customize it separately
st2client:
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""

# https://docs.stackstorm.com/reference/ha.html#st2garbagecollector
# Optional service that cleans up old executions and other operations data based on setup configurations.
# By default this process does nothing and needs to be setup in st2.conf to perform any work.
st2garbagecollector:
  # Having 1 st2garbagecollector unique replica is enough for periodic task like st2 history garbage collection
  replicas: 1
  resources:
    requests:
      memory: "80Mi"
      cpu: "10m"
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""

##
## StackStorm ChatOps (https://docs.stackstorm.com/chatops/index.html)
## As hubot can't be HA scaled properly, we deploy only single replica of st2chatops
##
st2chatops:
  # Enable st2chatops (default: false)
  enabled: false
  # Custom hubot adapter ENV variables to pass through which will override st2chatops.env defaults.
  # See https://github.com/StackStorm/st2chatops/blob/master/st2chatops.env
  # for the full list of supported adapters and example ENV variables.
  env:
    HUBOT_ADAPTER: slack
    HUBOT_SLACK_TOKEN: xoxb-CHANGE-ME-PLEASE
    # If ST2_API_KEY is defined, then ST2_AUTH_USERNAME/PASSWORD will not be exposed to st2chatops.
    # Please make sure that the key in ST2_API_KEY gets added to st2.apikeys above so that chatops can use this key.
    #ST2_API_KEY: 12345
  # Set custom generated st2chatops Docker image source
  # Otherwise default https://hub.docker.com/r/stackstorm/st2chatops is used
  image: {}
    #repository: stackstorm
    #name: st2chatops
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
    #pullPolicy: IfNotPresent
  # Tested requested resource consumption for st2chatops & hubot in normal mode
  # Please adjust based on your conscious choice
  resources:
    requests:
      memory: "50Mi"
      cpu: "5m"
  annotations: {}
  # Advanced use-cases only. If defined, this hubot-scripts volume gets mounted to: /opt/stackstorm/chatops/scripts
  # This volume (using any k8s storage solution, including configmap) allows for hubot customization.
  # Most installations should not use this.
  # For details on writing .js or .coffeescript hubot extensions, see: https://hubot.github.com/docs/scripting/
  hubotScriptsVolume: {}
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}
  serviceAccount:
    attach: false
  # postStartScript is optional. It has the contents of a bash script.
  # k8s will run the script in the st2 container in parallel with the ENTRYPOINT.
  # The pod will not be marked as "running" until this script completes successfully.
  # see: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
  postStartScript: ""

##
## Various batch jobs (apply-rbac-definitions, apikey-load, key-load, register-content)
##
jobs:
  annotations: {}
  # Override default image settings (for now, only tag can be overridden)
  # The Jobs use the st2actionrunner image
  image: {}
    ## Note that Helm templating is supported in this block!
    #tag: "{{ .Values.image.tag }}"
  # If defined, this preRegisterContentCommand runs in an initContainer on the st2-register-content Job.
  # The initContainer also uses the st2actionrunner image but runs this command instead of its entrypoint.
  preRegisterContentCommand: []
    # For example, to disable aliases in the "packs" system pack before content is registered:
    # - "/bin/bash"
    # - "-c"
    # - |
    #   sed -i -e 's/^\(\s*\)enabled: true/\1enabled: false/' /opt/stackstorm/packs/packs/aliases/*.yaml
    #   for alias in /opt/stackstorm/packs/packs/aliases/*.yaml; do
    #     grep -q 'enabled:' ${alias} || sed -i -e 's/^\(\s*\)name:\(.*\)$/\1name: \2\n\1enabled: false/m' ${alias}
    #   done
  # Additional advanced settings to control pod/deployment placement
  nodeSelector: {}
  tolerations: []
  affinity: {}

##
## MongoDB HA configuration (3rd party chart dependency)
##
## For values.yaml reference:
## https://github.com/bitnami/charts/tree/master/bitnami/mongodb
##
mongodb:
  # Change to `false` to disable in-cluster mongodb deployment.
  # Specify your external [database] connection parameters under st2.config
  enabled: true
  image:
    # StackStorm currently supports maximum MongoDB v4.0
    tag: "4.0"
  # MongoDB architecture. Allowed values: standalone or replicaset
  architecture: replicaset
  # Name of the replica set
  # Ignored when mongodb.architecture=standalone
  replicaSetName: rs0
  # Number of MongoDB replicas to deploy.
  # Ignored when mongodb.architecture=standalone
  replicaCount: 3
  auth:
    enabled: true
    # NB! It's highly recommended to change ALL insecure auth defaults!
    #
    # Stackstorm user that services will make connections using
    username: "st2-admin"
    # Stackstorm user that services will make connections using
    password: "XeL5Rxwj7F0Wt43tFZVTN7H8Sg5XDHmK"
    # Root password for the database
    rootPassword: "8fAzdnksdzPFUWm4a68EfY7nMhBPaa"
    # Initial database for stackstorm
    database: "st2"
    # Minimal key length is 6 symbols
    replicaSetKey: "82PItDpqroti5RngOA7UqbHH7c6bFUwy"
    # Whether to enable the arbiter
  arbiter:
    enabled: false
  resources: {}

##
## RabbitMQ configuration (3rd party chart dependency)
##
## For values.yaml reference:
## https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq
##
rabbitmq:
  # Change to `false` to disable in-cluster rabbitmq deployment.
  # Specify your external [messaging] connection parameters under st2.config
  enabled: true
  clustering:
    # On unclean cluster restarts forceBoot is required to cleanup Mnesia tables (see: https://github.com/helm/charts/issues/13485)
    # Use it only if you prefer availability over integrity.
    forceBoot: true
  # Authentication Details
  auth:
    username: admin
    # TODO: Use default random 10 character password, but need to fetch this string for use by downstream services
    password: 9jS+w1u07NbHtZke1m+jW4Cj
    # Up to 255 character string, should be fixed so that re-deploying the chart does not fail (see: https://github.com/helm/charts/issues/12371)
    # NB! It's highly recommended to change the default insecure rabbitmqErlangCookie value!
    erlangCookie: 8MrqQdCQ6AQ8U3MacSubHE5RqkSfvNaRHzvxuFcG
  # RabbitMQ Memory high watermark. See: http://www.rabbitmq.com/memory.html
  # Default values might not be enough for StackStorm deployment to work properly. We recommend to adjust these settings for you needs as well as enable Pod memory limits via "resources".
  #rabbitmqMemoryHighWatermark: 512MB
  #rabbitmqMemoryHighWatermarkType: absolute
  persistence:
    enabled: true
  # Enable Queue Mirroring between nodes
  # See https://www.rabbitmq.com/ha.html
  # This code block is commented out waiting for
  # https://github.com/bitnami/charts/issues/4635
  loadDefinition:
    enabled: true
    existingSecret: "{{ .Release.Name }}-rabbitmq-definitions"
  extraConfiguration: |
    load_definitions = /app/rabbitmq-definitions.json
  # We recommend to set the memory limit for RabbitMQ-HA Pods in production deployments.
  # Make sure to also change the rabbitmqMemoryHighWatermark following the formula:
  # rabbitmqMemoryHighWatermark = 0.4 * resources.limits.memory
  resources: {}
  # number of replicas in the rabbit cluster
  replicaCount: 3
  # As RabbitMQ enabled prometheus operator monitoring by default, disable it for non-prometheus users
  metrics:
    enabled: false
<nabled: true
    # Enable or disable static sentinel IDs for each replicas
    # If disabled each sentinel will generate a random id at startup
    # If enabled, each replicas will have a constant ID on each start-up
    staticID: true
  networkPolicy:
    enabled: false
  usePassword: false
  metrics:
    enabled: false

##
## Settings to be applied to all stackstorm-ha pods
##
# https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-policy
# "ClusterFirst" is the default. Other options: "Default", "ClusterFirstWithHostNet", "None"
#dnsPolicy: "ClusterFirst"
# https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-dns-config
dnsConfig: {}
  # example copied from the docs
  #nameservers:
  #- 1.2.3.4
  #searches:
  #- ns1.svc.cluster-domain.example
  #- my.dns.search.suffix
  #options:
  #- name: ndots
  #  value: "2"
  #- name: edns0

##
## External DNS configuration (3rd party chart dependency)
##
## Creates a deployment of external-dns within the cluster to update DNS with CNAME -> ELB
##
## For values.yaml reference:
## https://github.com/bitnami/charts/tree/master/bitnami/external-dns
##
## TODO: If eq st2web.service.type "LoadBalancer", set enabled to true. Any other cases?
external-dns:
  enabled: false
  provider: aws
  aws:
    zoneType: "public"
  domainFilters: []
cognifloyd commented 3 years ago

I just edited your comments to make them easier to review. When you post a comment with code or logs, please surround the block of text with ``` (3 backtics before and 3 after). Thanks!

Now to see if I can tease something out of these logs...

cognifloyd commented 3 years ago

Are all the other pods running? Eg st2api, st2actionrunner, and st2client?

Please try this:

Are the volumes mounted correctly so that they have read/write access for many pods?

elsopapa commented 3 years ago

The Volumes aré mounted (pv or nfs), i can write AnD read from the pods (ive uploaded the logs ) . All pods aré running ok, (it only fáils when it run the registry job ).

cognifloyd commented 3 years ago

Can you find the logs for the first instance of the failed register job? It tries 6 times. Maybe the first instance will have a more helpful error message

elsopapa commented 3 years ago

the error is cannot initializating 

stackstorm-job-st2-register-content-wfmdx

Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-wfmdx" is waiting to start: PodInitializing

----

I0907 10:10:32.714563   37176 loader.go:372] Config loaded from file:  /home/perrettaa/.kube/config
I0907 10:10:32.717984   37176 round_trippers.go:432] GET https://bckbs01.bue299.comafi.com.ar/k8s/clusters/c-tght4/api/v1/namespaces/st2/pods/stackstorm-job-st2-register-content-wfmdx
I0907 10:10:32.717999   37176 round_trippers.go:438] Request Headers:
I0907 10:10:32.718005   37176 round_trippers.go:442]     Accept: application/json, */*
I0907 10:10:32.718010   37176 round_trippers.go:442]     User-Agent: kubectl/v1.21.4 (linux/amd64) kubernetes/3cce4a8
I0907 10:10:32.718014   37176 round_trippers.go:442]     Authorization: Bearer <masked>
I0907 10:10:32.870064   37176 round_trippers.go:457] Response Status: 200 OK in 152 milliseconds
I0907 10:10:32.870128   37176 round_trippers.go:460] Response Headers:
I0907 10:10:32.870160   37176 round_trippers.go:463]     Audit-Id: 2bb2fd31-3570-41ce-93b6-93ea5982b839
I0907 10:10:32.870186   37176 round_trippers.go:463]     Cache-Control: no-cache, private
I0907 10:10:32.870212   37176 round_trippers.go:463]     Content-Type: application/json
I0907 10:10:32.870236   37176 round_trippers.go:463]     Date: Tue, 07 Sep 2021 13:10:32 GMT
I0907 10:10:32.870257   37176 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0907 10:10:32.870282   37176 round_trippers.go:463]     X-Kubernetes-Pf-Flowschema-Uid: 7a5e4804-2c03-4ad0-aaaa-3d62964c0b58
I0907 10:10:32.870307   37176 round_trippers.go:463]     X-Kubernetes-Pf-Prioritylevel-Uid: a21d7857-d0c7-4d4f-aef6-028ab0b5de91
I0907 10:10:32.890811   37176 request.go:1123] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"stackstorm-job-st2-register-content-wfmdx","generateName":"stackstorm-job-st2-register-content-","namespace":"st2","uid":"b92ca1c7-25de-4706-aa32-a038d001e93f","resourceVersion":"5854267","creationTimestamp":"2021-09-07T13:09:06Z","labels":{"app":"st2-register-content","chart":"stackstorm-ha-0.60.0","controller-uid":"afbcee53-867d-443a-a8d0-880b121ed2aa","heritage":"Helm","job-name":"stackstorm-job-st2-register-content","release":"stackstorm","tier":"backend","vendor":"stackstorm"},"annotations":{"checksum/config":"8a6b70b380ae893f99f0805552e399be85b45aecca06691e53ac6c3704db2f7f","checksum/packs":"e83e05073687f92640d08a9ba75e7bfcf310af7d86a6e2f5bae3890c0d216ae6","cni.projectcalico.org/podIP":"10.42.2.140/32","cni.projectcalico.org/podIPs":"10.42.2.140/32"},"ownerReferences":[{"apiVersion":"batch/v1","kind":"Job","name":"stackstorm-job-st2-register-content","uid":"afbcee53-867d-443a-a8d0-880b121ed2aa","controller":true,"blockOwnerDeletion":true}],"managedFiel [truncated 10646 chars]
I0907 10:10:32.909181   37176 round_trippers.go:432] GET https://bckbs01.bue299.comafi.com.ar/k8s/clusters/c-tght4/api/v1/namespaces/st2/pods/stackstorm-job-st2-register-content-wfmdx/log?container=st2-register-content
I0907 10:10:32.909257   37176 round_trippers.go:438] Request Headers:
I0907 10:10:32.909294   37176 round_trippers.go:442]     Accept: application/json, */*
I0907 10:10:32.909312   37176 round_trippers.go:442]     User-Agent: kubectl/v1.21.4 (linux/amd64) kubernetes/3cce4a8
I0907 10:10:32.909332   37176 round_trippers.go:442]     Authorization: Bearer <masked>
I0907 10:10:32.967657   37176 round_trippers.go:457] Response Status: 400 Bad Request in 58 milliseconds
I0907 10:10:32.967685   37176 round_trippers.go:460] Response Headers:
I0907 10:10:32.967693   37176 round_trippers.go:463]     Audit-Id: f96be680-0cb2-4409-a798-e657b18161b4
I0907 10:10:32.967700   37176 round_trippers.go:463]     Cache-Control: no-cache, private
I0907 10:10:32.967705   37176 round_trippers.go:463]     Content-Length: 239
I0907 10:10:32.967711   37176 round_trippers.go:463]     Content-Type: application/json
I0907 10:10:32.967721   37176 round_trippers.go:463]     Date: Tue, 07 Sep 2021 13:10:32 GMT
I0907 10:10:32.967732   37176 round_trippers.go:463]     X-Content-Type-Options: nosniff
I0907 10:10:32.967809   37176 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"container \"st2-register-content\" in pod \"stackstorm-job-st2-register-content-wfmdx\" is waiting to start: PodInitializing","reason":"BadRequest","code":400}
I0907 10:10:32.968563   37176 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "container \"st2-register-content\" in pod \"stackstorm-job-st2-register-content-wfmdx\" is waiting to start: PodInitializing",
  "reason": "BadRequest",
  "code": 400
}]
F0907 10:10:32.968614   37176 helpers.go:115] Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-wfmdx" is waiting to start: PodInitializing
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.stacks(0xc00000e001, 0xc0005f8120, 0xc7, 0x118)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).output(0x3065420, 0xc000000003, 0x0, 0x0, 0xc0003c00e0, 0x26007a3, 0xa, 0x73, 0x40e300)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).printDepth(0x3065420, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc00070d040, 0x1, 0x1)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:733 +0x16f
k8s.io/kubernetes/vendor/k8s.io/klog/v2.FatalDepth(...)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.fatal(0xc000014280, 0x98, 0x1)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x288
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.checkErr(0x207b300, 0xc00047f2c0, 0x1f07e90)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x935
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/logs.NewCmdLogs.func1(0xc000522b00, 0xc0003747c0, 0x1, 0x4)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/logs/logs.go:158 +0x1a5
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc000522b00, 0xc000374780, 0x4, 0x4, 0xc000522b00, 0xc000374780)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0003dd340, 0xc00003a240, 0xc00003a1e0, 0x6)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x375
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
main.main()
    _output/local/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x21d

goroutine 6 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x3065420)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
created by k8s.io/kubernetes/vendor/k8s.io/klog/v2.init.0
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/v2/klog.go:418 +0xdf

goroutine 35 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x1f07db0, 0x207b600, 0xc0004b4000, 0x1, 0xc000046ba0)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x118
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x1f07db0, 0x12a05f200, 0x0, 0x1, 0xc000046ba0)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0x1f07db0, 0x12a05f200, 0xc000046ba0)
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs.InitLogs
    /build/kubectl/parts/kubectl/go/src/github.com/kubernetes/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/util/logs/logs.go:51 +0x96

goroutine 14 [select]:
net/http.(*persistConn).writeLoop(0xc0003e30e0)
    /snap/go/7954/src/net/http/transport.go:2382 +0xf7
created by net/http.(*Transport).dialConn
    /snap/go/7954/src/net/http/transport.go:1744 +0xc9c

goroutine 13 [IO wait]:
internal/poll.runtime_pollWait(0x7f42cfeb1b30, 0x72, 0xffffffffffffffff)
    /snap/go/7954/src/runtime/netpoll.go:222 +0x55
internal/poll.(*pollDesc).wait(0xc000864898, 0x72, 0x2d00, 0x2d62, 0xffffffffffffffff)
    /snap/go/7954/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
    /snap/go/7954/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000864880, 0xc000224000, 0x2d62, 0x2d62, 0x0, 0x0, 0x0)
    /snap/go/7954/src/internal/poll/fd_unix.go:166 +0x1d5
net.(*netFD).Read(0xc000864880, 0xc000224000, 0x2d62, 0x2d62, 0x2d5d, 0xc000224000, 0x5)
    /snap/go/7954/src/net/fd_posix.go:55 +0x4f
net.(*conn).Read(0xc0005a4010, 0xc000224000, 0x2d62, 0x2d62, 0x0, 0x0, 0x0)
    /snap/go/7954/src/net/net.go:183 +0x91
crypto/tls.(*atLeastReader).Read(0xc00071a4b0, 0xc000224000, 0x2d62, 0x2d62, 0xc0004d69f8, 0xc000680000, 0x0)
    /snap/go/7954/src/crypto/tls/conn.go:776 +0x63
bytes.(*Buffer).ReadFrom(0xc0007d0278, 0x2079f60, 0xc00071a4b0, 0x40b985, 0x1bd0bc0, 0x1d8b680)
    /snap/go/7954/src/bytes/buffer.go:204 +0xbe
crypto/tls.(*Conn).readFromUntil(0xc0007d0000, 0x207cd00, 0xc0005a4010, 0x5, 0xc0005a4010, 0x203000)
    /snap/go/7954/src/crypto/tls/conn.go:798 +0xf3
crypto/tls.(*Conn).readRecordOrCCS(0xc0007d0000, 0x0, 0x0, 0x43ac1c)
    /snap/go/7954/src/crypto/tls/conn.go:605 +0x115
crypto/tls.(*Conn).readRecord(...)
    /snap/go/7954/src/crypto/tls/conn.go:573
crypto/tls.(*Conn).Read(0xc0007d0000, 0xc00069b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /snap/go/7954/src/crypto/tls/conn.go:1276 +0x165
net/http.(*persistConn).Read(0xc0003e30e0, 0xc00069b000, 0x1000, 0x1000, 0xc0001f8060, 0xc0004d6d40, 0x4057b5)
    /snap/go/7954/src/net/http/transport.go:1922 +0x77
bufio.(*Reader).fill(0xc0006c4b40)
    /snap/go/7954/src/bufio/bufio.go:101 +0x108
bufio.(*Reader).Peek(0xc0006c4b40, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)
    /snap/go/7954/src/bufio/bufio.go:139 +0x4f
net/http.(*persistConn).readLoop(0xc0003e30e0)
    /snap/go/7954/src/net/http/transport.go:2083 +0x1a8
created by net/http.(*Transport).dialConn
elsopapa commented 3 years ago

the helm only write the packs on the nfs server , the config and the virtualenv keeps empty

elsopapa commented 3 years ago

the syslog filter by register job 

Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.117916       1 scheduler.go:604] "Successfully bound pod to node" pod="st2/stackstorm-job-st2-register-content-wfmdx" node="bckbs03" evaluatedNodes=3 feasibleNodes=3
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.129291    1755 kubelet.go:1914] SyncLoop (ADD, "api"): "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)"
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.243414    1755 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "st2-packs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-packs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.243529    1755 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "st2-pack-configs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-pack-configs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.243575    1755 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-9ss6r" (UniqueName: "kubernetes.io/secret/b92ca1c7-25de-4706-aa32-a038d001e93f-default-token-9ss6r") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.243605    1755 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "st2-config-vol" (UniqueName: "kubernetes.io/configmap/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-config-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.243627    1755 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "st2-virtualenvs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-virtualenvs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.243644    1755 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "st2-pack-configs-from-helm-vol" (UniqueName: "kubernetes.io/configmap/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-pack-configs-from-helm-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344125    1755 reconciler.go:269] operationExecutor.MountVolume started for volume "st2-packs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-packs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344168    1755 reconciler.go:269] operationExecutor.MountVolume started for volume "st2-pack-configs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-pack-configs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344196    1755 reconciler.go:269] operationExecutor.MountVolume started for volume "default-token-9ss6r" (UniqueName: "kubernetes.io/secret/b92ca1c7-25de-4706-aa32-a038d001e93f-default-token-9ss6r") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344222    1755 reconciler.go:269] operationExecutor.MountVolume started for volume "st2-config-vol" (UniqueName: "kubernetes.io/configmap/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-config-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344542    1755 reconciler.go:269] operationExecutor.MountVolume started for volume "st2-virtualenvs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-virtualenvs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344596    1755 reconciler.go:269] operationExecutor.MountVolume started for volume "st2-pack-configs-from-helm-vol" (UniqueName: "kubernetes.io/configmap/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-pack-configs-from-helm-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.344703    1755 operation_generator.go:672] MountVolume.SetUp succeeded for volume "st2-config-vol" (UniqueName: "kubernetes.io/configmap/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-config-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.345021    1755 operation_generator.go:672] MountVolume.SetUp succeeded for volume "st2-pack-configs-from-helm-vol" (UniqueName: "kubernetes.io/configmap/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-pack-configs-from-helm-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.349419    1755 operation_generator.go:672] MountVolume.SetUp succeeded for volume "default-token-9ss6r" (UniqueName: "kubernetes.io/secret/b92ca1c7-25de-4706-aa32-a038d001e93f-default-token-9ss6r") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.381600    1755 operation_generator.go:672] MountVolume.SetUp succeeded for volume "st2-virtualenvs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-virtualenvs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.382173    1755 operation_generator.go:672] MountVolume.SetUp succeeded for volume "st2-pack-configs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-pack-configs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.397185    1755 operation_generator.go:672] MountVolume.SetUp succeeded for volume "st2-packs-vol" (UniqueName: "kubernetes.io/nfs/b92ca1c7-25de-4706-aa32-a038d001e93f-st2-packs-vol") pod "stackstorm-job-st2-register-content-wfmdx" (UID: "b92ca1c7-25de-4706-aa32-a038d001e93f") 
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.445581    1755 kuberuntime_manager.go:439] No sandbox for pod "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)" can be found. Need to start a new one
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:06.878231    1755 kubelet.go:1921] SyncLoop (UPDATE, "api"): "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)"
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:07.228254    1755 kubelet.go:1952] SyncLoop (PLEG): "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)", event: &pleg.PodLifecycleEvent{ID:"b92ca1c7-25de-4706-aa32-a038d001e93f", Type:"ContainerDied", Data:"2cf30c4fb435eac8a35ec0f2548d22eb5fe16434517ac91dde77f7ff36e9632a"}
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:07.228364    1755 kubelet.go:1952] SyncLoop (PLEG): "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)", event: &pleg.PodLifecycleEvent{ID:"b92ca1c7-25de-4706-aa32-a038d001e93f", Type:"ContainerStarted", Data:"36d6c2039e3b297e7179b236feb7e63c3ec53a8e2634dee8603baf90d369565d"}
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:08.259057    1755 kubelet.go:1952] SyncLoop (PLEG): "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)", event: &pleg.PodLifecycleEvent{ID:"b92ca1c7-25de-4706-aa32-a038d001e93f", Type:"ContainerDied", Data:"1ec60761258c93a67ddf01b94e8d76dbf9632b1f3aabec608085e9f8e768e4ed"}
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:I0907 13:09:09.291023    1755 kubelet.go:1952] SyncLoop (PLEG): "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)", event: &pleg.PodLifecycleEvent{ID:"b92ca1c7-25de-4706-aa32-a038d001e93f", Type:"ContainerDied", Data:"9b1fbbccee176d20435aa3c34c9c13f6638bc5b29ea4be8015cd9cb4e91f3094"}
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:E0907 13:09:09.307698    1755 kuberuntime_container.go:705] failed to remove pod init container "st2-system-packs": rpc error: code = Unknown desc = failed to remove container "1ec60761258c93a67ddf01b94e8d76dbf9632b1f3aabec608085e9f8e768e4ed": Error response from daemon: removal of container 1ec60761258c93a67ddf01b94e8d76dbf9632b1f3aabec608085e9f8e768e4ed is already in progress; Skipping pod "stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)"
Sep  7 13:09:33 rancher-logging-fluentd-linux-8fl2s cluster: log:E0907 13:09:09.308149    1755 pod_workers.go:191] Error syncing pod b92ca1c7-25de-4706-aa32-a038d001e93f ("stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)"), skipping: failed to "StartContainer" for "st2-system-packs" with CrashLoopBackOff: "back-off 10s restarting failed container=st2-system-packs pod=stackstorm-job-st2-register-content-wfmdx_st2(b92ca1c7-25de-4706-aa32-a038d001e93f)"
elsopapa commented 3 years ago

job

Name:           stackstorm-job-st2-register-content
Namespace:      st2
Selector:       controller-uid=afbcee53-867d-443a-a8d0-880b121ed2aa
Labels:         app=st2-register-content
                chart=stackstorm-ha-0.60.0
                heritage=Helm
                release=stackstorm
                tier=backend
                vendor=stackstorm
Annotations:    helm.sh/hook: post-install, post-upgrade, post-rollback
                helm.sh/hook-delete-policy: before-hook-creation
                helm.sh/hook-weight: 7
Parallelism:    1
Completions:    1
Start Time:     Tue, 07 Sep 2021 10:09:06 -0300
Pods Statuses:  0 Running / 0 Succeeded / 1 Failed
Pod Template:
  Labels:       app=st2-register-content
                chart=stackstorm-ha-0.60.0
                controller-uid=afbcee53-867d-443a-a8d0-880b121ed2aa
                heritage=Helm
                job-name=stackstorm-job-st2-register-content
                release=stackstorm
                tier=backend
                vendor=stackstorm
  Annotations:  checksum/config: 8a6b70b380ae893f99f0805552e399be85b45aecca06691e53ac6c3704db2f7f
                checksum/packs: e83e05073687f92640d08a9ba75e7bfcf310af7d86a6e2f5bae3890c0d216ae6
  Init Containers:
   wait-for-db:
    Image:      busybox:1.28
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      until nc -z -w 2 stackstorm-mongodb-headless 27017 && echo mongodb ok;
        do
          echo 'Waiting for MongoDB Connection...'
          sleep 2;
      done

    Environment:  <none>
    Mounts:       <none>
   st2-system-packs:
    Image:      stackstorm/st2actionrunner:3.5dev
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -ec
      /bin/cp -aR /opt/stackstorm/packs/. /opt/stackstorm/packs-shared &&
      /bin/cp -aR /opt/stackstorm/virtualenvs/. /opt/stackstorm/virtualenvs-shared

    Environment:  <none>
    Mounts:
      /opt/stackstorm/packs-shared from st2-packs-vol (rw)
      /opt/stackstorm/virtualenvs-shared from st2-virtualenvs-vol (rw)
   st2-pack-configs-from-helm:
    Image:      stackstorm/st2actionrunner:3.5dev
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -ec
      /bin/cp -aR /opt/stackstorm/configs/. /opt/stackstorm/configs-shared

    Environment:  <none>
    Mounts:
      /opt/stackstorm/configs from st2-pack-configs-from-helm-vol (rw)
      /opt/stackstorm/configs-shared from st2-pack-configs-vol (rw)
  Containers:
   st2-register-content:
    Image:      stackstorm/st2actionrunner:3.5dev
    Port:       <none>
    Host Port:  <none>
    Command:
      st2-register-content
      --config-file=/etc/st2/st2.conf
      --config-file=/etc/st2/st2.docker.conf
      --config-file=/etc/st2/st2.user.conf
      --register-all
      --register-fail-on-failure
    Environment:  <none>
    Mounts:
      /etc/st2/st2.docker.conf from st2-config-vol (rw,path="st2.docker.conf")
      /etc/st2/st2.user.conf from st2-config-vol (rw,path="st2.user.conf")
      /opt/stackstorm/configs-helm/ from st2-pack-configs-from-helm-vol (rw)
      /opt/stackstorm/configs/ from st2-pack-configs-vol (rw)
      /opt/stackstorm/packs from st2-packs-vol (rw)
      /opt/stackstorm/virtualenvs from st2-virtualenvs-vol (rw)
  Volumes:
   st2-config-vol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      stackstorm-st2-config
    Optional:  false
   st2-packs-vol:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    bcnfs03.bue299.comafi.com.ar
    Path:      /nfs/stackstorm/packs
    ReadOnly:  false
   st2-virtualenvs-vol:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    bcnfs03.bue299.comafi.com.ar
    Path:      /nfs/stackstorm/virtualenv
    ReadOnly:  false
   st2-pack-configs-vol:
    Type:      NFS (an NFS mount that lasts the lifetime of a pod)
    Server:    bcnfs03.bue299.comafi.com.ar
    Path:      /nfs/stackstorm/config
    ReadOnly:  false
   st2-pack-configs-from-helm-vol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      stackstorm-st2-pack-configs
    Optional:  false
Events:        <none>
cognifloyd commented 3 years ago

Does this return anything? kubectl logs -l app=st2-register-content --all-containers=true

elsopapa commented 3 years ago

No resources found in st2 namespace.

cognifloyd commented 3 years ago

What about? kubectl logs -p -l app=st2-register-content --all-containers=true

elsopapa commented 3 years ago

same

elsopapa commented 3 years ago

it only fails when i use a shared volumes, the register job works fine on a default install.

cognifloyd commented 3 years ago

Oh. I didn't specify the namespace. If adding -n st2 (or whatever you're namespace is) doesn't bring anything up, then we have to try to capture the logs while the containers start up.

To do that, run kubectl logs -f -n ... -l app=st2-register-content --all-containers=true in a separate window while you run helm install or helm upgrade. Hopefully that will capture the error message.

I think the st2-system-packs InitContainer is where the failure is since the system packs are getting copied, but not the virtualenvs. So we need to see what cp -aR is complaining about to diagnose why it is failing. The kubernetes controller logs won't show us that.

elsopapa commented 3 years ago

the log , 


db ok
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/register.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/search.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/setup_virtualenv.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/show_remote.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/unload.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/virtualenv_setup_prerun.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/register.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/search.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/setup_virtualenv.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/show_remote.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/unload.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/virtualenv_setup_prerun.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/register.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/search.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/setup_virtualenv.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/show_remote.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/unload.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt/virtualenv_setup_prerun.py': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions/pack_mgmt': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/actions': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
mongodb ok
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-register-content" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test2': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test3': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4/pack.yaml': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures/stackstorm-test4': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests/fixtures': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs/tests': Operation not permitted
/bin/cp: failed to preserve ownership for '/opt/stackstorm/packs-shared/./packs': Operation not permitted
/bin/cp: preserving times for '/opt/stackstorm/packs-shared/.': Operation not permitted
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
Error from server (BadRequest): container "st2-pack-configs-from-helm" in pod "stackstorm-job-st2-register-content-pzrpn" is waiting to start: PodInitializing
cognifloyd commented 3 years ago

Was that with NFS or the PVC? What is the definition for your PersistentVolume? What are the mountOptions? I assume that is using NFS as well, right?

elsopapa commented 3 years ago

The definition 


   volumes:
      enabled: true
      packs: 
        # mounted to /opt/stackstorm/packs
        # packs volume definition is required if st2.packs.volumes is enabled

        # example using persistentVolumeClaim:

         nfs:
          server: "bcnfs03.bue299.comafi.com.ar"
          path: /nfs/stackstorm/packs

        # example using a flexVolume + rook-ceph
        #flexVolume:
        #  driver: ceph.rook.io/rook
        #  fsType: ceph
        #  options:
        #    fsName: fs1
        #    clusterNamespace: rook-ceph
        #    path: /st2/packs

      virtualenvs: 
        # mounted to /opt/stackstorm/virtualenvs
        # virtualenvs volume definition is required if st2.packs.volumes is enabled
        # see the examples under st2.packs.volumes.packs
   #      persistentVolumeClaim:
   #       claimName: pvc-st2-virtualenvs
         nfs:
          server: "bcnfs03.bue299.comafi.com.ar"
          path: /nfs/stackstorm/virtualenv

      configs: 
        # mounted to /opt/stackstorm/configs
        # configs volume definition is optional, but only used if st2.packs.volumes is enabled
        # Anything in `st2.packs.configs` will be added to this volume automatically on helm install/upgrade.
        # see the examples under st2.packs.volumes.packs
   #     persistentVolumeClaim:
   #      claimName: pvc-st2-configs

        nfs:
          server: "bcnfs03.bue299.comafi.com.ar"
          path: /nfs/stackstorm/config
elsopapa commented 3 years ago

Name: nfs-client IsDefaultClass: Yes Annotations: meta.helm.sh/release-name=nfs-subdir-external-provisioner,meta.helm.sh/release-namespace=default,storageclass.kubernetes.io/is-default-class=true Provisioner: cluster.local/nfs-subdir-external-provisioner Parameters: archiveOnDelete=true AllowVolumeExpansion: True MountOptions: ReclaimPolicy: Delete VolumeBindingMode: Immediate Events:

cognifloyd commented 3 years ago

Since NFS is changing the ownership on the fly (the root_squash option), NFS clients cannot change ownership of NFS files.

That suggests that we need something a little more robust for copying pack contents. At one point, someone tried using rsync instead of cp, but it's not available in the st2packs image, so they reverted it: https://github.com/StackStorm/stackstorm-ha/pull/160#issuecomment-734927852

Do you have any recommendations for how to modify these lines so that they preserve ownership and permissions when possible without failing when the underlying filesystem doesn't support them?

https://github.com/StackStorm/stackstorm-ha/blob/21108182a1475f8ef8c87e3a3e2cbe1fb79ce579/templates/_helpers.tpl#L204-L205 https://github.com/StackStorm/stackstorm-ha/blob/21108182a1475f8ef8c87e3a3e2cbe1fb79ce579/templates/_helpers.tpl#L222-L223 https://github.com/StackStorm/stackstorm-ha/blob/21108182a1475f8ef8c87e3a3e2cbe1fb79ce579/templates/_helpers.tpl#L239

cognifloyd commented 3 years ago

Looking at your logs, we only have two errors we need to work around: preserving ownership and timestamps.

/bin/cp: failed to preserve ownership for '...': Operation not permitted
/bin/cp: preserving times for '...': Operation not permitted

Other filesystems might have additional restrictions (such as changing permissions).

cognifloyd commented 3 years ago

For now, maybe you could try changing your NFS export options. There have been several attempts to add persistent volumes to the chart. In one of those tries, based on NFS, the README suggested exports like: https://github.com/StackStorm/stackstorm-ha/blob/ef206060fe045ce5f8e5910e39a9287fcaa4a7b4/README.md#L218-L220

@elsopapa Can you change your NFS export options? Or do you have to use root_squash?

elsopapa commented 3 years ago

I Will change the options of nfs export and give a try.

El mar., 7 de septiembre de 2021 19:39, Jacob Floyd < @.***> escribió:

For now, maybe you could try changing your NFS export options. There have been several attempts to add persistent volumes to the chart. In one of those tries, based on NFS, the README suggested exports like:

https://github.com/StackStorm/stackstorm-ha/blob/ef206060fe045ce5f8e5910e39a9287fcaa4a7b4/README.md#L218-L220

@elsopapa https://github.com/elsopapa Can you change your NFS export options? Or do you have to use root_squash?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/StackStorm/stackstorm-ha/issues/238#issuecomment-914672590, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACK7TXKKJU7MCH5QW4EKWNDUA2IDXANCNFSM5DLVHRKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

arm4b commented 3 years ago

@cognifloyd we can definitely include rsync in the st2packs image as we added a new content-sharing option.

cognifloyd commented 3 years ago

If we add rsync to st2packs we should probably change these commands so that they use rsync if available, but fall back to using cp so that existing users don't have to rebuild all their custom images.

elsopapa commented 3 years ago

The problem was the options in the export file . Ive changed to no_root_squash and everything is working now.

Thank you for your patience and support!!!!

elsopapa commented 3 years ago

This is not an issue. thanks again