Closed YanFenGuo closed 4 years ago
Hi @YanFenGuo,
you need to add the ENV VAR in the operator.ymal and re-deploy it. See: here the example.
All ansible logs will be output in the container logs. Note that the Ansible logs will be outputted with the operator ones in the same way. It will increase the verbose which means output all ansible logs that we could find in the old versions in the ansible container.
I am closing this one as sorted out. However, feel free to ping us to re-open if you need.
Hi @YanFenGuo,
you need to add the ENV VAR in the operator.ymal and re-deploy it. See: here the example.
All ansible logs will be output in the container logs. Note that the Ansible logs will be outputted with the operator ones in the same way. It will increase the verbose which means output all ansible logs that we could find in the old versions in the ansible container.
I am closing this one as sorted out. However, feel free to ping us to re-open if you need.
Thank you for the quick response, your example is exactly what I did. But there is no difference, still can't get full log. Here is my deployment yaml file:
###############################################################################
#
# Licensed Materials - Property of IBM
#
# (C) Copyright IBM Corp. 2019. All Rights Reserved.
#
# US Government Users Restricted Rights - Use, duplication or
# disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
#
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
name: ibm-cp4a-operator
labels:
app.kubernetes.io/instance: ibm-dba
app.kubernetes.io/managed-by: ibm-dba
app.kubernetes.io/name: ibm-dba
release: 20.0.3
spec:
****
spec:
******
containers:
- name: icp4a-operator
# Replace this with the built image name
image: ****
imagePullPolicy: "Always"
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: false
runAsNonRoot: true
capabilities:
drop:
- ALL
resources:
limits:
cpu: '1'
memory: 1Gi
requests:
cpu: 500m
memory: 256Mi
volumeMounts:
- mountPath: /tmp/ansible-operator/runner
name: runner
- mountPath: /opt/ansible/share
name: operator-shared-folder
args:
- '--zap-time-encoding=iso8601'
- '--zap-encoder=console'
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "ibm-cp4a-operator"
- name: WORKER_FOOSERVICE_CACHE_EXAMPLE_COM
value: "10"
- name: ANSIBLE_GATHERING
value: explicit
- name: ANSIBLE_DEBUG_LOGS
value: "True"
# - name: ANSIBLE_VERBOSITY
# value: "4"
- name: dba_license
value: accept
imagePullSecrets:
- name: "admin.registrykey.operator"
volumes:
- name: runner
emptyDir: {}
- name: "operator-shared-folder"
persistentVolumeClaim:
claimName: "operator-shared-pvc"
I am unable to reproduce your issue.
See here that it will output the Ansible logs inside of the --------------------------- Ansible Debug Result -----------------------------
in the container logs.
Could you please check if you are unable to fin the Ansible Debug Result?
Also, I'd like to recommend you move forward and migrate and upgrade your project to the new layout to use the version 1.0.0. Note that 0.19.x and 1.0.0 are the versions supported. See here the migration guide.
I confirm there is no Ansible Debug Result
in my log.
We initially intend to use 0.19.2 but we are suggested to use 0.17.2 because " 0.19.2 is not supported with OCP 4.5. Also the version 0.19.2 does NOT get support from redhat yet."
According to documentation https://v0-17-x.sdk.operatorframework.io/docs/ansible/quickstart/, this feature should be supported in 0.17.*
@YanFenGuo,
0.19.2 is not supported with OCP 4.5
What do you mean with? Where you check this information? Are you using the downstream image?
Also, could you please share your Dockerfile?
According to documentation https://v0-17-x.sdk.operatorframework.io/docs/ansible/quickstart/, this feature should be supported in 0.17.*
Note that we can use the Memcached Sample tag 0.17.0 to ensure that it works succefully. Could you please check it with the Sample?
@camilamacedo86 ,
What do you mean with? Where you check this information? Are you using the downstream image?
I'm an IBMer and I get this info from our Cloud Pak Certification team when we onboard OLM.
Also, could you please share your Dockerfile?
I'm afraid I'm not allowed to share the Dockerfile but this is the base image we are using: FROM quay.io/operator-framework/ansible-operator:v0.17.2
Could you please check it with the Sample?
Thank you, I'll check the sample to see if it works on it later.
HI @YanFenGuo,
The versions supported are 0.19.x and 1.0.0 currently. See indeed the Operator-SDK Backport Policy.
I'd recommend you upgrade all your projects to 1.0.0 which is the stable and more mature version. See here the migration guide.
The above versions should work with OCP. I think that has some misunderstanding in the info provided to you. You are using the upstream images. What is not provided yet to downstream are images based on the latest release e.g https://catalog.redhat.com/software/containers/openshift4/ose-helm-operator/5e6bbb8f5a1346687692f501. However, you are not using the downstream images at all. So, I truly recommend you move forward to the upper version. See that the legacy layout will no longer be supported for the future versions.
Hi @jmrodri, @dmesser it shows more one case/scenario that we need to address related to downstream which we might need to find a better way to communicate as well.
@camilamacedo86 My test on the memcached example can't pass because no Ansible log output:
{"level":"info","ts":1597916702.1923316,"logger":"cmd","msg":"Go Version: go1.13.10"}
{"level":"info","ts":1597916702.1923697,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1597916702.1923745,"logger":"cmd","msg":"Version of operator-sdk: v0.17.0"}
{"level":"info","ts":1597916702.192904,"logger":"cmd","msg":"Watching single namespace.","Namespace":"memcached"}
{"level":"info","ts":1597916704.5493422,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1597916704.549793,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"WORKER_MEMCACHED_CACHE_EXAMPLE_COM","default":1}
{"level":"info","ts":1597916704.5498126,"logger":"watches","msg":"Environment variable not set; using default value","envVar":"ANSIBLE_VERBOSITY_MEMCACHED_CACHE_EXAMPLE_COM","default":2}
{"level":"error","ts":1597916704.5498512,"logger":"watches","msg":"Invalid ansible path for GVK: cache.example.com/v1alpha1, Kind=Memcached","error":"role: memcached was not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/watches.(*Watch).Validate\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/watches/watches.go:231\ngithub.com/operator-framework/operator-sdk/pkg/ansible/watches.Load\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/watches/watches.go:298\ngithub.com/operator-framework/operator-sdk/pkg/ansible.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/run.go:122\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunAnsibleCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/ansible.go:44\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.10.linux.amd64/src/runtime/proc.go:203"}
{"level":"error","ts":1597916704.5499146,"logger":"watches","msg":"Watch with GVK cache.example.com/v1alpha1, Kind=Memcached failed validation","error":"role: memcached was not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/watches.Load\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/watches/watches.go:300\ngithub.com/operator-framework/operator-sdk/pkg/ansible.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/run.go:122\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunAnsibleCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/ansible.go:44\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.10.linux.amd64/src/runtime/proc.go:203"}
{"level":"error","ts":1597916704.5499678,"logger":"cmd","msg":"Failed to load watches.","Namespace":"memcached","error":"role: memcached was not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible.Run\n\tsrc/github.com/operator-framework/operator-sdk/pkg/ansible/run.go:124\ngithub.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint.newRunAnsibleCmd.func1\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/execentrypoint/ansible.go:44\ngithub.com/spf13/cobra.(*Command).execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914\ngithub.com/spf13/cobra.(*Command).Execute\n\tpkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864\nmain.main\n\tsrc/github.com/operator-framework/operator-sdk/cmd/operator-sdk/main.go:39\nruntime.main\n\t/home/travis/.gimme/versions/go1.13.10.linux.amd64/src/runtime/proc.go:203"}
time="2020-08-20T09:45:04Z" level=fatal msg="role: memcached was not found"
Even I created the role:
[root@ocp1 memcached-operator]# oc get role
NAME CREATED AT
memcached 2020-08-20T09:39:30Z
memcached-operator 2020-08-20T09:30:00Z
I think this maybe this is the reason we stay on 0.17.2: https://docs.openshift.com/container-platform/4.5/release_notes/ocp-4-5-release-notes.html#ocp-4-5-notable-technical-changes
Did you install it with make install
? The error shows that the manifests are not all applied.
Did you install it with
make install
? The error shows that the manifests are not all applied.
Yes, I follow exactly these steps: https://github.com/operator-framework/operator-sdk-samples/tree/v0.17.0/ansible/memcached-operator
Hi @YanFenGuo,
I tested it with the Memcached sample tag v0.17.0 and operator-sdk version 0.17.0 with a Kind cluster. I found an issue with the role name in the watches file. After fixing it, ALL worked fine and I am able to check all ansible logs. See:
To fix the Memcached Sample 0.17.0 error which has no relation with the Ansible logs at all
Error;
time="2020-08-21T10:29:57Z" level=fatal msg="role: memcached was not found"
Update the watches file with the correct role name:
$ cat watches.yaml
---
- version: v1alpha1
group: cache.example.com
kind: Memcached
role: Memcached
You were facing a FATAL error and because of this, you will NOT get the Ansible logs. The Ansible roles/playbooks were not executed at all. Please, check your logs.
Following all steps executed to run the sample after the above fix and the logs with the Ansible ones as well.
$ operator-sdk version
operator-sdk version: "v0.17.0", commit: "2fd7019f856cdb6f6618e2c3c80d15c3c79d1b6c", kubernetes version: "v1.17.2", go version: "go1.13.12 darwin/amd64"
Building
$ export IMAGE=quay.io/camilamacedo86/memcached-operator:log
$ operator-sdk build $IMAGE
INFO[0000] Building OCI image quay.io/camilamacedo86/memcached-operator:log
Sending build context to Docker daemon 78.85kB
Step 1/5 : FROM quay.io/operator-framework/ansible-operator:v0.17.0
---> 5d34c0760fd5
Step 2/5 : COPY requirements.yml ${HOME}/requirements.yml
---> Using cache
---> ea7cf7920991
Step 3/5 : RUN ansible-galaxy collection install -r ${HOME}/requirements.yml && chmod -R ug+rwx ${HOME}/.ansible
---> Using cache
---> f7f9366a04d2
Step 4/5 : COPY watches.yaml ${HOME}/watches.yaml
---> Using cache
---> b1ecba5ae744
Step 5/5 : COPY roles/ ${HOME}/roles/
---> Using cache
---> 50388993e07f
Successfully built 50388993e07f
Successfully tagged quay.io/camilamacedo86/memcached-operator:log
INFO[0000] Operator build complete.
Pushing
$ docker push $IMAGE
The push refers to repository [quay.io/camilamacedo86/memcached-operator]
6d7fa781a4a5: Pushed
464da4ce5a76: Pushed
0a42b8b78204: Pushed
679b81b2bb74: Pushed
f210119c57b3: Layer already exists
14b6a122ad02: Layer already exists
1ceb45c5a64a: Layer already exists
458a464992fd: Layer already exists
4a9fb1e36b8f: Layer already exists
f32acfd05b3b: Layer already exists
56c5d564c4e5: Layer already exists
668db11eda93: Layer already exists
d3ada5af5602: Layer already exists
v0.0.1: digest: sha256:963063404bdfce02581b5026922ea9e001c2cd084581589a5cf5753f201f03bf size: 3028
operator.yaml
manifests$ sed -i 's|REPLACE_IMAGE|quay.io/camilamacedo86/memcached-operator:log|g' deploy/operator.yaml
See:
$ cat deploy/operator.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: memcached-operator
spec:
replicas: 1
selector:
matchLabels:
name: memcached-operator
template:
metadata:
labels:
name: memcached-operator
spec:
serviceAccountName: memcached-operator
containers:
- name: memcached-operator
# Replace this with the built image name
image: "quay.io/camilamacedo86/memcached-operator:log"
...
Running make install
Checking that all is OK:
$ kubectl get all -n memcached
NAME READY STATUS RESTARTS AGE
pod/example-memcached-memcached-6456bdd5fc-v9db4 1/1 Running 0 6m56s
pod/example-memcached-memcached-6456bdd5fc-vxmx5 1/1 Running 0 6m56s
pod/example-memcached-memcached-6456bdd5fc-w5q55 1/1 Running 0 6m56s
pod/memcached-operator-756699f96c-xrfs9 1/1 Running 0 7m6s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/memcached-operator-metrics ClusterIP 10.96.151.116 <none> 8383/TCP,8686/TCP 7m3s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/example-memcached-memcached 3/3 3 3 6m56s
deployment.apps/memcached-operator 1/1 1 1 7m6s
NAME DESIRED CURRENT READY AGE
replicaset.apps/example-memcached-memcached-6456bdd5fc 3 3 3 6m56s
replicaset.apps/memcached-operator-756699f96c 1 1 1 7m6s
$ kubectl logs deployment.apps/memcached-operator -n memcached | grep Result
--------------------------- Ansible Debug Result -----------------------------
--------------------------- Ansible Debug Result -----------------------------
--------------------------- Ansible Debug Result -----------------------------
ANSIBLE DEBUG LOG
Env var is working well as expected with 0.17.0 and upper versions
- ensure that your project is running succefully (use kubect get all -n namespace and kubectl get events -n namespace )
- troboolshoting your project and ensure that no errors are faced.
- ensure that you are using the same SDK tool version of the project Ansible Operator-based image (check the dockerfile and run operator-sdk version)
It is the right time to upgrade your project to use the new layout and SDK 1.0 version. The version used by you, 0.17.0, is no longer supported. See here the migration guide.
Also, I am closing it as sorted out see that;
How to get full Ansible log from Operator pod?
You need to add the ENV VAR in the operator.ymal and re-deploy it. See: here the example.
We could ensure and confirm that it has been working well with 0.17. See: https://github.com/operator-framework/operator-sdk/issues/3755#issuecomment-678229639
However, please feel free to raise new issues and/or your ping us if you need.
Type of question
Previously we ask our customer to export Ansible logs with this command:
kubectl logs deployment/ibm-cp4a-operator -c ansible > Ansible.log
. But since operator-sdk:0.18.0, the side-car Ansible container is removed. While in operator container, the Ansible log doesn't have enough detail. I know in documentation there is a variableANSIBLE_DEBUG_LOGS: True
, but even with it, still can't get the complete log as it shows in/tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>
, actually, I don't see a difference with or withoutANSIBLE_DEBUG_LOGS: True
, maybe I'm not checking the right place?What did you expect to see? What I'm seeking is without going into operator pod, can get a complete Ansible log with detailed information. For example the same task shows below in
/tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>
:But from the pod log console, with or without
ANSIBLE_DEBUG_LOGS: True
, it is always just this:Environment I'm using 0.17.2 but I'm assuming the higher version have similar result.
operator-sdk version: 0.17.2
Kubernetes version information: v1.16.2
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.2", GitCommit:"a02f27a", GitTreeState:"clean", BuildDate:"2020-04-13T12:04:13Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}