Closed mhrivnak closed 6 years ago
I'm starting to think it's a kubernetes bug. The service bundle's pod definition has the correct data:
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
openshift.io/scc: restricted
creationTimestamp: 2018-02-28T17:33:21Z
labels:
apb-action: provision
apb-fqname: dh-redis-apb
apb-pod-name: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479
name: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479
namespace: dh-redis-apb-prov-xj6pj
resourceVersion: "9907"
selfLink: /api/v1/namespaces/dh-redis-apb-prov-xj6pj/pods/apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479
uid: 78d5990e-1cad-11e8-8dbc-68f72877eaca
spec:
containers:
- args:
- provision
- --extra-vars
- '{"_apb_plan_id":"default","_apb_service_class_id":"4b12a667a4ffdcaf61a52c7f72c3fe67","_apb_service_instance_id":"2886e33a-4fae-479c-96b9-141dbf8a3b03","cluster":"openshift","namespace":"foo3","values":"##
Bitnami Redis image version\n## ref: https://hub.docker.com/r/bitnami/redis/tags/\n##\nimage:
bitnami/redis:4.0.8-r0\n\n## Specify a imagePullPolicy\n## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images\n##\nimagePullPolicy:
IfNotPresent\n\n## Kubernetes service type\nserviceType: ClusterIP\n\n## Pod
Security Context\nsecurityContext:\n enabled: false\n fsGroup: 1001\n runAsUser:
1001\n\n## Use password authentication\nusePassword: true\n\n## Redis password\n##
Defaults to a random 10-character alphanumeric string if not set and usePassword
is true\n## ref: https://github.com/bitnami/bitnami-docker-redis#setting-the-server-password-on-first-run\n##\n#
redisPassword:\n\n## Redis command arguments\n##\n## Can be used to specify
command line arguments, for example:\n##\n## args:\n## - \"redis-server\"\n## -
\"--maxmemory-policy volatile-ttl\"\nargs:\n\n## Enable persistence using
Persistent Volume Claims\n## ref: http://kubernetes.io/docs/user-guide/persistent-volumes/\n##\npersistence:\n enabled:
true\n\n ## The path the volume will be mounted at, useful when using different\n ##
Redis images.\n path: /bitnami\n\n ## The subdirectory of the volume to
mount to, useful in dev environments and one PV for multiple services.\n subPath:
\"\"\n\n ## A manually managed Persistent Volume and Claim\n ## Requires
persistence.enabled: true\n ## If defined, PVC must be created manually before
volume will be bound\n # existingClaim:\n\n ## redis data Persistent Volume
Storage Class\n ## If defined, storageClassName: \u003cstorageClass\u003e\n ##
If set to \"-\", storageClassName: \"\", which disables dynamic provisioning\n ##
If undefined (the default) or set to null, no storageClassName spec is\n ## set,
choosing the default provisioner. (gp2 on AWS, standard on\n ## GKE, AWS
\u0026 OpenStack)\n ##\n # storageClass: \"-\"\n accessMode: ReadWriteOnce\n size:
8Gi\n\nmetrics:\n enabled: false\n image: oliver006/redis_exporter\n imageTag:
v0.11\n imagePullPolicy: IfNotPresent\n resources: {}\n annotations:\n prometheus.io/scrape:
\"true\"\n prometheus.io/port: \"9121\"\n\n## Configure resource requests
and limits\n## ref: http://kubernetes.io/docs/user-guide/compute-resources/\n##\nresources:\n requests:\n memory:
256Mi\n cpu: 100m\n\n## Node labels and tolerations for pod assignment\n##
ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector\n##
ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#taints-and-tolerations-beta-feature\nnodeSelector:
{}\ntolerations: []\n\n## Additional pod labels\n## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/\npodLabels:
{}\n\n## annotations for redis pods\npodAnnotations: {}\n\nnetworkPolicy:\n ##
Enable creation of NetworkPolicy resources.\n ##\n enabled: false\n\n ##
The Policy model to apply. When set to false, only pods with the correct\n ##
client label will have network access to the port Redis is listening\n ##
on. When true, Redis will accept connections from any source\n ## (with the
correct destination port).\n ##\n allowExternal: true\n\nservice:\n annotations:
{}\n loadBalancerIP:"}'
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: docker.io/mhrivnak/redis-helm-apb:latest
imagePullPolicy: IfNotPresent
name: apb
resources: {}
securityContext:
capabilities:
drop:
- KILL
- MKNOD
- SETGID
- SETUID
runAsUser: 1000220000
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479-token-pgd8q
readOnly: true
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479-dockercfg-2kwdz
nodeName: localhost
restartPolicy: Never
schedulerName: default-scheduler
securityContext:
fsGroup: 1000220000
seLinuxOptions:
level: s0:c15,c5
serviceAccount: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479
serviceAccountName: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479
terminationGracePeriodSeconds: 30
volumes:
- name: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479-token-pgd8q
secret:
defaultMode: 420
secretName: apb-55a1cd1d-905f-4dcd-81ae-bd2ec9b38479-token-pgd8q
status:
conditions:
- lastProbeTime: null
lastTransitionTime: 2018-02-28T17:33:21Z
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: 2018-02-28T17:33:25Z
message: 'containers with unready status: [apb]'
reason: ContainersNotReady
status: "False"
type: Ready
- lastProbeTime: null
lastTransitionTime: 2018-02-28T17:33:21Z
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://7ad5386631b53d1053336919cc878f456729ddcfe091916dbe51de8b7459cd52
image: docker.io/mhrivnak/redis-helm-apb:latest
imageID: docker-pullable://docker.io/mhrivnak/redis-helm-apb@sha256:d7f65c86e4ac36aa70adad90a8929c36d3ad99c28c3398d6813c40a49af63457
lastState: {}
name: apb
ready: false
restartCount: 0
state:
terminated:
containerID: docker://7ad5386631b53d1053336919cc878f456729ddcfe091916dbe51de8b7459cd52
exitCode: 1
finishedAt: 2018-02-28T17:33:24Z
reason: Error
startedAt: 2018-02-28T17:33:23Z
hostIP: 10.13.57.139
phase: Failed
podIP: 172.17.0.3
qosClass: BestEffort
startTime: 2018-02-28T17:33:21Z
kind: List
metadata:
resourceVersion: ""
selfLink: ""
One option is to base64 encode the payload at the moment it enters the broker. Then pass the encoded value to the APB where it will decode it. Like you mentioned this would be a bandage but you've already ruled out the broker from being the culprit. And the value the APB is getting at the entrypoint script is mangled, that means it is happening somewhere between the pod definition and the container being created.
This is not a broker bug after all. But it is a good reminder to quote variable names when working with JSON in a bash script.
The bug in my entrypoint.sh
basically came down to the difference between these two echo
commands:
$ foo='a b'
$ echo $foo
a b
$ echo "$foo"
a b
What happened: The service catalog made a provision request that included a parameter whose value was a string. The string happened to contain YAML. That parameter name and value were expected to be passed through as-is to the service bundle.
When passed into the running service bundle, the string had changed to remove a lot of white space. This caused YAML indentation to change, which fundamentally changed/mangled the structure of that data.
What you expected to happen: I expect the string to be preserved.
How to reproduce it: Make a request like this one:
Compare the contents of the
values
parameter with the data that gets passed into the running service bundle on the command line:JSON blobs pretty-printed
Looking in the
values
parameter value, the first example I see of whitespace change is just before the word "enabled".Request Body
Service Bundle command-line argument: