search

openshift / ansible-service-broker

Ansible Service Broker
Apache License 2.0
226 stars 84 forks source link

apb provisioning sets ServiceInstance as successful when apb fails #506

Closed jaymccon closed 7 years ago

jaymccon commented 7 years ago

Bug: when the provision playbook exits non-zero the ServiceInstance still get marked as successfully provisioned.

What happened: When provisioning an apb, and the provision playbook fails, the service catalog shows no sign of error and shows that the ServiceInstance (Provisioned Service in the gui) was successfully provisioned.

What you expected to happen: the failure is shown in the api and the gui (hopefully with the ansible error message)

How to reproduce it: launch an apb with parameters that will cause failure, an easy/quick one is the sns apb using the Topic with Subscription plan. select email for the subscription protocol and enter something that's not an email address for the subscription endpoint. The cloudformation stack should launch, fail due to the invalid address and then the playbook fails and exits 2. When the playbook exits the ServiceInstance is marked as successfully provisioned.

ServiceInstance:

$ oc describe ServiceInstance dh-sns-apb-8s2ss
Name:       dh-sns-apb-8s2ss
Namespace:  test-sns
Labels:     <none>
Events:
  FirstSeen LastSeen    Count   From                    SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----                    -------------   --------    ------          -------
  11m       11m     1   service-catalog-controller-manager          Normal      Provisioning        The instance is being provisioned asynchronously
  10m       10m     1   service-catalog-controller-manager          Normal      ProvisionedSuccessfully The instance was provisioned successfully

apb logs:

+ [[ provision --extra-vars {"SubscriptionEndPoint":"this-is-not-an-email-addess","SubscriptionProtocol":"email","_apb_plan_id":"sns-topicwithsub","namespace":"test-sns"} == *\s\2\i\/\a\s\s\e\m\b\l\e* ]]
+ ACTION=provision
+ shift
+ playbooks=/opt/apb/actions
+ CREDS=/var/tmp/bind-creds
+ TEST_RESULT=/var/tmp/test-result
+ whoami
+ '[' -w /etc/passwd ']'
++ id -u
+ echo 'apb:x:1000080000:0:apb user:/opt/apb:/sbin/nologin'
+ oc-login.sh
Attempting to login with a service account...
Logged into "https://kubernetes.default:443" as "system:serviceaccount:aws-service-broker:apb-052cdb85-12d7-4d26-9988-eb67732a34f6" using the token provided.
You have access to the following projects and can switch between them with 'oc project <projectname>':
  * aws-service-broker
    test-sns
Using project "aws-service-broker".
Welcome! See 'oc help' to get started.
+ set +x
+ [[ -e /opt/apb/actions/provision.yaml ]]
+ [[ -e /opt/apb/actions/provision.yml ]]
+ ANSIBLE_ROLES_PATH=/etc/ansible/roles:/opt/ansible/roles
+ ansible-playbook /opt/apb/actions/provision.yml --extra-vars '{"SubscriptionEndPoint":"this-is-not-an-email-addess","SubscriptionProtocol":"email","_apb_plan_id":"sns-topicwithsub","namespace":"test-sns"}' --extra-vars no_log=true --extra-vars @/tmp/secrets
 [WARNING]: provided hosts list is empty, only localhost is available
 [WARNING]: Found variable using reserved name: no_log
PLAY [Deploy sns to openshift] *************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [ansible.kubernetes-modules : Install latest openshift client] ************
skipping: [localhost]
TASK [ansibleplaybookbundle.asb-modules : debug] *******************************
skipping: [localhost]
TASK [provision-sns-apb-openshift : include plan vars] *************************
ok: [localhost]
TASK [provision-sns-apb-openshift : generate b62 hash] *************************
changed: [localhost]
TASK [provision-sns-apb-openshift : set fact] **********************************
ok: [localhost]
TASK [provision-sns-apb-openshift : Launch SNS topic] **************************
changed: [localhost]
TASK [provision-sns-apb-openshift : Check for SNS CloudFormation create error logs] ***
ok: [localhost] => {
    "sns.log": [
        "AWS::SNS::Subscription SNSSubscription CREATE_FAILED: Invalid parameter: Email address"
    ]
}
TASK [provision-sns-apb-openshift : Create IAM user] ***************************
changed: [localhost]
TASK [provision-sns-apb-openshift : Generate IAM policy temporary json file] ***
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'TopicARN'"}
 [WARNING]: Could not create retry file '/opt/apb/actions/provision.retry'.
[Errno 13] Permission denied: u'/opt/apb/actions/provision.retry'
PLAY RECAP *********************************************************************
localhost                  : ok=7    changed=3    unreachable=0    failed=1   
+ EXIT_CODE=2
+ set +ex
+ '[' -f /var/tmp/test-result ']'
+ '[' -f /var/tmp/bind-creds ']'
+ exit 2

asb logs:

[2017-10-19T12:40:04.279Z] [DEBUG] Auto Escalate has been set to true, we are escalating permissions
[2017-10-19T12:40:04.28Z] [DEBUG] Injecting PlanID as parameter: { _apb_plan_id: sns-topicwithsub }
[2017-10-19T12:40:04.282Z] [INFO] ASYNC provisioning in progress
[2017-10-19T12:40:04.282Z] [NOTICE] ============================================================
[2017-10-19T12:40:04.282Z] [NOTICE]                        PROVISIONING                         
[2017-10-19T12:40:04.282Z] [NOTICE] ============================================================
[2017-10-19T12:40:04.282Z] [NOTICE] Spec.ID: 6f418d63aefb3f622d58117b2cbba90c
[2017-10-19T12:40:04.282Z] [NOTICE] Spec.Name: dh-sns-apb
[2017-10-19T12:40:04.282Z] [NOTICE] Spec.Image: docker.io/jaytestcatasborg/sns-apb:latest
[2017-10-19T12:40:04.282Z] [NOTICE] Spec.Description: Managed sns topic
[2017-10-19T12:40:04.282Z] [NOTICE] ============================================================
[2017-10-19T12:40:04.282Z] [INFO] Checking if project aws-service-broker exists...
172.17.0.7 - - [19/Oct/2017:12:40:04 +0000] "PUT /aws-service-broker/v2/service_instances/3ec651a4-52e6-4eac-a20a-11813d5e6e37?accepts_incomplete=true HTTP/1.1" 202 58
[2017-10-19T12:40:04.476Z] [DEBUG] Auto Escalate has been set to true, we are escalating permissions
[2017-10-19T12:40:04.477Z] [DEBUG] Injecting PlanID as parameter: { _apb_plan_id: sns-topicwithsub }
[2017-10-19T12:40:04.478Z] [DEBUG] already have this instance returning 200
172.17.0.7 - - [19/Oct/2017:12:40:04 +0000] "PUT /aws-service-broker/v2/service_instances/3ec651a4-52e6-4eac-a20a-11813d5e6e37?accepts_incomplete=true HTTP/1.1" 200 3
[2017-10-19T12:40:04.539Z] [DEBUG] ExecutingApb:
[2017-10-19T12:40:04.539Z] [DEBUG] name:[ dh-sns-apb ]
[2017-10-19T12:40:04.539Z] [DEBUG] image:[ docker.io/jaytestcatasborg/sns-apb:latest ]
[2017-10-19T12:40:04.539Z] [DEBUG] action:[ provision ]
[2017-10-19T12:40:04.539Z] [DEBUG] pullPolciy:[ IfNotPresent ]
[2017-10-19T12:40:04.539Z] [DEBUG] role:[ edit ]
[2017-10-19T12:40:04.539Z] [DEBUG] Creating resource file dir: /tmp/asb-resource-files
[2017-10-19T12:40:04.539Z] [DEBUG] Creating resource file /tmp/asb-resource-files/apb-df743b94-6f60-4153-b211-1e0c70bac227.yaml
[2017-10-19T12:40:04.539Z] [INFO] Successfully wrote resources to /tmp/asb-resource-files/apb-df743b94-6f60-4153-b211-1e0c70bac227.yaml
[2017-10-19T12:40:04.539Z] [DEBUG] Trying to create apb sandbox: [ apb-df743b94-6f60-4153-b211-1e0c70bac227 ], with  edit permissions in namespace aws-service-broker
[2017-10-19T12:40:04.539Z] [DEBUG] Creating resources from file at path: /tmp/asb-resource-files/apb-df743b94-6f60-4153-b211-1e0c70bac227.yaml
[2017-10-19T12:40:04.679Z] [DEBUG] service_id: 6f418d63aefb3f622d58117b2cbba90c
[2017-10-19T12:40:04.679Z] [DEBUG] plan_id: 5a6570a738ee3b996901f9d8a91cae2f
[2017-10-19T12:40:04.679Z] [DEBUG] operation:  4a323bf4-aaaf-4d4f-80a7-721c5bc19f93
172.17.0.7 - - [19/Oct/2017:12:40:04 +0000] "GET /aws-service-broker/v2/service_instances/3ec651a4-52e6-4eac-a20a-11813d5e6e37/last_operation?operation=4a323bf4-aaaf-4d4f-80a7-721c5bc19f93&plan_id=5a6570a738ee3b996901f9d8a91cae2f&service_id=6f418d63aefb3f622d58117b2cbba90c HTTP/1.1" 200 29
[2017-10-19T12:40:04.793Z] [ERROR] Something went wrong trying to create resources in cluster
[2017-10-19T12:40:04.793Z] [ERROR] Returned error:
[2017-10-19T12:40:04.793Z] [ERROR] exit status 1
[2017-10-19T12:40:04.793Z] [ERROR] oc create -f output:
[2017-10-19T12:40:04.793Z] [ERROR] serviceaccount "apb-df743b94-6f60-4153-b211-1e0c70bac227" created
rolebinding "apb-df743b94-6f60-4153-b211-1e0c70bac227" created
Error from server (AlreadyExists): error when creating "/tmp/asb-resource-files/apb-df743b94-6f60-4153-b211-1e0c70bac227.yaml": rolebindings "apb-df743b94-6f60-4153-b211-1e0c70bac227" already exists

[2017-10-19T12:40:04.793Z] [INFO] Successfully created apb sandbox: [ apb-df743b94-6f60-4153-b211-1e0c70bac227 ], with edit permissions in namespace aws-service-broker
[2017-10-19T12:40:04.793Z] [NOTICE] Creating pod "apb-df743b94-6f60-4153-b211-1e0c70bac227" in the aws-service-broker namespace
[2017-10-19T12:40:04.8Z] [DEBUG] Calling monitorOutput on apb-df743b94-6f60-4153-b211-1e0c70bac227
[2017-10-19T12:40:05.003Z] [INFO] error: unable to upgrade connection: container not found ("apb")

[2017-10-19T12:40:05.003Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 1: Failed to exec into the container
[2017-10-19T12:40:05.003Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 1: exec into apb-df743b94-6f60-4153-b211-1e0c70bac227 failed
[2017-10-19T12:40:05.474Z] [DEBUG] service_id: 6f418d63aefb3f622d58117b2cbba90c
[2017-10-19T12:40:05.474Z] [DEBUG] plan_id: 5a6570a738ee3b996901f9d8a91cae2f
[2017-10-19T12:40:05.474Z] [DEBUG] operation:  4a323bf4-aaaf-4d4f-80a7-721c5bc19f93
172.17.0.7 - - [19/Oct/2017:12:40:05 +0000] "GET /aws-service-broker/v2/service_instances/3ec651a4-52e6-4eac-a20a-11813d5e6e37/last_operation?operation=4a323bf4-aaaf-4d4f-80a7-721c5bc19f93&plan_id=5a6570a738ee3b996901f9d8a91cae2f&service_id=6f418d63aefb3f622d58117b2cbba90c HTTP/1.1" 200 29
[2017-10-19T12:40:09.488Z] [DEBUG] service_id: 6f418d63aefb3f622d58117b2cbba90c
[2017-10-19T12:40:09.488Z] [DEBUG] plan_id: 5a6570a738ee3b996901f9d8a91cae2f
[2017-10-19T12:40:09.488Z] [DEBUG] operation:  4a323bf4-aaaf-4d4f-80a7-721c5bc19f93
172.17.0.7 - - [19/Oct/2017:12:40:09 +0000] "GET /aws-service-broker/v2/service_instances/3ec651a4-52e6-4eac-a20a-11813d5e6e37/last_operation?operation=4a323bf4-aaaf-4d4f-80a7-721c5bc19f93&plan_id=5a6570a738ee3b996901f9d8a91cae2f&service_id=6f418d63aefb3f622d58117b2cbba90c HTTP/1.1" 200 29
[2017-10-19T12:40:10.21Z] [INFO] error: unable to upgrade connection: container not found ("apb")

[2017-10-19T12:40:10.21Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 2: Failed to exec into the container
[2017-10-19T12:40:10.21Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 2: exec into apb-df743b94-6f60-4153-b211-1e0c70bac227 failed
[2017-10-19T12:40:15.714Z] [INFO] Bind credentials not available.
command terminated with exit code 2

[2017-10-19T12:40:15.714Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 3: Bind credentials not available yet
[2017-10-19T12:40:15.714Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 3: exec into apb-df743b94-6f60-4153-b211-1e0c70bac227 failed
[2017-10-19T12:40:17.501Z] [DEBUG] service_id: 6f418d63aefb3f622d58117b2cbba90c
[2017-10-19T12:40:17.501Z] [DEBUG] plan_id: 5a6570a738ee3b996901f9d8a91cae2f
[2017-10-19T12:40:17.501Z] [DEBUG] operation:  4a323bf4-aaaf-4d4f-80a7-721c5bc19f93
172.17.0.7 - - [19/Oct/2017:12:40:17 +0000] "GET /aws-service-broker/v2/service_instances/3ec651a4-52e6-4eac-a20a-11813d5e6e37/last_operation?operation=4a323bf4-aaaf-4d4f-80a7-721c5bc19f93&plan_id=5a6570a738ee3b996901f9d8a91cae2f&service_id=6f418d63aefb3f622d58117b2cbba90c HTTP/1.1" 200 29
[2017-10-19T12:40:21.12Z] [INFO] Bind credentials not available.
command terminated with exit code 2

[2017-10-19T12:40:21.12Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 4: Bind credentials not available yet
[2017-10-19T12:40:21.12Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 4: exec into apb-df743b94-6f60-4153-b211-1e0c70bac227 failed
[2017-10-19T12:40:26.899Z] [INFO] error: unable to upgrade connection: container not found ("apb")

[2017-10-19T12:40:26.899Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 5: Failed to exec into the container
[2017-10-19T12:40:26.899Z] [WARNING] [apb-df743b94-6f60-4153-b211-1e0c70bac227] Retry attempt 5: exec into apb-df743b94-6f60-4153-b211-1e0c70bac227 failed
[2017-10-19T12:40:32.078Z] [NOTICE] [apb-df743b94-6f60-4153-b211-1e0c70bac227] APB completed
[2017-10-19T12:40:32.078Z] [INFO] Destroying APB sandbox...
[2017-10-19T12:40:32.079Z] [DEBUG] Deleting rolebinding apb-df743b94-6f60-4153-b211-1e0c70bac227, namespace aws-service-broker
[2017-10-19T12:40:32.272Z] [DEBUG] Successfully deleted rolebinding apb-df743b94-6f60-4153-b211-1e0c70bac227, namespace aws-service-broker
[2017-10-19T12:40:32.272Z] [DEBUG] oc delete output:
[2017-10-19T12:40:32.272Z] [DEBUG] rolebinding "apb-df743b94-6f60-4153-b211-1e0c70bac227" deleted
jmrodri commented 7 years ago

this is identical to the one that @djzager filed. https://bugzilla.redhat.com/show_bug.cgi?id=1504927

jmrodri commented 7 years ago

Once PR posts this bug will appear in the subsequent build.