Webhook-Shims issue with vROps 8.5 and vRO 8.5

[Uday-Mantri]

While working for auto-remediation use-case using vROps 8.5, vRO 8.5 and webhook shims. I am facing an issue in which the workflow is not getting triggered as soon as the alert is triggered in vROps.

The setup used to work perfectly with earlier versions of vROps and vRO.

We have been using vRO Management Pack as it does support to create workflow actions for vSphere objects but we cannot bind actions to non-vSphere objects like VM services, NSX Edges, etc.

Hence, had to perform this use-case with webhook shims.

Getting and error exception as below followed by refused connection

2021-09-24 13:48:55,495 ERROR Exception on /endpoint/vro/1172e9c1-2e1e-4561-b509-9f9a827e420d [POST] Traceback (most recent call last): File "/root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/root/webhook-shims/loginsightwebhookdemo/vrealizeorchestrator.py", line 108, in vro "value": a['NumHits'] KeyError: 'NumHits'

[johnddias]

Recommend you not use webhook shims with 8.4 or higher as we have now built in webhook notifications in vROps, thus shims are no longer required.

[Uday-Mantri]

Hello John,

Appreciate your help. I understand the recommendation but how can the webhook notification parse the alert payload, the way webhook shim was doing. What we want is to enable vro workflow execution for non vsphere objects when a alert gets triggered on that object. This cannot be done by vRO management pack as it only support vsphere objects and I am not sure if this can be achieved by webhook outbound setting. Request your guidance in achieving the use-case.

[johnddias]

With webhooks you don't have to parse the alert payload. The payload can be customized as needed for your endpoint. For example, I created a sample webhook you can import for PagerDuty - check it out here. https://code.vmware.com/samples/7607/vrops-webhook-payload-for-pagerduty-alert-events-example?h=pagerduty

[Uday-Mantri]

Okay, what should be the payload template the for executing a vRO workflow. Have you tried this to run a vRO workflow. I am also not sure what should be url that I must be using, will it be the vRO workflow execution api.

[johnddias]

Ah, yes I'm sorry I totally blanked on what you were trying to do. So, the problem with using webhook notifications would be that we don't support a two-step authentication method so you won't be able to authenticate to your vRO.

I took a look at the error above and it looks like the alert ID is missing from the request - thus the script treats it like a Log Insight alert, not a vROps alert. As far as I know the REST Notification plugin still works the same so it should be appending the alert ID to the end of the request URL. I just tested in my lab and it is sending the alert ID as expected.

[Uday-Mantri]

The Rest Notification Plugin shows deprecated in vROps 8.5 and when tested, it still does not work. May be due to the deprecated nature of the functionality.

I am using a standalone vRO of version 8.5 with vROps 8.5. I have worked on webhook shims and vRO + vROps when they were on version 7.x.

Below is what I am getting at webhook shim end, using port 443 because of the standalone vRO. When using port 8281 it is unable to establish the connection because of the fact that embedded vRO works on the port 8281.

2021-09-25 19:17:54,734 INFO URL=https://xxx.xxx.xxx:443/vco/api/workflows/1172e9c1-2e1e-4561-b509-9f9a827e420d/executions 2021-09-25 19:17:54,734 INFO Auth=None 2021-09-25 19:17:54,734 INFO Headers={'Content-type': 'application/json'} 2021-09-25 19:17:54,734 INFO Body={"parameters": [{"name": "alertId", "scope": "local", "type": "string", "value": {"string": {"value": "452fddfc-c717-47e6-9c2f-6b7b9574d8bc"}}}]} 2021-09-25 19:17:54,734 INFO Check=False 2021-09-25 19:17:54,736 DEBUG Starting new HTTPS connection (1): xxx.xxx.xxx.xxx:443 /root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'xxx.xxx.xxx.xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings InsecureRequestWarning, 2021-09-25 19:17:54,747 DEBUG https://xxx.xxx.xxx.xxx:443 "POST /vco/api/workflows/1172e9c1-2e1e-4561-b509-9f9a827e420d/executions HTTP/1.1" 404 19 2021-09-25 19:17:54,748 INFO xxx.xxx.xxx.xxx- - [25/Sep/2021 19:17:54] "POST /endpoint/vro/1172e9c1-2e1e-4561-b509-9f9a827e420d/452fddfc-c717-47e6-9c2f-6b7b9574d8bc HTTP/1.1" 404 -

Please advise, is there anything we can use the webhook payload or make it work.

[johnddias]

404 indicates one of two things - either the workflow with that ID cannot be found, or the user does not have permissions to execute the workflow. I would first verify that the workflow ID is correctly configured in the REST Notification Plugin instance.

[Uday-Mantri]

Hello John,

Understood the above point, re-verified the API URL, workflow ID and the permissions.

URL in REST API Plugin of vROps is http://xxx.xxx.xxx.xxx:5001/endpoint/vro/1172e9c1-2e1e-4561-b509-9f9a827e420d URL looks fine, verified the workflow ID which is also correct.

To test the permissions, tried hitting a POST request using POSTMAN and was able to execute the workflow, Code 202 with below response. Basic Authentication over SSO is also enabled.

{ "id": "fa0367a7-3151-4b0c-b17a-647066c5deaf", "state": "running", "start-date": "2021-09-27T05:11:25.003+00:00", "started-by": "xxxxxxxxxxxx", "running-instance-id": "xxxxxxxxxxxx.xxxxxxxxxxxx.xxxxxxxxxxxx-vco-app-c6bcd6f4c-55n4d", "name": "Test Powershell", "current-item-for-display": "item-undefined", "input-parameters": [ { "value": { "string": { "value": "" } }, "type": "string", "name": "alertId", "scope": "local" } ], "output-parameters": [], "workflow-attributes": [ { "value": { "string": { "value": "xxxxxxxxxxxx" } }, "type": "string", "name": "username", "scope": "local" }, { "value": { "secure-string": { "value": "xxxxxxxxxxxx", "isPlainText": false } }, "type": "SecureString", "name": "password", "scope": "local" }, { "value": { "string": { "value": "xxxxxxxxxxxx" } }, "type": "string", "name": "virtualmachine", "scope": "local" }, { "value": { "sdk-object": { "type": "PowerShell:PowerShellHost", "href": "https://xxxxxxxxxxxx.xxxxxxxxxxxx.xxxxxxxxxxxx:443/vco/api/catalog/PowerShell/PowerShellHost/14c88c2e-ae81-4dab-a395-70909cdd749e/", "id": "14c88c2e-ae81-4dab-a395-70909cdd749e" } }, "type": "PowerShell:PowerShellHost", "name": "host", "scope": "local" } ] }

But on Webhook Shims, whenever the call is made, it ends up in 404

2021-09-27 05:48:51,887 INFO URL=https://xxx.xxx.xxx.xxx:443/vco/api/workflows/1172e9c1-2e1e-4561-b509-9f9a827e420d/executions 2021-09-27 05:48:51,888 INFO Auth=None 2021-09-27 05:48:51,888 INFO Headers={'Content-type': 'application/json'} 2021-09-27 05:48:51,888 INFO Body={"parameters": [{"name": "alertId", "scope": "local", "type": "string", "value": {"string": {"value": "c16fe695-7732-41bc-bc68-3c3a921d54f4"}}}]} 2021-09-27 05:48:51,888 INFO Check=False 2021-09-27 05:48:51,889 DEBUG Starting new HTTPS connection (1): xxx.xxx.xxx.xxx:443 /root/webhook-shims/venv-webhookshims/lib/python2.7/site-packages/urllib3/connectionpool.py:1020: InsecureRequestWarning: Unverified HTTPS request is being made to host 'xxx.xxx.xxx.xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings InsecureRequestWarning, 2021-09-27 05:48:51,902 DEBUG https://xxx.xxx.xxx.xxx:443 "POST /vco/api/workflows/1172e9c1-2e1e-4561-b509-9f9a827e420d/executions HTTP/1.1" 404 19 2021-09-27 05:48:51,903 INFO xxx.xxx.xxx.xxx - - [27/Sep/2021 05:48:51] "POST /endpoint/vro/1172e9c1-2e1e-4561-b509-9f9a827e420d/c16fe695-7732-41bc-bc68-3c3a921d54f4 HTTP/1.1" 404 -

It seems that the basic authentication parameters are not being passed over the API. Did this ever work with vROps and vRO 8.x setup, Requesting your help.

[Uday-Mantri]

It finally worked. The problem was that the .netrc authentication was not working. had to disable it and then provided the credentials in vrealizeorchestrator.py

USENETRC = False VROUSER = 'xxxxusernamexxxx' VROPASS = 'xxxxpasswordxxxx'

Also, the vRO host is listening at port 443 instead of 8281as it is standalone deployment.

Thank You so much John for the help.