cancel endpoint not working

[JaeAeich]

Describe the bug The /runs/:id/cancel is not working as expected. It throws a CORS error.

Response:

{
  "message": "Could not reach remote WES service.",
  "code": "500"
}

Traceback:

[2024-02-04 18:07:00,829: ERROR   ] {'message': 'Could not reach remote WES service.', 'code': '500'} [foca.errors.exceptions]
[2024-02-04 18:07:00,835: ERROR   ] Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request\n    self._validate_conn(conn)\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn\n    conn.connect()\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 642, in connect\n    sock_and_verified = _ssl_wrap_socket_and_match_hostname(\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 783, in _ssl_wrap_socket_and_match_hostname\n    ssl_sock = ssl_wrap_socket(\n  File "/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 469, in ssl_wrap_socket\n    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)\n  File "/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 513, in _ssl_wrap_socket_impl\n    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)\n  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket\n    return self.sslsocket_class._create(\n  File "/usr/local/lib/python3.10/ssl.py", line 1071, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.10/ssl.py", line 1342, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)\n\nDuring handling of the above exception, another exception occurred:\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen\n    response = self._make_request(\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request\n    raise new_e\nurllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)\n\nThe above exception was the direct cause of the following exception:\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send\n    resp = conn.urlopen(\n  File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen\n    retries = retries.increment(\n  File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='csc-wes-noauth.rahtiapp.fi', port=443): Max retries exceeded with url: /ga4gh/wes/v1/runs/VVXHO8/cancel (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))\n\nDuring handling of the above exception, another exception occurred:\nTraceback (most recent call last):\n  File "/app/pro_wes/ga4gh/wes/client_wes.py", line 276, in cancel_run\n    response_unvalidated = self.session.post(url, **kwargs).json()\n  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 637, in post\n    return self.request("POST", url, data=data, json=json, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n  File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send\n    r = adapter.send(request, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send\n    raise SSLError(e, request=request)\nrequests.exceptions.SSLError: HTTPSConnectionPool(host='csc-wes-noauth.rahtiapp.fi', port=443): Max retries exceeded with url: /ga4gh/wes/v1/runs/VVXHO8/cancel (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))\n\nThe above exception was the direct cause of the following exception:\nTraceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1823, in full_dispatch_request\n    rv = self.dispatch_request()\n  File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1799, in dispatch_request\n    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)\n  File "/usr/local/lib/python3.10/site-packages/connexion/decorators/decorator.py", line 68, in wrapper\n    response = function(request)\n  File "/usr/local/lib/python3.10/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper\n    response = function(request)\n  File "/usr/local/lib/python3.10/site-packages/connexion/decorators/validation.py", line 399, in wrapper\n    return function(request)\n  File "/usr/local/lib/python3.10/site-packages/connexion/decorators/parameter.py", line 120, in wrapper\n    return function(**kwargs)\n  File "/usr/local/lib/python3.10/site-packages/foca/utils/logging.py", line 61, in _wrapper\n    response = fn(*args, **kwargs)\n  File "/app/pro_wes/ga4gh/wes/controllers.py", line 164, in CancelRun\n    response = workflow_run.cancel_run(run_id=run_id, **kwargs)\n  File "/app/pro_wes/ga4gh/wes/workflow_runs.py", line 383, in cancel_run\n    wes_client.cancel_run(run_id=document["wes_endpoint"]["run_id"])\n  File "/app/pro_wes/ga4gh/wes/client_wes.py", line 278, in cancel_run\n    raise EngineUnavailable("external workflow engine unavailable") from exc\npro_wes.exceptions.EngineUnavailable: 500 Internal Server Error: external workflow engine unavailable [foca.errors.exceptions]

To Reproduce

1. Try canceling any run from Swagger.
2. Check if the run has been canceled or not, from the /runs endpoint.

[JaeAeich]

Hey @uniqueg, I have checked this out and can't seem to find where the issue might be coming from; I have checked commits as far as last year but still have the same result. Could there be an issue with the wes_client, the WES API itself? Any idea where the issue might be arising from?

[uniqueg]

I don't know either. First of all: Did you verify if it works if you try to cancel runs via Swagger (at least that should not give a CORS error). What other steps did you do to debug?

[JaeAeich]

Yeah, no CORS errors. To debug I tried logging everything and checked if it was raising any errors that were not caught.

But today for some reason I encountered another issue, /runs endpoint is working but /cancel/:id is throwing a 500 error, ie could not reach remote wes service.

[uniqueg]

Is this on a local deployment via Docker Compose? If not, can you try it locally and check the error trace? I really think that cancellation in proWES was never properly implemented. It might be that it's just a line or two that need changes, but it could also be something bigger.

Do you actually need this function? I think as long as you don't get a 400 reply, your client should be sending the right request, so the problem is not really on your side. To further check if your client behaves as expected, you could check whether cURL and the Web Component give different results (setting aside the CORS issue).

[JaeAeich]

Yeah, this is not needed for the Web Components, I was checking the issue independent of that, hope thats ok! :sweat_smile:.