nodet / dowml

A library and command line client to use Decision Optimization on IBM Watson Machine Learning (WML). NOT SUPPORTED BY IBM.
Apache License 2.0
3 stars 0 forks source link

Timeouts should not stop dowml #27

Closed nodet closed 2 years ago

nodet commented 2 years ago

When a timeout occurs on a WML request, the exception is not caught and dowml exits. This is inconvenient, as the user has to restart it (that's not too bad) and reconfigure it (type, size, inputs, outputs).

Here's an example:

2021-09-25 14:54:29,596 {'solve_parameters': {'oaas.logAttachmentName': 'log.txt', 'oaas.logTailEnabled': 'true', 'oaas.includeInputData': 'false', 'oaas.resultsFormat': 'JSON', 'oaas.timeLimit': 20000}, 'input_data_references': [{'id': '24974-10s.cpo', 'type': 'data_asset', 'location': {'href': '/v2/assets/3214e7ba-f51a-4881-b10c-ecc120ad44e2?space_id=4645be25-e08e-4a34-a475-ff3cd0dc9635'}}], 'output_data': [{'id': '.*\\.*'}]}
Traceback (most recent call last):
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 1337, in getresponse
    response.begin()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
TimeoutError: [Errno 60] Operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/packages/six.py", line 770, in reraise
    raise value
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/urllib3/connectionpool.py", line 353, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='eu-de.ml.cloud.ibm.com', port=443): Read timed out. (read timeout=None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/bin/dowml", line 8, in <module>
    sys.exit(interactive())
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/dowml/interactive.py", line 514, in interactive
    main_loop(args.wml_cred_file, args.space, args.commands, args.input)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/dowml/interactive.py", line 417, in main_loop
    instance.cmdloop()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/cmd.py", line 217, in onecmd
    return func(arg)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/dowml/interactive.py", line 247, in do_solve
    job_id = self.lib.solve(paths)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/dowml/dowmllib.py", line 294, in solve
    job_id = self.create_job(paths, deployment_id)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/dowml/dowmllib.py", line 762, in create_job
    job_details = client.deployments.create_job(deployment_id, solve_payload)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/ibm_watson_machine_learning/deployments.py", line 1389, in create_job
    return self._score_async(deployment_id, payload,
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/ibm_watson_machine_learning/deployments.py", line 1206, in _score_async
    response_scoring = self.session.post(
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/Users/nodet/Documents/work/git/master/dowml/dowml/lib/python3.8/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='eu-de.ml.cloud.ibm.com', port=443): Read timed out. (read timeout=None)