fractal-analytics-platform / fractal-server

Fractal backend
https://fractal-analytics-platform.github.io/fractal-server/
BSD 3-Clause "New" or "Revised" License
11 stars 3 forks source link

Handle errors in SSH handshake #1682

Open tcompa opened 2 months ago

tcompa commented 2 months ago

An error like the following one should be captured and re-raised with some more reasonable explanation.

TRACEBACK:
Traceback (most recent call last):
  File "/xxx/fractal-server/venv/lib/python3.10/site-packages/fractal_server/app/runner/v2/__init__.py", line 318, in submit_workflow
    new_dataset_attributes = await process_workflow(
  File "/xxx/fractal-server/venv/lib/python3.10/site-packages/fractal_server/app/runner/v2/_slurm_ssh/__init__.py", line 114, in process_workflow
    new_dataset_attributes = await async_wrap(_process_workflow)(
  File "/xxx/fractal-server/venv/lib/python3.10/site-packages/fractal_server/app/runner/async_wrap.py", line 25, in async_wrapper
    return await loop.run_in_executor(executor, pfunc)
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/xxx/fractal-server/venv/lib/python3.10/site-packages/fractal_server/app/runner/v2/_slurm_ssh/__init__.py", line 63, in _process_workflow
    with FractalSlurmSSHExecutor(
  File "/xxx/fractal-server/venv/lib/python3.10/site-packages/fractal_server/app/runner/executors/slurm/ssh/executor.py", line 171, in __init__
    self.handshake()
  File "/xxx/fractal-server/venv/lib/python3.10/site-packages/fractal_server/app/runner/executors/slurm/ssh/executor.py", line 1468, in handshake
    remote_versions = json.loads(stdout.strip("\n"))
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
ychiucco commented 2 months ago

This is reproducible if the remote Python does not have fractal-server available