Keys are not automatically loaded and mismanaged between providers

[blokhin]

A normal connection succeeds:

root@aiida9:~# ssh X.X.X.X
Linux labs 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64

...
root@labs:~# logout
Connection to X.X.X.X closed.

but scheduler connection fails:

root@aiida9:~# yasetnode X.X.X.X~4
Traceback (most recent call last):
  File "/usr/local/bin/yasetnode", line 8, in <module>
    sys.exit(manage_node())
  File "/usr/local/lib/python3.9/dist-packages/yascheduler/utils.py", line 430, in manage_node
    asyncio.run(_manage_node())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.9/dist-packages/yascheduler/utils.py", line 411, in _manage_node
    machine = await RemoteMachine.create(
  File "/usr/local/lib/python3.9/dist-packages/backoff/_async.py", line 151, in retry
    ret = await target(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/yascheduler/remote_machine/remote_machine.py", line 192, in create
    conn = await asyncssh.connection.connect(
  File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 7834, in connect
    return await asyncio.wait_for(
  File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 447, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

[blokhin]

Another case, very frustrating:

• allocate and connect one machine manually
• launch a long-running task there
• setup an API key e.g. for Hetzner
• launch another long-running task, so that the new machine is allocated at Hetzner
• the keys are loaded erroneously and the connection to the first machine fails:

  root@aiida9:~# yastatus -v
  ..................................................ID68 CRYSTAL by AiiDA at root@X.X.X.X::data/tasks/20221128_235115_68
  Traceback (most recent call last):
  File "/usr/local/bin/yastatus", line 8, in <module>
  sys.exit(check_status())
  File "/usr/local/lib/python3.9/dist-packages/yascheduler/utils.py", line 237, in check_status
  asyncio.run(_check_status())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
  return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
  return future.result()
  File "/usr/local/lib/python3.9/dist-packages/yascheduler/utils.py", line 148, in _check_status
  machine = await RemoteMachine.create(
  File "/usr/local/lib/python3.9/dist-packages/backoff/_async.py", line 151, in retry
  ret = await target(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/yascheduler/remote_machine/remote_machine.py", line 192, in create
  conn = await asyncssh.connection.connect(
  File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 7834, in connect
  return await asyncio.wait_for(
  File "/usr/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
  return await fut
  File "/usr/local/lib/python3.9/dist-packages/asyncssh/connection.py", line 447, in _connect
  await options.waiter
  asyncssh.misc.PermissionDenied: Permission denied

[blokhin]

@knopki please could you have a look?

[blokhin]

Solution (FIXME?): delete keys from data_dir and retry.

[blokhin]

also #127