"NotImplementedError: list_key_pairs not implemented for this driver" when using libcloud's `libvirt` driver

[riccardomurri]

Reported by John-Paul Robinson on the ElastiCluster mailing-list:

It appears the attempt to start a cluster relies on a list_key_pairs() method that is not supported in libcloud for the libvirt driver. Unfortunately the docs for libcloud's libvirt docs suggest this method is implemented even though the support matrix does state the method is not implemented.

Here's the output from an attempted start of a listed template for the local cluster. Note line 146 in libcloud_provider.py calls the list_key_pairs() method which triggers the not implemented section. It's not clear if this is an blocker error or if things would still work if the cluster nodes were up an running.

    (venv) jpr@laptop:~/projects/elasticluster$ elasticluster -c slurmtest/config start slurmtest
    Starting cluster `slurmtest` with:
    * 1 frontend nodes.
    * 2 compute nodes.
    (This may take a while...)
    2017-06-07 12:41:21 laptop gc3.elasticluster[20880] ERROR Could not start node `frontend001`: list_key_pairs not implemented for this driver -- <type 'exceptions.NotImplementedError'>
    Traceback (most recent call last):
      File "/home/jpr/projects/elasticluster/src/elasticluster/cluster.py", line 493, in _start_node
        node.start()
      File "/home/jpr/projects/elasticluster/src/elasticluster/cluster.py", line 1085, in start
        **self.extra)
      File "/home/jpr/projects/elasticluster/src/elasticluster/providers/libcloud_provider.py", line 72, in start_instance
        options.get('image_user_password'))
      File "/home/jpr/projects/elasticluster/src/elasticluster/providers/libcloud_provider.py", line 146, in __prepare_key_pair
        if key_name in [k.name for k in list_key_pairs()]:
      File "/home/jpr/projects/elasticluster/venv/local/lib/python2.7/site-packages/libcloud/compute/base.py", line 1202, in list_key_pairs
        'list_key_pairs not implemented for this driver')
    NotImplementedError: list_key_pairs not implemented for this driver
    ...

I tried working around this by not giving the key a name which seems like it would make __prepare_key_pair() skip the initialization steps, but that causes parsing issues for the login configuration section in the config.

[jprorama]

I retried the command today and it still persisted with the same error. Not surprising, since my libcloud install hadn't changed so the implemented functions wouldn't be different.

The errors pop up when the code tries to start the three nodes (frontend and compute) for the slurm cluster. There is one error for each node that looks like this:

2017-06-21 15:05:13 laptop gc3.elasticluster[11481] ERROR Could not start node `frontend001`: list_key_pairs not implemented for this driver -- <type 'exceptions.NotImplementedError'>
Traceback (most recent call last):
  File "/home/jpr/projects/elasticluster/src/elasticluster/cluster.py", line 493, in _start_node
    node.start()
  File "/home/jpr/projects/elasticluster/src/elasticluster/cluster.py", line 1085, in start
    **self.extra)
  File "/home/jpr/projects/elasticluster/src/elasticluster/providers/libcloud_provider.py", line 72, in start_instance
    options.get('image_user_password'))
  File "/home/jpr/projects/elasticluster/src/elasticluster/providers/libcloud_provider.py", line 146, in __prepare_key_pair
    if key_name in [k.name for k in list_key_pairs()]:
  File "/home/jpr/projects/elasticluster/venv/local/lib/python2.7/site-packages/libcloud/compute/base.py", line 1202, in list_key_pairs
    'list_key_pairs not implemented for this driver')
NotImplementedError: list_key_pairs not implemented for this driver

My libcloud version is at apache-libcloud (2.0.0).

I tried updating libcloud to see if there might be some improvements:

pip install --upgrade apache-libcloud

This didn't update libcloud but did catch some newer packages it depends on:

Requirement already up-to-date: apache-libcloud in ./venv/lib/python2.7/site-packages
Collecting requests (from apache-libcloud)
  Downloading requests-2.18.1-py2.py3-none-any.whl (88kB)
    100% |████████████████████████████████| 92kB 817kB/s 
Requirement already up-to-date: idna<2.6,>=2.5 in ./venv/lib/python2.7/site-packages (from requests->apache-libcloud)
Collecting urllib3<1.22,>=1.21.1 (from requests->apache-libcloud)
  Downloading urllib3-1.21.1-py2.py3-none-any.whl (131kB)
    100% |████████████████████████████████| 133kB 800kB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests->apache-libcloud)
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 956kB/s 
Requirement already up-to-date: certifi>=2017.4.17 in ./venv/lib/python2.7/site-packages (from requests->apache-libcloud)
Installing collected packages: urllib3, chardet, requests
  Found existing installation: requests 2.14.2
    Uninstalling requests-2.14.2:
      Successfully uninstalled requests-2.14.2
Successfully installed chardet-3.0.4 requests-2.18.1 urllib3-1.21.1

Unfortunately, it appears the update to requests-2.18.1 has now made elasiticluster unhappy:

pkg_resources.DistributionNotFound: The 'requests~=2.14.1' distribution was not found and is required by elasticluster

I haven't seen any changes to the libvirt driver in the upstream libcloud, so would expect the situation to remain the same.

If there is some guidance on how to avoid the __prepare_key_pair() error, I might be able to work around the startup issue by manually starting the nodes.

[riccardomurri]

@witlox can you please have a look?

[jprorama]

On the last error about the requests dependency, I should have tested the updated elasticluster first. I just did:

pip install --upgrade elasticluster

Now the command elasticluster -c slurmtest/config start slurmtest is back to it's original complaint about the list_key_pairs() function implementation missing.

2017-06-21 15:26:50 laptop gc3.elasticluster[13343] ERROR Could not start node `frontend001`: list_key_pairs not implemented for this driver -- <type 'exceptions.NotImplementedError'>
Traceback (most recent call last):
  File "/home/jpr/projects/elasticluster/src/elasticluster/cluster.py", line 493, in _start_node
    node.start()
  File "/home/jpr/projects/elasticluster/src/elasticluster/cluster.py", line 1085, in start
    **self.extra)
  File "/home/jpr/projects/elasticluster/src/elasticluster/providers/libcloud_provider.py", line 72, in start_instance
    options.get('image_user_password'))
  File "/home/jpr/projects/elasticluster/src/elasticluster/providers/libcloud_provider.py", line 146, in __prepare_key_pair
    if key_name in [k.name for k in list_key_pairs()]:
  File "/home/jpr/projects/elasticluster/venv/local/lib/python2.7/site-packages/libcloud/compute/base.py", line 1202, in list_key_pairs
    'list_key_pairs not implemented for this driver')
NotImplementedError: list_key_pairs not implemented for this driver

The attempted start does end with a supposed success but since the cluster nodes aren't started, it's doesn't matter much:

Configuring the cluster.
(this too may take a while...)
Your cluster is ready!

Cluster name:     slurmtest
Cluster template: slurmtest
Default ssh to node: frontend001
- frontend nodes: 1
- compute nodes: 2

To login on the frontend node, run the command:

    elasticluster ssh slurmtest

To upload or download files to the cluster, use the command:

    elasticluster sftp slurmtest

[riccardomurri]

Hello John-Paul.

Pim submitted a fix in a PR which was just merged ( https://github.com/gc3-uzh-ch/elasticluster/pull/454 for the record).

Can you please try again with the latest "master" code?

Kind regards, Riccardo ​