Support for OpenStack using tokens only (EGI FedCloud)

[alahiff]

EGI FedCloud seems to be moving away from OCCI towards native APIs with oidc or voms authentication. For example, to access IN2P3-IRES using the OpenStack CLI the following options are needed:

openstack --os-auth-type v3oidcaccesstoken \
          --os-protocol oidc \
          --os-identity-provider egi.eu \
          --os-auth-url https://sbgcloud.in2p3.fr:5000/v3 \
          --os-access-token $ACCESS_TOKEN \
          --os-project-name EGI_FCTF \
          --os-project-domain-name Default \
          <command>

Are there any plans for IM to support this? I notice there is an option to specify auth_token but there is still an assumption that a username and password are needed unless an X509 proxy is used:

ERROR creating the infrastructure: Error Creating Inf.: No correct auth data has been specified to OpenStack: username, password and tenant or proxy

due to https://github.com/grycap/im/blob/master/IM/connectors/OpenStack.py#L125

Also, there isn't any way to specify the identity provider. Looking at https://libcloud.readthedocs.io/en/latest/compute/drivers/openstack.html I'm not sure if specifying an identity provider is even possible with Libcloud...

[micafer]

The IM currently supports OIDC auth in OpenStack.

We used it for the INDIGO IAM OIDC: https://imdocs.readthedocs.io/en/latest/client.html#examples

In your case, the auth line needed should be something like:

id = ost; type = OpenStack; host = https://sbgcloud.in2p3.fr:5000; username = egi.eu; tenant = oidc; password = $ACCESS_TOKEN; auth_version = 3.x_oidc_access_token

[alahiff]

Thanks, I hadn't noticed that section at the bottom of the page. With this authentication seems to work, however I haven't managed to successfully deploy any VMs due to string indices must be integers errors. From the IM logs:

2019-03-04 14:38:21,030 - CloudConnector - ERROR - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: Error launching some of the VMs
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/IM/connectors/CloudConnector.py", line 173, in launch_with_retry
    vms = self.launch(inf, radl, requested_radl, num_vm - len(res_ok), auth_data)
  File "/usr/local/lib/python2.7/dist-packages/IM/connectors/OpenStack.py", line 572, in launch
    nets = self.get_networks(driver, radl)
  File "/usr/local/lib/python2.7/dist-packages/IM/connectors/OpenStack.py", line 495, in get_networks
    get_subnets, ost_nets = self.get_ost_network_info(driver, pool_names)
  File "/usr/local/lib/python2.7/dist-packages/IM/connectors/OpenStack.py", line 427, in get_ost_network_info
    ost_nets = driver.ex_list_networks()
  File "/usr/local/lib/python2.7/dist-packages/libcloud/compute/drivers/openstack.py", line 2824, in ex_list_networks
    self._networks_url_prefix).object
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 224, in request
    raw=raw)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 637, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/compute/drivers/openstack.py", line 1036, in __init__
    super(OpenStack_1_1_Response, self).__init__(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 152, in __init__
    message=self.parse_error(),
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 394, in parse_error
    if len(values) > 0 and values[0]['code'] == 404 and key_pair_name:
TypeError: string indices must be integers
2019-03-04 14:38:21,031 - InfrastructureManager - ERROR - Error launching some of the VMs: Attempt 1: Error: string indices must be integers
Attempt 2: Error: string indices must be integers
Attempt 3: Error: string indices must be integers

2019-03-04 14:38:21,067 - InfrastructureManager - WARNING - (Re)definition of <class 'radl.radl.system'> node
2019-03-04 14:38:21,068 - InfrastructureManager - ERROR - VMs failed when adding to Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9
2019-03-04 14:38:21,086 - InfrastructureManager - INFO - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: Finalizing the VM id: None
2019-03-04 14:38:21,086 - CloudConnector - WARNING - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: No VM ID. Ignoring
2019-03-04 14:38:21,086 - CloudConnector - WARNING - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: VM None not found.
2019-03-04 14:38:21,088 - CloudConnector - ERROR - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: Error deleting keypair.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/IM/connectors/OpenStack.py", line 934, in finalize
    if (vm.keypair and public_key is None or len(public_key) == 0 or
AttributeError: VirtualMachine instance has no attribute 'keypair'
2019-03-04 14:38:22,937 - CloudConnector - INFO - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: The SG im-1e62a5c0-3e8b-11e9-8eae-fa163e581aa9 does not exist. Do not delete it.
2019-03-04 14:38:23,562 - CloudConnector - INFO - Inf ID: 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9: The SG im-1e62a5c0-3e8b-11e9-8eae-fa163e581aa9-publica does not exist. Do not delete it.
2019-03-04 14:38:23,563 - InfrastructureManager - ERROR - Error Creating Inf ID 1e62a5c0-3e8b-11e9-8eae-fa163e581aa9
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/IM/InfrastructureManager.py", line 1372, in CreateInfrastructure
    raise Exception(error_msg)
Exception: Error launching the VMs of type node to cloud ID ost of type OpenStack. Attempt 1: Error: string indices must be integers
Attempt 2: Error: string indices must be integers
Attempt 3: Error: string indices must be integers

[micafer]

This error is caused by an error return by OpenStack that is not correctly returned due to a bug in Libcloud. It is fixed in master branch but it has not been released yet: https://github.com/apache/libcloud/blob/trunk/libcloud/common/openstack.py#L394 In next IM version I will install the master branch of Libcloud in the IM container to solve this issue.

[alahiff]

Thanks, my original error went away after updating that line in that file. Now I just need to work out why I'm getting this error:

Exception: Error launching the VMs of type node to cloud ID ost of type OpenStack. Attempt 1: Error: 404 Not Found {u'message': u'The resource could not be found.<br /><br />\n\n\n', u'code': u'404 Not Found', u'title': u'Not Found'}
Attempt 2: Error: 404 Not Found {u'message': u'The resource could not be found.<br /><br />\n\n\n', u'code': u'404 Not Found', u'title': u'Not Found'}
Attempt 3: Error: 404 Not Found {u'message': u'The resource could not be found.<br /><br />\n\n\n', u'code': u'404 Not Found', u'title': u'Not Found'}

[micafer]

It seems that the OpenStack site does not publish correctly the endpoints in the keystone. So you have to specify them in the auth data (base_url, image_url, network_url). See IM documentation: https://imdocs.readthedocs.io/en/latest/client.html#openstack-additional-fields

[alahiff]

I already had base_url defined (without this nothing worked), but didn't have network_url or image_url - using Libcloud on it's own I could create VMs on the same cloud just with base_url defined.

After adding network_url and image_url it works. Thanks!