ibm-watson-data-lab / ibmos2spark

Facilitates Data I/O between Spark and IBM Object Storage services.
10 stars 8 forks source link

Accept Bluemix Credentials #43

Open gadamc opened 6 years ago

gadamc commented 6 years ago

In the scenario where a user has multiple object stores or buckets, they'll likely want to look up the credentials for their object store in Bluemix. However, the credentials found in the "Service Credentials" tab aren't exactly what you need.

Currently, Service Credentials for "Cloud Object Storage" look like:

service_credentials_from_bluemix = {
  "apikey": "bbbdddddffffffaaaaaaeeeeeee1111122222233333455566677788",
  "endpoints": "https://cos-service.bluemix.net/endpoints",
  "iam_apikey_description": "Auto generated apikey during resource-key operation for Instance - crn:v1:bluemix:public:cloud-object-storage:global:a/xxxx:xxxx-bbbb-xxxx-aaaa-xxxxxx::",
  "iam_apikey_name": "auto-generated-apikey-aaa-bbbb-cccc-dddd-eeeeeeeeeeee",
  "iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Manager",
  "iam_serviceid_crn": "crn:v1:bluemix:public:iam-identity::a/xxxxxx::serviceid:ServiceId-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx",
  "resource_instance_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/xxx:bbb-ddd-4444-aaaa-ffffffff::"
}

In order to get the correct set of credentials to use the 'api_key' auth, one must:

iam_serviceid_crn = service_credentials_from_bluemix['iam_serviceid_crn']
service_id = iam_serviceid_crn[iam_serviceid_crn.find('ServiceId'):]

endpoints = requests.get(service_credentials_from_bluemix.get('endpoints')).json()
cos_endpoint = endpoints['service-endpoints']['cross-region']['us']['private']['us-geo']  #or other option

# Credentials

credentials_auth_method_api_key = {
    'endpoint':cos_endpoint,
    'api_key':service_credentials_from_bluemix['apikey'],
    'service_id':service_id
}

We could reduce this overhead to the user in this library by allowing for the service_credentials_from_bluemix to be passed in at instantiation. The ibmos2spark library could detect these are bluemix creds and then perform the appropriate requests and string splicing to get the needed values.

On the other hand, this might be too much overhead to maintain compatibility with they way that Bluemix presents the credentails.