search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.38k stars 2.72k forks source link

Parameter to support append Libraries by databricks_notebook_activity #4144

Closed sgupta227 closed 5 years ago

sgupta227 commented 5 years ago

Hi,

It seems like we don't have parameter option available while creation of DatabricksNotebookActivity using databricks_notebook_activity.py scipt. Is it something can be achieved by some other way or do we need to fix this issue ?

To append the libraries we need another parameter let's say append_libraries which can be something like this:

'append_libraries ': {'key': 'typeProperties.baseParameters', 'type': '{object}'} may be need to consider to update in _attribute_map as well.

_attribute_map = {
    'additional_properties': {'key': '', 'type': '{object}'},
    'name': {'key': 'name', 'type': 'str'},
    'description': {'key': 'description', 'type': 'str'},
    'depends_on': {'key': 'dependsOn', 'type': '[ActivityDependency]'},
    'type': {'key': 'type', 'type': 'str'},
    'linked_service_name': {'key': 'linkedServiceName', 'type': 'LinkedServiceReference'},
    'policy': {'key': 'policy', 'type': 'ActivityPolicy'},
    'notebook_path': {'key': 'typeProperties.notebookPath', 'type': 'object'},
    'base_parameters': {'key': 'typeProperties.baseParameters', 'type': '{object}'},
hvermis commented 5 years ago

@sgupta227 DatabricksNotebookActivity has property called "libraries". Is this what you are looking for? https://github.com/Azure/azure-sdk-for-python/blob/master/azure-mgmt-datafactory/azure/mgmt/datafactory/models/databricks_notebook_activity.py#L46

sgupta227 commented 5 years ago

@hvermis ya thanks for your comment. I see these features are available with the new release of 0.7.0 version of data-factory-mgmt. But i saw a issue while using it. If i am trying to create a databricks linked service using AzureDatabricksLinkedService it needs the access token to passed as SecureString but when i provide the token as SecureString i am getting this error: TypeError: init() takes 1 positional argument but 2 were given. Which is working fine version 0.6.0. And if i provide access_token without SecureString it works fine. I am just wondering if i am missing here something ? looking forward to hearback from you.

hvermis commented 5 years ago

Can you paste your script here?

sgupta227 commented 5 years ago

credentials = ServicePrincipalCredentials(client_id=service_principal_id, secret=service_principal_key, tenant=tenant_id) adfv2_client_obj = DataFactoryManagementClient(credentials, subscription_id) azure_databricks = AzureDatabricksLinkedService(domain=domain, access_token=SecureString(adb_access_token)) databricks_ls = adfv2_client_obj.linked_services.create_or_update(resource_group_name=rg_name, factory_name=df_name, linked_service_name='AzureDatabricks_Python1',properties=azure_databricks)

But the same code is working with azure-datafactory-mgmt 0.6.0 version

hvermis commented 5 years ago

@lmazuel can you help? The difference between previous version of AzureDatabricksLinkedService and the current one is the latest has **kwargs instead named arguments. Same for SecureString.

@sgupta227 Are you sure the AzureDatabricksLinkedService line is the one giving you the error? Can you paste full error here?

sgupta227 commented 5 years ago

@hvermis yes i am sure about the error. Since i tested quite number of times. As i know create_or_update method will take parameter if we provide the securestring for token else we will get an error while creation.

image

sgupta227 commented 5 years ago

@hvermis, did you get any updates on the issue running adf v2 pipelines using azure 0.7.0 version ?

hvermis commented 5 years ago

@sgupta227 Did you get your error resolved?

hvermis commented 5 years ago

Closing as it diverged from original topic which was resolved.