search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.55k stars 2.77k forks source link

argument "reference_pipeline_run_id" in DataFactoryManagementClient.pipelines.pipelines cannot be used to specify run_id #30984

Closed Lee-W closed 1 year ago

Lee-W commented 1 year ago

Describe the bug The argument reference_pipeline_run_id in DataFactoryManagementClient.pipelines.pipelines cannot be used to specify run_id

To Reproduce

from azure.identity import ClientSecretCredential
from azure.mgmt.datafactory import DataFactoryManagementClient

client_id = <CLIENT_ID>
client_secret = <CLIENT_SECRET>
tenant_id = <TENANT_ID>
subscription_id = <SUBSCRIPTION_ID>
factory_name = <FACTORY_NAME>
resource_group_name = <RESOURCE_GROUP_NAME>
pipeline_name = <PIPELINE_NAME>
reference_pipeline_run_id = <REFERENCE_PIPELINE_RUN_ID>

credential = ClientSecretCredential(
    client_id=client_id,
    client_secret=client_secret,
    tenant_id=tenant_id
)
client =  DataFactoryManagementClient(
    credential=credential,
    subscription_id=subscription_id,
)

response = client.pipelines.create_run(
    resource_group_name=resource_group_name,
    factory_name=factory_name,
    pipeline_name=pipeline_name,
    is_recovery=None,
    start_activity_name=None,
    start_from_failure=None,
    reference_pipeline_run_id=reference_pipeline_run_id,
    parameters={"myParam": "value"},
)
pipeline_run = client.pipeline_runs.get(
    resource_group_name,
    factory_name,
    reference_pipeline_run_id
)

pipeline_run.run_id == response.run_id

the comparison result in the last line is False

Expected behavior The generated run_id should be the same as reference_pipeline_run_id

Screenshots

Additional context

kashifkhan commented 1 year ago

Thank you for the feedback @Lee-W. We will investigate and get back to you asap.

msyyc commented 1 year ago

@Lee-W About your code:

response = client.pipelines.pipelines(
    resource_group_name=resource_group_name,
    factory_name=factory_name,
    pipeline_name=pipeline_name,
    is_recovery=None,
    start_activity_name=None,
    start_from_failure=None,
    reference_pipeline_run_id=reference_pipeline_run_id,
    parameters={"myParam": "value"},
)

I don't find API client.pipelines.pipelines. Do you mean client.pipelines.create_run?

Lee-W commented 1 year ago

ah, yes, what I mean is create_run

msyyc commented 1 year ago

I am not domain expert of data factory. According to the examples, it seems that pipeline_run.run_id shall equal reference_pipeline_run_id instead of response.run_id returned from pipelines.create_run() image

like:

pipeline_run = client.pipeline_runs.get(
    resource_group_name,
    factory_name,
    reference_pipeline_run_id
)

assert pipeline_run.run_id == reference_pipeline_run_id
Lee-W commented 1 year ago

yep, but that doesn't seem to work on our side. did you manage to run the example? or is there an updated version of this SDK?

msyyc commented 1 year ago

(1) Here is latest version: https://pypi.org/project/azure-mgmt-datafactory/3.1.0/ and you can have a try (2) If it still doesn't work, please follow https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/debug_guide.md to get detailed log(nit: remember to conceal sensitive info of log if provided in public)

github-actions[bot] commented 1 year ago

Hi @Lee-W. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

Lee-W commented 1 year ago

Hi @msyyc , sorry for late reply. I've tested it and it still does not work.

The following is the log you're asking

2023-07-20 17:58:07,586 - azure.core.pipeline.policies.http_logging_policy - INFO - Request URL: 'https://login.microsoftonline.com:443/REDACTED/v2.0/.well-known/openid-configuration'
Request method: 'GET'
Request headers:
    'User-Agent': 'azsdk-python-identity/1.13.0 Python/3.10.6 (macOS-13.4.1-arm64-arm-64bit)'
No body was attached to the request
2023-07-20 17:58:07,617 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): login.microsoftonline.com:443
2023-07-20 17:58:07,904 - urllib3.connectionpool - DEBUG - https://login.microsoftonline.com:443 "GET /REDACTED/v2.0/.well-known/openid-configuration HTTP/1.1" 200 1753
2023-07-20 17:58:07,905 - azure.core.pipeline.policies.http_logging_policy - INFO - Response status: 200
Response headers:
    'Cache-Control': 'max-age=86400, private'
    'Content-Type': 'application/json; charset=utf-8'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'Access-Control-Allow-Origin': 'REDACTED'
    'Access-Control-Allow-Methods': 'REDACTED'
    'P3P': 'REDACTED'
    'x-ms-request-id': 'REDACTED'
    'x-ms-ests-server': 'REDACTED'
    'X-XSS-Protection': 'REDACTED'
    'Set-Cookie': 'REDACTED'
    'Date': 'Thu, 20 Jul 2023 09:58:06 GMT'
    'Content-Length': '1753'
2023-07-20 17:58:07,906 - msal.authority - DEBUG - openid_config = {'token_endpoint': 'https://login.microsoftonline.com/REDACTED/oauth2/v2.0/token', 'token_endpoint_auth_methods_supported': ['client_secret_post', 'private_key_jwt', 'client_secret_basic'], 'jwks_uri': 'https://login.microsoftonline.com/REDACTED/discovery/v2.0/keys', 'response_modes_supported': ['query', 'fragment', 'form_post'], 'subject_types_supported': ['pairwise'], 'id_token_signing_alg_values_supported': ['RS256'], 'response_types_supported': ['code', 'id_token', 'code id_token', 'id_token token'], 'scopes_supported': ['openid', 'profile', 'email', 'offline_access'], 'issuer': 'https://login.microsoftonline.com/REDACTED/v2.0', 'request_uri_parameter_supported': False, 'userinfo_endpoint': 'https://graph.microsoft.com/oidc/userinfo', 'authorization_endpoint': 'https://login.microsoftonline.com/REDACTED/oauth2/v2.0/authorize', 'device_authorization_endpoint': 'https://login.microsoftonline.com/REDACTED/oauth2/v2.0/devicecode', 'http_logout_supported': True, 'frontchannel_logout_supported': True, 'end_session_endpoint': 'https://login.microsoftonline.com/REDACTED/oauth2/v2.0/logout', 'claims_supported': ['sub', 'iss', 'cloud_instance_name', 'cloud_instance_host_name', 'cloud_graph_host_name', 'msgraph_host', 'aud', 'exp', 'iat', 'auth_time', 'acr', 'nonce', 'preferred_username', 'name', 'tid', 'ver', 'at_hash', 'c_hash', 'email'], 'kerberos_endpoint': 'https://login.microsoftonline.com/REDACTED/kerberos', 'tenant_region_scope': 'NA', 'cloud_instance_name': 'microsoftonline.com', 'cloud_graph_host_name': 'graph.windows.net', 'msgraph_host': 'graph.microsoft.com', 'rbac_url': 'https://pas.windows.net'}
2023-07-20 17:58:07,906 - msal.application - DEBUG - Broker enabled? False
2023-07-20 17:58:07,906 - msal.application - DEBUG - Region to be used: None
2023-07-20 17:58:07,906 - msal.application - DEBUG - Found 0 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': None, 'family_id': '1'}
2023-07-20 17:58:07,906 - msal.application - DEBUG - Found 0 RTs matching {'environment': 'login.microsoftonline.com', 'home_account_id': None, 'client_id': 'REDACTED'}
2023-07-20 17:58:07,906 - azure.core.pipeline.policies.http_logging_policy - INFO - Request URL: 'https://login.microsoftonline.com/common/discovery/instance?api-version=REDACTED&authorization_endpoint=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'User-Agent': 'azsdk-python-identity/1.13.0 Python/3.10.6 (macOS-13.4.1-arm64-arm-64bit)'
No body was attached to the request
2023-07-20 17:58:08,014 - urllib3.connectionpool - DEBUG - https://login.microsoftonline.com:443 "GET /common/discovery/instance?api-version=1.1&authorization_endpoint=https://login.microsoftonline.com/common/oauth2/authorize HTTP/1.1" 200 945
2023-07-20 17:58:08,015 - azure.core.pipeline.policies.http_logging_policy - INFO - Response status: 200
Response headers:
    'Cache-Control': 'max-age=86400, private'
    'Content-Type': 'application/json; charset=utf-8'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'Access-Control-Allow-Origin': 'REDACTED'
    'Access-Control-Allow-Methods': 'REDACTED'
    'P3P': 'REDACTED'
    'x-ms-request-id': 'REDACTED'
    'x-ms-ests-server': 'REDACTED'
    'X-XSS-Protection': 'REDACTED'
    'Set-Cookie': 'REDACTED'
    'Date': 'Thu, 20 Jul 2023 09:58:07 GMT'
    'Content-Length': '945'
2023-07-20 17:58:08,015 - msal.telemetry - DEBUG - Generate or reuse correlation_id: ae978072-c0bb-43e0-8b12-a9024d7a6ff3
2023-07-20 17:58:08,015 - azure.core.pipeline.policies.http_logging_policy - INFO - Request URL: 'https://login.microsoftonline.com/REDACTED/oauth2/v2.0/token'
Request method: 'POST'
Request headers:
    'Accept': 'application/json'
    'x-client-sku': 'REDACTED'
    'x-client-ver': 'REDACTED'
    'x-client-os': 'REDACTED'
    'x-client-cpu': 'REDACTED'
    'x-ms-lib-capability': 'REDACTED'
    'client-request-id': 'REDACTED'
    'x-client-current-telemetry': 'REDACTED'
    'x-client-last-telemetry': 'REDACTED'
    'User-Agent': 'azsdk-python-identity/1.13.0 Python/3.10.6 (macOS-13.4.1-arm64-arm-64bit)'
A body is sent with the request
2023-07-20 17:58:08,205 - urllib3.connectionpool - DEBUG - https://login.microsoftonline.com:443 "POST /REDACTED/oauth2/v2.0/token HTTP/1.1" 200 1409
2023-07-20 17:58:08,205 - azure.core.pipeline.policies.http_logging_policy - INFO - Response status: 200
Response headers:
    'Cache-Control': 'no-store, no-cache'
    'Pragma': 'no-cache'
    'Content-Type': 'application/json; charset=utf-8'
    'Expires': '-1'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'P3P': 'REDACTED'
    'client-request-id': 'REDACTED'
    'x-ms-request-id': 'REDACTED'
    'x-ms-ests-server': 'REDACTED'
    'x-ms-clitelem': 'REDACTED'
    'X-XSS-Protection': 'REDACTED'
    'Set-Cookie': 'REDACTED'
    'Date': 'Thu, 20 Jul 2023 09:58:07 GMT'
    'Content-Length': '1409'
2023-07-20 17:58:08,206 - msal.token_cache - DEBUG - event={
    "client_id": "REDACTED",
    "data": {
        "claims": "{\"access_token\": {\"xms_cc\": {\"values\": [\"CP1\"]}}}",
        "scope": [
            "https://management.azure.com/.default"
        ]
    },
    "environment": "login.microsoftonline.com",
    "grant_type": "client_credentials",
    "params": null,
    "response": {
        "access_token": "********",
        "expires_in": 3599,
        "ext_expires_in": 3599,
        "token_type": "Bearer"
    },
    "scope": [
        "https://management.azure.com/.default"
    ],
    "token_endpoint": "https://login.microsoftonline.com/REDACTED/oauth2/v2.0/token"
}
2023-07-20 17:58:08,206 - azure.identity._internal.get_token_mixin - INFO - ClientSecretCredential.get_token succeeded
2023-07-20 17:58:08,206 - azure.core.pipeline.policies.http_logging_policy - INFO - Request URL: 'https://management.azure.com/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.DataFactory/factories/REDACTED/pipelines/REDACTED/createRun?api-version=REDACTED&referencePipelineRunId=REDACTED'
Request method: 'POST'
Request headers:
    'Content-Type': 'application/json'
    'Content-Length': '20'
    'Accept': 'application/json'
    'x-ms-client-request-id': 'REDACTED'
    'User-Agent': 'azsdk-python-mgmt-datafactory/3.1.0 Python/3.10.6 (macOS-13.4.1-arm64-arm-64bit)'
    'Authorization': 'REDACTED'
A body is sent with the request
2023-07-20 17:58:08,276 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): management.azure.com:443
2023-07-20 17:58:09,894 - urllib3.connectionpool - DEBUG - https://management.azure.com:443 "POST /subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.DataFactory/factories/REDACTED/pipelines/REDACTED/createRun?api-version=2018-06-01&referencePipelineRunId=REDACTED HTTP/1.1" 200 None
2023-07-20 17:58:09,899 - azure.core.pipeline.policies.http_logging_policy - INFO - Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'Server': 'Kestrel'
    'x-ms-ratelimit-remaining-subscription-writes': '1199'
    'x-ms-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'Date': 'Thu, 20 Jul 2023 09:58:09 GMT'
2023-07-20 17:58:09,899 - azure.core.pipeline.policies.http_logging_policy - INFO - Request URL: 'https://management.azure.com/subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.DataFactory/factories/REDACTED/pipelineruns/REDACTED?api-version=REDACTED'
Request method: 'GET'
Request headers:
    'Accept': 'application/json'
    'x-ms-client-request-id': 'REDACTED'
    'User-Agent': 'azsdk-python-mgmt-datafactory/3.1.0 Python/3.10.6 (macOS-13.4.1-arm64-arm-64bit)'
    'Authorization': 'REDACTED'
No body was attached to the request
2023-07-20 17:58:10,417 - urllib3.connectionpool - DEBUG - https://management.azure.com:443 "GET /subscriptions/REDACTED/resourceGroups/REDACTED/providers/Microsoft.DataFactory/factories/REDACTED/pipelineruns/REDACTED?api-version=2018-06-01 HTTP/1.1" 200 None
2023-07-20 17:58:10,421 - azure.core.pipeline.policies.http_logging_policy - INFO - Response status: 200
Response headers:
    'Cache-Control': 'no-cache'
    'Pragma': 'no-cache'
    'Transfer-Encoding': 'chunked'
    'Content-Type': 'application/json; charset=utf-8'
    'Content-Encoding': 'REDACTED'
    'Expires': '-1'
    'Vary': 'REDACTED'
    'x-ms-correlation-request-id': 'REDACTED'
    'Server': 'Kestrel'
    'x-ms-ratelimit-remaining-subscription-reads': '11999'
    'x-ms-request-id': 'REDACTED'
    'x-ms-routing-request-id': 'REDACTED'
    'Strict-Transport-Security': 'REDACTED'
    'X-Content-Type-Options': 'REDACTED'
    'Date': 'Thu, 20 Jul 2023 09:58:09 GMT'
ChenxiJiang333 commented 1 year ago

Hi, @Lee-W. According to the definition, create_run function will still create a new pipeline run when the reference_pipeline_run_id specified, so the response.run_id corresponds to the run id of new created pipeline run, which certainly cannot match the reference_pipeline_run_id previously passed in. image If you want to get the new pipeline run, please reassign the value of reference_pipeline_run_id as it shows below.

response = client.pipelines.create_run(
    resource_group_name=resource_group_name,
    factory_name=factory_name,
    pipeline_name=pipeline_name,
    is_recovery=None,
    start_activity_name=None,
    start_from_failure=None,
    reference_pipeline_run_id=reference_pipeline_run_id,
    parameters={"myParam": "value"},
)

reference_pipeline_run_id = response.run_id

pipeline_run = client.pipeline_runs.get(
    resource_group_name,
    factory_name,
    reference_pipeline_run_id
)
Lee-W commented 1 year ago

Yes, but I thought it would create another new run with the provided run_id? Is there anyway for use to specific the run_id when creating a pipeline run?

ChenxiJiang333 commented 1 year ago

I ‘m afraid there's not. The most you can do is to set is_recovery as true with reference_pipeline_run_id specified when trying to create pipeline runs. It would mark the new runs as reruns of one original pipeline run, so that they could share one run group id (run id of the original run). Then you can get them once for all through this method:

response = client.pipeline_runs.query_by_factory(
        resource_group_name=resource_group_name,
        factory_name=factory_name,
        filter_parameters={"filters": [{"operand": "PipelineName", "operator": "Equals", "values": [<PIPELINE_NAME>]}, {"operand": "RunGroupId", "operator": "Equals", "values": [<ORIGINAL_PIPELINE_RUN_ID>]}],
            "lastUpdatedAfter": "2023-07-20T00:00:00.0000000Z",
            "lastUpdatedBefore": "2023-07-22T00:00:00.0000000Z",},
    )
Lee-W commented 1 year ago

Got it. Thanks!