Azure-Player / azure.datafactory.devops

Azure DevOps Deployment Tasks for Azure Data Factory objects
MIT License
39 stars 21 forks source link

ADF Datasets Not getting deployed #140

Closed dayalus closed 1 year ago

dayalus commented 1 year ago

Hi,

Is anyone having issues deploying datasets? Linked services, dataflows, etc. work fine. However, of late, all our datasets are not getting deployed.

Here's a snippet from the logs:

2023-06-01T06:19:17.9446572Z Azure Data Factory exists. 2023-06-01T06:19:17.9485210Z =================================================================================== 2023-06-01T06:19:17.9495870Z STEP: Reading Azure Data Factory from JSON files... 2023-06-01T06:19:18.2580680Z IntegrationRuntimes: 3 object(s) loaded. 2023-06-01T06:19:19.8060934Z LinkedServices: 54 object(s) loaded. 2023-06-01T06:21:14.0226392Z Pipelines: 188 object(s) loaded. 2023-06-01T06:21:36.2502433Z DataSets: 521 object(s) loaded. 2023-06-01T06:21:44.9889263Z DataFlows: 99 object(s) loaded. 2023-06-01T06:21:46.3668523Z Triggers: 63 object(s) loaded. 2023-06-01T06:21:46.3768432Z Managed VNet: 1 object(s) loaded. 2023-06-01T06:21:46.3961335Z Managed Private Endpoints: 1 object(s) loaded. 2023-06-01T06:21:46.3997584Z Credentials: 0 object(s) loaded. 2023-06-01T06:21:46.4295994Z Factories: 1 object(s) loaded. 2023-06-01T06:21:46.5155202Z =================================================================================== 2023-06-01T06:21:46.5167793Z STEP: Pre-deployment 2023-06-01T06:21:46.5178828Z =================================================================================== 2023-06-01T06:21:46.5189996Z STEP: Replacing all properties environment-related... 2023-06-01T06:21:46.5218214Z Stage parameter was not provided - action skipped. 2023-06-01T06:21:46.5229377Z =================================================================================== 2023-06-01T06:21:46.5240996Z STEP: Determining the objects to be deployed... 2023-06-01T06:21:47.4702225Z # Number of objects marked as to be deployed: 2/931 2023-06-01T06:21:47.4861645Z - [dataflow].[DataFlowxxx] 2023-06-01T06:21:47.4876918Z - [dataflow].[DataFlowyyy] 2023-06-01T06:21:47.4889025Z =================================================================================== 2023-06-01T06:21:47.4899210Z STEP: Stopping triggers... 2023-06-01T06:21:47.4924110Z Operation skipped as publish option 'StopStartTriggers' = false 2023-06-01T06:21:47.4935837Z =================================================================================== 2023-06-01T06:21:47.5057167Z STEP: Deployment of all ADF objects... 2023-06-01T06:21:47.7922421Z Start deploying object: [dataflow].[DataFlowxx] (4 dependency/ies) 2023-06-01T06:21:47.9616294Z ##[warning]ADFT0006: Referenced object [Dataset].[AzureDataLakeStorexx] was not found. No error raised as user wanted to carry on. 2023-06-01T06:21:48.7922961Z ##[error]BadRequest : The document creation or update failed because of invalid reference 'azuredatalakestoreyyy'. CorrelationId: 5896fc05-f0d3-49ea-bbbb-ee293c138af8

dayalus commented 1 year ago

Please ignore it for now. We think it may be an error on our side with naming the JSON files..Will close this if confirmed

dayalus commented 1 year ago

Yes, it was a dataset type error from our side. My apologies!