search

Azure-Player / azure.datafactory.devops

Azure DevOps Deployment Tasks for Azure Data Factory objects
MIT License
39 stars 21 forks source link

BuildADFTask action Export encounters errors, but still succeeds #95

Closed peterhut closed 2 years ago

peterhut commented 2 years ago

Currently using the BuildADFTask with Action: Export in the build for Pull Requests in Azure DevOps to validate the Data Factory configuration. It clearly shows any Validation errors, so it is working very well for that, but I expected that if the validation failed that the Task would fail as well, but instead it finishes as if it succeeded.

Is that the expected behaviour?

Azure DevOps build log

NowinskiK commented 2 years ago

The problem is that npm package doesn't return any value (true/false, 0, -1, etc) in order to indicate whether the process finished successfully or failed. I can implement some workaround for the issue by checking if ARMTemplate file has been created as a result of export command.

NowinskiK commented 2 years ago

This is for the Microsoft Team record: Guys, there are a lot of confusing (error) messages produce by export command from npm library. They look like errors, however, the whole process finishes successfully. Example: 6D89C8E4-2C27-4362-8D83-906726F170AA

Text LOG:

=== Preparing package.json file...
=== File 'package.json' created.
=== Check NPM Version...
{
  npm: '8.1.0',
  node: '17.0.1',
  v8: '9.5.172.21-node.12',
  uv: '1.42.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.17.2',
  modules: '102',
  nghttp2: '1.45.1',
  napi: '8',
  llhttp: '6.0.4',
  openssl: '3.0.0+quic',
  cldr: '39.0',
  icu: '69.1',
  tz: '2021a',
  unicode: '13.0',
  ngtcp2: '0.1.0-DEV',
  nghttp3: '0.1.0-DEV'
}
=== Check finished.
=== Installing NPM azure-data-factory-utilities...

added 2 packages, and audited 3 packages in 2s

found 0 vulnerabilities
npm notice
npm notice New minor version of npm available! 8.1.0 -> 8.16.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.16.0
npm notice Run npm install -g npm@8.16.0 to update!
npm notice
=== Installation finished.
=== Validating & exporting ARM Template...

> build
> node node_modules/@microsoft/azure-data-factory-utilities/lib/index "export" "X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1" "/subscriptions/ffff-ffff/resourceGroups/abcxyz/providers/Microsoft.DataFactory/factories/adf1" "ArmTemplate"

Downloading bundle from: https://adf.azure.com/assets/cmd-api/main.js
Process cwd: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1
Bundle downloaded successfully, saved in: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\downloads\main.js
Executing bundle...
Inserting the following arguments: export X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1 /subscriptions/ffff-ffff/resourceGroups/abcxyz/providers/Microsoft.DataFactory/factories/adf1 ArmTemplate
Executing bundle file, full command:

node  X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\downloads\main.js export X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1 /subscriptions/ffff-ffff/resourceGroups/abcxyz/providers/Microsoft.DataFactory/factories/adf1 ArmTemplate

 CmdApiApp: Initializing application.
Resource: /subscriptions/ffff-ffff/resourceGroups/abcxyz/providers/Microsoft.DataFactory/factories/adf1
RootFolder: X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1
 ModelService: synchronize - start
 ModelService: Dynamic connector - Start registering dynamic connectors
 CmdApiApp: Initializing resource registries...
 CmdApiApp: Initializing model service...
 DynamicConnectorService: Dynamic connector - Start registering connector: Dataworld
 DynamicConnectorService: Dynamic connector - Finished registering connector: Dataworld
 DynamicConnectorService: Dynamic connector - Start registering connector: Asana
 DynamicConnectorService: Dynamic connector - Finished registering connector: Asana
 DynamicConnectorService: Dynamic connector - Start registering connector: Twilio
 DynamicConnectorService: Dynamic connector - Finished registering connector: Twilio
 ModelService: Dynamic connector - Finished registering dynamic connectors
 BaseFileResourceProviderService: populateAllResources - start
 CmdApiApp: Initializing publish config service...
 PublishConfigService: _getLatestPublishConfig - retrieving config file.
 LocalFileClientService: Unable to list files for: integrationRuntime, error: Error: ENOENT: no such file or directory, scandir 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\integrationRuntime'
 LocalFileClientService: Unable to list files for: pipeline, error: Error: ENOENT: no such file or directory, scandir 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\pipeline'
 LocalFileClientService: Unable to list files for: trigger, error: Error: ENOENT: no such file or directory, scandir 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\trigger'
 LocalFileClientService: Unable to list files for: factory, error: Error: ENOENT: no such file or directory, scandir 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\factory'
 LocalFileClientService: Unable to list files for: credential, error: Error: ENOENT: no such file or directory, scandir 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\credential'
 LocalFileClientService: Unable to list files for: managedVirtualNetwork, error: Error: ENOENT: no such file or directory, scandir 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\managedVirtualNetwork'
 ERROR === LocalFileClientService: Unable to read file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\publish_config.json, error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","message":"ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json"}
 ERROR === PublishConfigService: _getLatestPublishConfig - Unable to process publish config file, error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","message":"ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json"}
 CmdApiApp: App initialized.
 BaseFileResourceProviderService: populateAllResources - downloaded git resources
 BaseFileResourceProviderService: populateAllResources - loaded last version id
 BaseFileResourceProviderService: populateAllResources - finished
 ModelService: fetchAllInBackground - start
 ModelService: synchronize - finished
 CmdApiApp: Starting to validate all resources
 ModelService: _backgroundFetch - finished, duration: 220.18340015411377, total nameds: 5
 CmdApiApp:
Validation finished. No errors found.
 ArmTemplateGeneratorService: Generating templates
 ArmTemplateGeneratorService: Checking if dataflows need parsing.
 ModelService: fetchAll - start. Fetch named count: 4, total resource count: 4
 ArmTemplateGeneratorService: Dataflows need parsing: true
 ERROR === LocalFileClientService: Unable to read file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\arm-template-parameters-definition.json, error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","message":"ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json"}
 WARNING === ArmTemplateUtils: _getUserParameterDefinitionJson - Unable to load custom param file from repo, will use default file. Error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","message":"ENOENT: no such file or directory, open 
'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json"}
 ArmTemplateGeneratorService: Generating full template
 ModelService: fetchAll - start. Fetch named count: 4, total resource count: 4
 ArmTemplateGeneratorService: Generating full template parameters file
 ArmTemplateGeneratorService: Generating Linked Templates
 ModelService: fetchAll - start. Fetch named count: 4, total resource count: 4
 ERROR === LocalFileClientService: Unable to read file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\arm-template-parameters-definition.json, error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","message":"ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json"}
 WARNING === ArmTemplateUtils: _getUserParameterDefinitionJson - Unable to load custom param file from repo, will use default file. Error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","message":"ENOENT: no such file or directory, open 
'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\arm-template-parameters-definition.json"}
 ArmTemplateGeneratorService: Building master arm template and parameters file
 ModelService: fetchAll - start. Fetch named count: 4, total resource count: 4
 ArmTemplateGeneratorService: Cleaning dataflows.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\ARMTemplateForFactory.json
 LocalFileUtils: Dir: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate does not exist, creating it....
 LocalFileUtils: File saved.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\ARMTemplateParametersForFactory.json
 LocalFileUtils: File saved.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\linkedTemplates\ArmTemplate_0.json
 LocalFileUtils: Dir: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\linkedTemplates does not exist, creating it....
 LocalFileUtils: File saved.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\linkedTemplates\ArmTemplate_master.json
 LocalFileUtils: File saved.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\linkedTemplates\ArmTemplateParameters_master.json
 LocalFileUtils: File saved.
 CmdApiApp: ARM template generated. Writing files...
 PublishConfigService: _getLatestPublishConfig - retrieving config file.
 ERROR === LocalFileClientService: Unable to read file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\publish_config.json, error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","message":"ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json"}
 ERROR === PublishConfigService: _getLatestPublishConfig - Unable to process publish config file, error: {"stack":"Error: ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","message":"ENOENT: no such file or directory, open 'X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json'","errno":-4058,"code":"ENOENT","syscall":"open","path":"X:\\!WORK\\GitHub\\!SQLPlayer\\azure.datafactory.tools\\test\\adf1\\publish_config.json"}
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\PrePostDeploymentScript.ps1
 LocalFileUtils: File saved.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\GlobalParametersUpdateScript.ps1
 LocalFileUtils: File saved.
 LocalFileUtils: Saving file: X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\adf1\ArmTemplate\adf1_GlobalParameters.json
 LocalFileUtils: File saved.
 CmdApiApp: ARM template export finished successfully.

Execution finished....
=== Export finished.

The problem is that the error like this should NOT even appear: image

Some files and folders are not required to make ADF valid.

NowinskiK commented 2 years ago

Released in ver. 1.13