Azure-Player / azure.datafactory.tools

Tools for deploying Data Factory (v2) in Microsoft Azure
https://azureplayer.net/adftools
MIT License
207 stars 69 forks source link

Test-AdfCode fails when using pwsh inside a docker image #394

Closed cristianastorino closed 1 month ago

cristianastorino commented 1 month ago

Describe the bug While executing the Test-AdfCode cmdlet, I'm receiving the following error:

=== Loading files from location: /tmp/ADFDev2/ ...
IntegrationRuntimes: 0 object(s) loaded.
LinkedServices: 1 object(s) loaded.
Pipelines: 2 object(s) loaded.
DataSets: 1 object(s) loaded.
DataFlows: 0 object(s) loaded.
Triggers: 0 object(s) loaded.
Managed VNet: 0 object(s) loaded.
Managed Private Endpoints: 0 object(s) loaded.
Credentials: 0 object(s) loaded.
Factories: 0 object(s) loaded.
=== Validating files ...
Checking: [linkedService].[AzureDataLakeStorage1]...
Checking: [pipeline].[pipeline1]...
Checking: [pipeline].[pipeline2]...
Checking: [dataset].[Parquet1]...
=== Validating other rules ...
Checking duplicated names...
Checking names of linkedservices, datasets, dataflows...
Checking: Global parameter names...
=== Validating config files ...
Get-ChildItem: /root/.local/share/powershell/Modules/azure.datafactory.tools/1.9.0/public/Test-AdfCode.ps1:121
Line |
 121 |  …    $files = Get-ChildItem -Path $filePattern -Include '*.csv','*.json …
     |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     | Cannot process argument because the value of argument "path" is not valid. Change the value of the "path" argument and run the operation again.
ERROR: The property 'FullName' cannot be found on this object. Verify that the property exists.

looks like the problem is on that line: https://github.com/Azure-Player/azure.datafactory.tools/blob/master/public/Test-AdfCode.ps1#L121

if I execute the following command inside a linux docker image with powershell installed, I'm getting the exact same error:

Get-ChildItem -Path $null -Include '*.csv','*.json'
Get-ChildItem: Cannot process argument because the value of argument "path" is not valid. Change the value of the "path" argument and run the operation again.

but if I do the same on a windows PS, I get no errors:

Get-ChildItem -Path $null -Include '*.csv','*.json'

To Reproduce Have pwsh installed inside a linux docker image. I'm using this version:

PS /> $PSVersionTable.PSVersion

Major  Minor  Patch  PreReleaseLabel BuildLabel
-----  -----  -----  --------------- ----------
7      4      2

Expected behaviour No error is returned

Version of PowerShell module of:

NowinskiK commented 1 month ago

What value do you provide for $ConfigPath input parameter? Line 116 resets the $filePattern variable when the file doesn't exist in 'deployment' folder. I'll add additional logs in debug mode, but check all paths and files you're pointing.

cristianastorino commented 1 month ago

Actually I'm not passing $ConfigPath.

I think the problem is related to this issue: https://github.com/PowerShell/PowerShell/issues/17793 where Get-ChildItem -Path $null returns current directory and in my case is / and looks like with / the command is failing:

PS /> ls -l
total 156116
lrwxrwxrwx   1 root root         7 May 30 02:03 bin -> usr/bin
drwxr-xr-x   2 root root      4096 Apr 18  2022 boot
-rwxrwxr-x   1 root root      1182 Jun  5 04:54 __cacert_entrypoint.sh
drwxr-xr-x   5 root root       360 Jun 11 10:02 dev
drwxr-xr-x   1 root root      4096 Jun 11 10:09 etc
drwxr-xr-x   2 root root      4096 Apr 18  2022 home
lrwxrwxrwx   1 root root         7 May 30 02:03 lib -> usr/lib
lrwxrwxrwx   1 root root         9 May 30 02:03 lib32 -> usr/lib32
lrwxrwxrwx   1 root root         9 May 30 02:03 lib64 -> usr/lib64
lrwxrwxrwx   1 root root        10 May 30 02:03 libx32 -> usr/libx32
drwxr-xr-x   2 root root      4096 May 30 02:03 media
drwxr-xr-x   2 root root      4096 May 30 02:03 mnt
drwxr-xr-x   1 root root      4096 Jun 11 09:48 opt
dr-xr-xr-x 270 root root         0 Jun 11 10:02 proc
drwx------   1 root root      4096 Jun 11 10:03 root
drwxr-xr-x   1 root root      4096 Jun 11 09:48 run
lrwxrwxrwx   1 root root         8 May 30 02:03 sbin -> usr/sbin
drwxr-xr-x   2 root root      4096 May 30 02:03 srv
dr-xr-xr-x  11 root root         0 Jun 11 10:02 sys
drwxrwxrwt   1 root root      4096 Jun 11 13:55 tmp
drwxr-xr-x   1 root root      4096 May 30 02:03 usr
drwxr-xr-x   1 root root      4096 May 30 02:06 var

PS /> Get-ChildItem -Path $null -Include '*.csv','*.json'
Get-ChildItem: Cannot process argument because the value of argument "path" is not valid. Change the value of the "path" argument and run the operation again.

PS /> Get-ChildItem -Path / -Include '*.csv','*.json'
Get-ChildItem: Cannot process argument because the value of argument "path" is not valid. Change the value of the "path" argument and run the operation again.
PS />

but if I pass another folder it doesn't throw:

PS /> Get-ChildItem -Path /tmp -Include '*.csv','*.json'
PS />
NowinskiK commented 1 month ago

Thanks for all the details and references. I believe the problem is with Path parameter being $null. Weird, but it works fine with PS 5.1 and even in 7.4.2: image However, it will be worth to fix it as this function is the only one which doesn't complain (in most cases) of null value for path.

cristianastorino commented 1 month ago

My suspect is that in your case is not throwing because your current work directory is /home/kamil. Can you try with:

PS /> cd /
PS /> Get-ChildItem -Path $null -Include '*.csv','*.json'

I'm not sure of the reason why when the working directory is / it throws, maybe because of symbolic links.

Anyway, I think Get-ChildItem should be called only if $path is not $null, else you risk to read csv or json that are in your current working directory and are not part of your configuration