search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.6k forks source link

CopyFiles doesn't match !(dirA|dirB)/** correctly #10395

Closed vtbassmatt closed 5 years ago

vtbassmatt commented 5 years ago

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: CopyFiles

Environment

Azure Pipelines, hosted agent, Hosted Ubuntu 16.04 pool

Issue Description

Given this task input:

steps:
- task: CopyFiles@2
  inputs:
    SourceFolder: $(Build.SourcesDirectory)
    TargetFolder: $(Build.ArtifactStagingDirectory)
    contents: |
      **
      !(node_modules|.git)/**

I expect all files except the contents of node_modules and .git to be copied. Instead, all files are copied.

Expanding the parenthetical myself results in correct behavior:

steps:
- task: CopyFiles@2
  inputs:
    SourceFolder: $(Build.SourcesDirectory)
    TargetFolder: $(Build.ArtifactStagingDirectory)
    contents: |
      **
      !node_modules/**
      !.git/**

Task logs

1##[debug]Evaluating condition for step: 'CopyFiles'
2##[debug]Evaluating: SucceededNode()
3##[debug]Evaluating SucceededNode:
4##[debug]=> True
5##[debug]Result: True
6Starting: CopyFiles
7==============================================================================
8Task         : Copy Files
9Description  : Copy files from source folder to target folder using match patterns (The match patterns will only match file paths, not folder paths)
10Version      : 2.117.2
11Author       : Microsoft Corporation
12Help         : [More Information](https://go.microsoft.com/fwlink/?LinkID=708389)
13==============================================================================
14##[debug]agent.workFolder=/home/vsts/work
15##[debug]loading inputs and endpoints
16##[debug]loading INPUT_SOURCEFOLDER
17##[debug]loading INPUT_CONTENTS
18##[debug]loading INPUT_TARGETFOLDER
19##[debug]loading INPUT_CLEANTARGETFOLDER
20##[debug]loading INPUT_OVERWRITE
21##[debug]loading INPUT_FLATTENFOLDERS
22##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
23##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
24##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
25##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
26##[debug]loaded 10
27##[debug]check path : /home/vsts/work/_tasks/CopyFiles_5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c/2.117.2/task.json
28##[debug]set resource file to: /home/vsts/work/_tasks/CopyFiles_5bfb729a-a7c8-4a78-a7c3-8d717bb7c13c/2.117.2/task.json
29##[debug]system.culture=en-US
30##[debug]Contents=**
31!(node_modules|.git)/**
32##[debug]SourceFolder=/home/vsts/work/1/s
33##[debug]check path : /home/vsts/work/1/s
34##[debug]TargetFolder=/home/vsts/work/1/a
35##[debug]CleanTargetFolder=false
36##[debug]OverWrite=false
37##[debug]flattenFolders=false
38##[debug]findPath: '/home/vsts/work/1/s'
39##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
40##[debug]findOptions.followSymbolicLinks: 'true'
41##[debug]  /home/vsts/work/1/s (directory)
42##[debug]  /home/vsts/work/1/s/.git (directory)
43##[debug]  /home/vsts/work/1/s/.git/FETCH_HEAD (file)
44##[debug]  /home/vsts/work/1/s/.git/HEAD (file)
45##[debug]  /home/vsts/work/1/s/.git/branches (directory)
46##[debug]  /home/vsts/work/1/s/.git/config (file)
47##[debug]  /home/vsts/work/1/s/.git/description (file)
48##[debug]  /home/vsts/work/1/s/.git/hooks (directory)
49##[debug]  /home/vsts/work/1/s/.git/hooks/applypatch-msg.sample (file)
50##[debug]  /home/vsts/work/1/s/.git/hooks/commit-msg.sample (file)
51##[debug]  /home/vsts/work/1/s/.git/hooks/fsmonitor-watchman.sample (file)
52##[debug]  /home/vsts/work/1/s/.git/hooks/post-update.sample (file)
53##[debug]  /home/vsts/work/1/s/.git/hooks/pre-applypatch.sample (file)
54##[debug]  /home/vsts/work/1/s/.git/hooks/pre-commit.sample (file)
55##[debug]  /home/vsts/work/1/s/.git/hooks/pre-push.sample (file)
56##[debug]  /home/vsts/work/1/s/.git/hooks/pre-rebase.sample (file)
57##[debug]  /home/vsts/work/1/s/.git/hooks/pre-receive.sample (file)
58##[debug]  /home/vsts/work/1/s/.git/hooks/prepare-commit-msg.sample (file)
59##[debug]  /home/vsts/work/1/s/.git/hooks/update.sample (file)
60##[debug]  /home/vsts/work/1/s/.git/index (file)
61##[debug]  /home/vsts/work/1/s/.git/info (directory)
62##[debug]  /home/vsts/work/1/s/.git/info/exclude (file)
63##[debug]  /home/vsts/work/1/s/.git/logs (directory)
64##[debug]  /home/vsts/work/1/s/.git/logs/HEAD (file)
65##[debug]  /home/vsts/work/1/s/.git/logs/refs (directory)
66##[debug]  /home/vsts/work/1/s/.git/logs/refs/remotes (directory)
67##[debug]  /home/vsts/work/1/s/.git/logs/refs/remotes/origin (directory)
68##[debug]  /home/vsts/work/1/s/.git/logs/refs/remotes/origin/master (file)
69##[debug]  /home/vsts/work/1/s/.git/objects (directory)
70##[debug]  /home/vsts/work/1/s/.git/objects/info (directory)
71##[debug]  /home/vsts/work/1/s/.git/objects/pack (directory)
72##[debug]  /home/vsts/work/1/s/.git/objects/pack/pack-7f92682f6cfd1ebb986608aa48640217db036e23.idx (file)
73##[debug]  /home/vsts/work/1/s/.git/objects/pack/pack-7f92682f6cfd1ebb986608aa48640217db036e23.pack (file)
74##[debug]  /home/vsts/work/1/s/.git/refs (directory)
75##[debug]  /home/vsts/work/1/s/.git/refs/heads (directory)
76##[debug]  /home/vsts/work/1/s/.git/refs/remotes (directory)
77##[debug]  /home/vsts/work/1/s/.git/refs/remotes/origin (directory)
78##[debug]  /home/vsts/work/1/s/.git/refs/remotes/origin/master (file)
79##[debug]  /home/vsts/work/1/s/.git/refs/tags (directory)
80##[debug]  /home/vsts/work/1/s/azure-pipelines.yml (file)
81##[debug]  /home/vsts/work/1/s/do-want (directory)
82##[debug]  /home/vsts/work/1/s/do-want/i-want-this-file.js (file)
83##[debug]  /home/vsts/work/1/s/node_modules (directory)
84##[debug]  /home/vsts/work/1/s/node_modules/a-module.js (file)
85##[debug]  /home/vsts/work/1/s/plain-file-i-want.js (file)
86##[debug]45 results
87##[debug]patternRoot: '/home/vsts/work/1/s'
88##[debug]matchOptions.debug: 'false'
89##[debug]matchOptions.nobrace: 'true'
90##[debug]matchOptions.noglobstar: 'false'
91##[debug]matchOptions.dot: 'true'
92##[debug]matchOptions.noext: 'false'
93##[debug]matchOptions.nocase: 'false'
94##[debug]matchOptions.nonull: 'false'
95##[debug]matchOptions.matchBase: 'false'
96##[debug]matchOptions.nocomment: 'false'
97##[debug]matchOptions.nonegate: 'false'
98##[debug]matchOptions.flipNegate: 'false'
99##[debug]pattern: '**'
100##[debug]rooted pattern: '/home/vsts/work/1/s/**'
101##[debug]applying include pattern against original list
102##[debug]44 matches
103##[debug]pattern: '!(node_modules|.git)/**'
104##[debug]trimmed leading '!'. pattern: '(node_modules|.git)/**'
105##[debug]rooted pattern: '/home/vsts/work/1/s/(node_modules|.git)/**'
106##[debug]applying exclude pattern against original list
107##[debug]0 matches
108##[debug]44 final results
109found 26 files
110##[debug]testing directory '/home/vsts/work/1/a'
111##[debug]testing directory '/home/vsts/work/1/a/.git'
112##[debug]testing directory '/home/vsts/work/1/a'
113##[debug]mkdir '/home/vsts/work/1/a/.git'
114Copying /home/vsts/work/1/s/.git/FETCH_HEAD to /home/vsts/work/1/a/.git/FETCH_HEAD
115Copying /home/vsts/work/1/s/.git/HEAD to /home/vsts/work/1/a/.git/HEAD
116Copying /home/vsts/work/1/s/.git/config to /home/vsts/work/1/a/.git/config
117Copying /home/vsts/work/1/s/.git/description to /home/vsts/work/1/a/.git/description
118##[debug]testing directory '/home/vsts/work/1/a/.git/hooks'
119##[debug]testing directory '/home/vsts/work/1/a/.git'
120##[debug]mkdir '/home/vsts/work/1/a/.git/hooks'
121Copying /home/vsts/work/1/s/.git/hooks/applypatch-msg.sample to /home/vsts/work/1/a/.git/hooks/applypatch-msg.sample
122Copying /home/vsts/work/1/s/.git/hooks/commit-msg.sample to /home/vsts/work/1/a/.git/hooks/commit-msg.sample
123Copying /home/vsts/work/1/s/.git/hooks/fsmonitor-watchman.sample to /home/vsts/work/1/a/.git/hooks/fsmonitor-watchman.sample
124Copying /home/vsts/work/1/s/.git/hooks/post-update.sample to /home/vsts/work/1/a/.git/hooks/post-update.sample
125Copying /home/vsts/work/1/s/.git/hooks/pre-applypatch.sample to /home/vsts/work/1/a/.git/hooks/pre-applypatch.sample
126Copying /home/vsts/work/1/s/.git/hooks/pre-commit.sample to /home/vsts/work/1/a/.git/hooks/pre-commit.sample
127Copying /home/vsts/work/1/s/.git/hooks/pre-push.sample to /home/vsts/work/1/a/.git/hooks/pre-push.sample
128Copying /home/vsts/work/1/s/.git/hooks/pre-rebase.sample to /home/vsts/work/1/a/.git/hooks/pre-rebase.sample
129Copying /home/vsts/work/1/s/.git/hooks/pre-receive.sample to /home/vsts/work/1/a/.git/hooks/pre-receive.sample
130Copying /home/vsts/work/1/s/.git/hooks/prepare-commit-msg.sample to /home/vsts/work/1/a/.git/hooks/prepare-commit-msg.sample
131Copying /home/vsts/work/1/s/.git/hooks/update.sample to /home/vsts/work/1/a/.git/hooks/update.sample
132Copying /home/vsts/work/1/s/.git/index to /home/vsts/work/1/a/.git/index
133##[debug]testing directory '/home/vsts/work/1/a/.git/info'
134##[debug]testing directory '/home/vsts/work/1/a/.git'
135##[debug]mkdir '/home/vsts/work/1/a/.git/info'
136Copying /home/vsts/work/1/s/.git/info/exclude to /home/vsts/work/1/a/.git/info/exclude
137##[debug]testing directory '/home/vsts/work/1/a/.git/logs'
138##[debug]testing directory '/home/vsts/work/1/a/.git'
139##[debug]mkdir '/home/vsts/work/1/a/.git/logs'
140Copying /home/vsts/work/1/s/.git/logs/HEAD to /home/vsts/work/1/a/.git/logs/HEAD
141##[debug]testing directory '/home/vsts/work/1/a/.git/logs/refs/remotes/origin'
142##[debug]testing directory '/home/vsts/work/1/a/.git/logs/refs/remotes'
143##[debug]testing directory '/home/vsts/work/1/a/.git/logs/refs'
144##[debug]testing directory '/home/vsts/work/1/a/.git/logs'
145##[debug]mkdir '/home/vsts/work/1/a/.git/logs/refs'
146##[debug]mkdir '/home/vsts/work/1/a/.git/logs/refs/remotes'
147##[debug]mkdir '/home/vsts/work/1/a/.git/logs/refs/remotes/origin'
148Copying /home/vsts/work/1/s/.git/logs/refs/remotes/origin/master to /home/vsts/work/1/a/.git/logs/refs/remotes/origin/master
149##[debug]testing directory '/home/vsts/work/1/a/.git/objects/pack'
150##[debug]testing directory '/home/vsts/work/1/a/.git/objects'
151##[debug]testing directory '/home/vsts/work/1/a/.git'
152##[debug]mkdir '/home/vsts/work/1/a/.git/objects'
153##[debug]mkdir '/home/vsts/work/1/a/.git/objects/pack'
154Copying /home/vsts/work/1/s/.git/objects/pack/pack-7f92682f6cfd1ebb986608aa48640217db036e23.idx to /home/vsts/work/1/a/.git/objects/pack/pack-7f92682f6cfd1ebb986608aa48640217db036e23.idx
155Copying /home/vsts/work/1/s/.git/objects/pack/pack-7f92682f6cfd1ebb986608aa48640217db036e23.pack to /home/vsts/work/1/a/.git/objects/pack/pack-7f92682f6cfd1ebb986608aa48640217db036e23.pack
156##[debug]testing directory '/home/vsts/work/1/a/.git/refs/remotes/origin'
157##[debug]testing directory '/home/vsts/work/1/a/.git/refs/remotes'
158##[debug]testing directory '/home/vsts/work/1/a/.git/refs'
159##[debug]testing directory '/home/vsts/work/1/a/.git'
160##[debug]mkdir '/home/vsts/work/1/a/.git/refs'
161##[debug]mkdir '/home/vsts/work/1/a/.git/refs/remotes'
162##[debug]mkdir '/home/vsts/work/1/a/.git/refs/remotes/origin'
163Copying /home/vsts/work/1/s/.git/refs/remotes/origin/master to /home/vsts/work/1/a/.git/refs/remotes/origin/master
164##[debug]testing directory '/home/vsts/work/1/a'
165Copying /home/vsts/work/1/s/azure-pipelines.yml to /home/vsts/work/1/a/azure-pipelines.yml
166##[debug]testing directory '/home/vsts/work/1/a/do-want'
167##[debug]testing directory '/home/vsts/work/1/a'
168##[debug]mkdir '/home/vsts/work/1/a/do-want'
169Copying /home/vsts/work/1/s/do-want/i-want-this-file.js to /home/vsts/work/1/a/do-want/i-want-this-file.js
170##[debug]testing directory '/home/vsts/work/1/a/node_modules'
171##[debug]testing directory '/home/vsts/work/1/a'
172##[debug]mkdir '/home/vsts/work/1/a/node_modules'
173Copying /home/vsts/work/1/s/node_modules/a-module.js to /home/vsts/work/1/a/node_modules/a-module.js
174Copying /home/vsts/work/1/s/plain-file-i-want.js to /home/vsts/work/1/a/plain-file-i-want.js
175Finishing: CopyFiles
damccorm commented 5 years ago

I think this is actually right behavior. We follow minimatch's globbing behavior, which in turn follows fnmatch and produces the following:

minimatch("bar/foo", "(bar|abc)/**") = false minimatch("bar/foo", "+(bar|abc)/**") = true minimatch("bar/foo", "!(bar|abc)/**") = true minimatch("bar/foo", "!+(bar|abc)/**") = false

So I think this should work if you change the above to !+(node_modules|.git)/**, its currently missing the leading + which indicates that we're entering into a pattern list

vtbassmatt commented 5 years ago

Ah, thanks. I didn't find that guide to fnmatch in my (admittedly brief) search.

DaleyKD commented 1 year ago

I know this is super old, but this is the only thing I found online to assist me in figuring this out, so I wanted to add one other quick glob example (since fnmatch/minimatch is new to me).

In my case, I wanted to exclude a bunch of directories with the CopyFiles task, and some were subdirs.

I ended up using the following to copy a bunch of stuff but exclude the proper folders:

!**/+(x86|x64|obj|packages|.git)/**

The trick for me was to put the negative operator at the beginning of the row, then use the + operator next to the list.

Thank you for your help!

Also, this codepen helped me figure it out: https://codepen.io/mrmlnc/pen/OXQjMe