Externalise ArtifactSource filtering

[kipz]

Problem:

TS projects will always have a .gitignore containing node_modules. However, this currently means that rug-cli will ignore this too, and not make it available to the compiler and not publish it. The current solution is to add !node_modules to .atomist/ignore. Although this works, this cause huge confusion because the expectation is always that the typescript compiler (TSC) will have nothing to do with .gitignore and will decide how/where to load modules from.

Additionally, this only currently works for FileSystemArtifactSource's, which will make it operating on Github repos tricky.

It also feels like it shouldn't be the responsibility of the ArtifactSource to decide what is filter and when.

Proposal:

Rather than complicating the model in this project further, @cdupuis and I discussed the following solution so that users of this library can have finer grained control over filtering.

• Do not apply any filters by default when created ArtifactSources (e.g. for .gitignore or .atomist/ignore)
• Provide constructors (for all ArtifactSource implementations) that accept implementations of some filter interface which filters the ArtifactSource during filesystem recursion (for performance and so that the ArtifactSource doesn't ever know about excluded files)
• Create implementations of said interface for the current filters used in FileArtifactSource (.gitignore etc)
• Create a filter method on all ArtifactSource instances that accept implementations of the filter interface and returns a new ArtifactSource without the excluded files (i.e. non-mutating).

This is a breaking change to the behaviour of this library...

[alankstewart]

I agree that filtering is not the responsibility of this library. Will begin making the changes. ArtifactSource already does have a filter method which takes in a DirFilter and FileFilter.

[johnsonr]

Just discussed with Alan. We should add ability to pass two functions to anything that returns an ArtifactSource: One that checks a directory path to see whether it (and its descendants) should be included in the ArtifactSource; and one that checks a file path.

The present filtering mechanism once an ArtifactSource has been constructed should remain unchanged.

Our existing .gitignore support should migrate to the former approach.

[alankstewart]

Callers can send in zero or more filters implementing the ArtifactFilter trait which will apply in order. I have removed JGit from artifact-source and am now using Spring's AntPathMatcher from spring-core. JGit did not filter out the .git directory and so now I created a separate filter called GitDirFilter which can be passed in to the list of filters when creating an ArtifactSource. There is also a GitignoreFileFilter and a AtomistIgnoreFileFilter.

@cdupuis , @kipz I haven't released 0.14.0 of artifact-source yet and I added some sample (commented-out) code to rug-cli which uses the new filters. You can pull artifact-source and use locally. There is no negation of patterns anymore like putting in !node_modules in .atomist/ignore and hoping it will be included if it is specified in .gitignore. These filters work independently so if you want previous behaviour of having to negate !node_modules in .atomist/ignore for example, you will have to write a new filter which handles both ignore files.

[alankstewart]

Sorry, last commits were incorrectly assigned to this issue. Anyhow, this should be complete now