(Optionally) Exclude tests and demos from packages (PyQt, matplotlib, etc.)

[mfitzp]

Including tests and demos in the installer increases the size of the installer and the installation time. Is it feasible to add an option to exclude these subsets of packages somehow?

I'm not really clear on how you would do this without possibly breaking things but a path filter may be the simplest way to go about it.

[takluyver]

That makes sense, though I'm not sure what's the best way to handle it. Is it safe to remove any subdirectories with a fixed list of names like 'test', 'demo', etc.? Or should that be configurable, either globally or per-package? Or should the user need to write custom scripts to do things like this?

[jbaiter]

I'd welcome the option to add an exclude parameter that takes a list of regular expression strings that match on directory/file names, akin to .gitignore.

[takluyver]

Seems reasonable. Would you treat packages and other files separately, or just have one set that can refer to the pkgs directory?

[jbaiter]

I think the easiest would be to just make the patterns match on pkgs subdirectories/files. That way you can either do catch-all patterns like .*/test[s]?/.* or more fine-grained ones like PyQt-.*/demo/.*.

[mfitzp]

My preference would be for what @jbaiter suggests, leaving it up to individual users to define the exclusions i the config rather than a fixed list. The neat thing about pynsist is that there's no esoteric weirdness to the packaging.

If I understand correctly the files are copied on-bulk to the pkgs directory then added into the installer. So if it would be possible to filter them at this point that would be perfect. The PyQt demo files were what initially made me want to request this.

[takluyver]

@jbaiter I wonder if the root of the patterns should be the build directory, rather than pkgs, so that you can use the same config to do something like this:

[Include] files = somedir/ exclude = somedir/blah

i.e. Copy the folder somedir, apart from the file/subfolder blah within that. Then patterns to filter packages would be e.g. pkgs/PyQt-.*/demo. I'm ambivalent about this, though, because people can script that kind of thing as a separate preparation step if they prefer.

@mfitzp Yes, they are just copied into pkgs, and the installer is built including that whole folder. So I would implement this as 'copy into pkgs, then delete everything specified by excludes, then build'.

Another question: regex patterns (powerful but fiddly) or globs (easy but less flexible)? More complex options include allowing both like .hgignore does, or defining an extended glob syntax, as .gitignore appears to. I prefer working with globs, and I think they'll suffice for 99% of these cases, but there is that 1%. Maybe it should be globs in the config file, with a mechanism in the Python API to supply regex patterns.

[mfitzp]

I'm a bit ambivalent on the pkgs vs. build question too, but I think the latter has the benefit that it covers the more general case and keeps the path target structure consistent as for the files= command. I think that is more 'obvious' behaviour.

On the globs question I suspect that globs will do all of the work that's likely to be needed. I'm only looking to bulk-exclude very simply, very regularly named directories. Other things can likely be managed by processing scripts with the API?

[jbaiter]

Maybe it should be globs in the config file, with a mechanism in the Python API to supply regex patterns.

That sounds reasonable. One could also keep it consistent by allowing either unicode/str in the pattern list as well as compiled regular expression patterns. Strings would be evaluated as glob-patterns and the patterns used as-is.

[takluyver]

OK. Let's build it with just glob patterns for now, and work out what makes sense for regex patterns as and when the need arises.

I'm leaning towards the patterns being rooted in the build directory, and documenting that to exclude things from packages, you should specify the pkgs/ directory at the start of the pattern.

It will be a couple of weeks before I have time to work on this, but if either of you want to make a PR, I should be able to review it sooner than that.

[raphaelm]

:+1: for this! :)

If you're going for glob patterns, why copy first and delete them afterwards instead of using shutil.copytree's ignore patterns when you're already using shutil?

[raphaelm]

Alright, you got me, I'm working on this ;)

[takluyver]

Fixed by PR #34 .