search

mgedmin / check-manifest

Tool to check the completeness of MANIFEST.in for Python packages
https://pypi.org/p/check-manifest
MIT License
289 stars 38 forks source link

`prune **` excludes absolutely everything, even if followed by `graft` and `include` directives #144

Open matthewfeickert opened 3 years ago

matthewfeickert commented 3 years ago

After looking around in the Issues related to prune I think the answer is "no", but I thought I'd ask here explicitly though I think this is just (maybe?) related to Issue #115.

A project I work on has prune ** in our MANIFEST.in, which I'll copy here as well:

$ cat MANIFEST.in 
prune **
graft src

include setup.py
include setup.cfg
include LICENSE
include README.rst
include pyproject.toml
include MANIFEST.in
include AUTHORS

global-exclude __pycache__ *.py[cod]

I've recently noticed that while check-manifest is passing when run here, it isn't doing anything as it finds 0 files

$ check-manifest --version
check-manifest version 0.46
$ check-manifest --verbose
listing source files under version control: 0 files and directories
building an sdist: pyhf-0.6.4.dev8.tar.gz: 0 files and directories
copying source files to a temporary directory
building a clean sdist: pyhf-0.6.4.dev8.tar.gz: 0 files and directories
lists of files in version control and sdist match
$ echo $?
0

(Aside: This might be a nice place to fail out on the user)

However, as (according to the Including files in source distributions with MANIFEST.in PyPA guide)

Setuptools also has undocumented support for ** matching zero or more characters including forward slash, backslash, and colon.

build will happily make an sdist and wheel that has the correct file contents.

$ python -m build --outdir dist/ .
$ python -m tarfile --list dist/*.tar.gz | wc -l
105
$ python -m zipfile --list dist/*.whl | wc -l
81
Full dump from the above: ```console $ python -m build --outdir dist/ . $ python -m tarfile --list dist/*.tar.gz pyhf-0.6.4.dev8/ pyhf-0.6.4.dev8/AUTHORS pyhf-0.6.4.dev8/LICENSE pyhf-0.6.4.dev8/MANIFEST.in pyhf-0.6.4.dev8/PKG-INFO pyhf-0.6.4.dev8/README.rst pyhf-0.6.4.dev8/pyproject.toml pyhf-0.6.4.dev8/setup.cfg pyhf-0.6.4.dev8/setup.py pyhf-0.6.4.dev8/src/ pyhf-0.6.4.dev8/src/pyhf/ pyhf-0.6.4.dev8/src/pyhf/__init__.py pyhf-0.6.4.dev8/src/pyhf/_version.py pyhf-0.6.4.dev8/src/pyhf/cli/ pyhf-0.6.4.dev8/src/pyhf/cli/__init__.py pyhf-0.6.4.dev8/src/pyhf/cli/cli.py pyhf-0.6.4.dev8/src/pyhf/cli/complete.py pyhf-0.6.4.dev8/src/pyhf/cli/infer.py pyhf-0.6.4.dev8/src/pyhf/cli/patchset.py pyhf-0.6.4.dev8/src/pyhf/cli/rootio.py pyhf-0.6.4.dev8/src/pyhf/cli/spec.py pyhf-0.6.4.dev8/src/pyhf/compat.py pyhf-0.6.4.dev8/src/pyhf/constraints.py pyhf-0.6.4.dev8/src/pyhf/contrib/ pyhf-0.6.4.dev8/src/pyhf/contrib/__init__.py pyhf-0.6.4.dev8/src/pyhf/contrib/cli.py pyhf-0.6.4.dev8/src/pyhf/contrib/utils.py pyhf-0.6.4.dev8/src/pyhf/contrib/viz/ pyhf-0.6.4.dev8/src/pyhf/contrib/viz/__init__.py pyhf-0.6.4.dev8/src/pyhf/contrib/viz/brazil.py pyhf-0.6.4.dev8/src/pyhf/data/ pyhf-0.6.4.dev8/src/pyhf/data/citation.bib pyhf-0.6.4.dev8/src/pyhf/events.py pyhf-0.6.4.dev8/src/pyhf/exceptions/ pyhf-0.6.4.dev8/src/pyhf/exceptions/__init__.py pyhf-0.6.4.dev8/src/pyhf/infer/ pyhf-0.6.4.dev8/src/pyhf/infer/__init__.py pyhf-0.6.4.dev8/src/pyhf/infer/calculators.py pyhf-0.6.4.dev8/src/pyhf/infer/intervals.py pyhf-0.6.4.dev8/src/pyhf/infer/mle.py pyhf-0.6.4.dev8/src/pyhf/infer/test_statistics.py pyhf-0.6.4.dev8/src/pyhf/infer/utils.py pyhf-0.6.4.dev8/src/pyhf/interpolators/ pyhf-0.6.4.dev8/src/pyhf/interpolators/__init__.py pyhf-0.6.4.dev8/src/pyhf/interpolators/code0.py pyhf-0.6.4.dev8/src/pyhf/interpolators/code1.py pyhf-0.6.4.dev8/src/pyhf/interpolators/code2.py pyhf-0.6.4.dev8/src/pyhf/interpolators/code4.py pyhf-0.6.4.dev8/src/pyhf/interpolators/code4p.py pyhf-0.6.4.dev8/src/pyhf/mixins.py pyhf-0.6.4.dev8/src/pyhf/modifiers/ pyhf-0.6.4.dev8/src/pyhf/modifiers/__init__.py pyhf-0.6.4.dev8/src/pyhf/modifiers/histosys.py pyhf-0.6.4.dev8/src/pyhf/modifiers/lumi.py pyhf-0.6.4.dev8/src/pyhf/modifiers/normfactor.py pyhf-0.6.4.dev8/src/pyhf/modifiers/normsys.py pyhf-0.6.4.dev8/src/pyhf/modifiers/shapefactor.py pyhf-0.6.4.dev8/src/pyhf/modifiers/shapesys.py pyhf-0.6.4.dev8/src/pyhf/modifiers/staterror.py pyhf-0.6.4.dev8/src/pyhf/optimize/ pyhf-0.6.4.dev8/src/pyhf/optimize/__init__.py pyhf-0.6.4.dev8/src/pyhf/optimize/common.py pyhf-0.6.4.dev8/src/pyhf/optimize/mixins.py pyhf-0.6.4.dev8/src/pyhf/optimize/opt_jax.py pyhf-0.6.4.dev8/src/pyhf/optimize/opt_minuit.py pyhf-0.6.4.dev8/src/pyhf/optimize/opt_numpy.py pyhf-0.6.4.dev8/src/pyhf/optimize/opt_pytorch.py pyhf-0.6.4.dev8/src/pyhf/optimize/opt_scipy.py pyhf-0.6.4.dev8/src/pyhf/optimize/opt_tflow.py pyhf-0.6.4.dev8/src/pyhf/parameters/ pyhf-0.6.4.dev8/src/pyhf/parameters/__init__.py pyhf-0.6.4.dev8/src/pyhf/parameters/paramsets.py pyhf-0.6.4.dev8/src/pyhf/parameters/paramview.py pyhf-0.6.4.dev8/src/pyhf/parameters/utils.py pyhf-0.6.4.dev8/src/pyhf/patchset.py pyhf-0.6.4.dev8/src/pyhf/pdf.py pyhf-0.6.4.dev8/src/pyhf/probability.py pyhf-0.6.4.dev8/src/pyhf/readxml.py pyhf-0.6.4.dev8/src/pyhf/schemas/ pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/ pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/defs.json pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/jsonpatch.json pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/measurement.json pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/model.json pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/patchset.json pyhf-0.6.4.dev8/src/pyhf/schemas/1.0.0/workspace.json pyhf-0.6.4.dev8/src/pyhf/schemas/HistFactorySchema.dtd pyhf-0.6.4.dev8/src/pyhf/simplemodels.py pyhf-0.6.4.dev8/src/pyhf/tensor/ pyhf-0.6.4.dev8/src/pyhf/tensor/__init__.py pyhf-0.6.4.dev8/src/pyhf/tensor/common.py pyhf-0.6.4.dev8/src/pyhf/tensor/jax_backend.py pyhf-0.6.4.dev8/src/pyhf/tensor/numpy_backend.py pyhf-0.6.4.dev8/src/pyhf/tensor/pytorch_backend.py pyhf-0.6.4.dev8/src/pyhf/tensor/tensorflow_backend.py pyhf-0.6.4.dev8/src/pyhf/utils.py pyhf-0.6.4.dev8/src/pyhf/workspace.py pyhf-0.6.4.dev8/src/pyhf/writexml.py pyhf-0.6.4.dev8/src/pyhf.egg-info/ pyhf-0.6.4.dev8/src/pyhf.egg-info/PKG-INFO pyhf-0.6.4.dev8/src/pyhf.egg-info/SOURCES.txt pyhf-0.6.4.dev8/src/pyhf.egg-info/dependency_links.txt pyhf-0.6.4.dev8/src/pyhf.egg-info/entry_points.txt pyhf-0.6.4.dev8/src/pyhf.egg-info/requires.txt pyhf-0.6.4.dev8/src/pyhf.egg-info/top_level.txt $ python -m zipfile --list dist/*.whl File Name Modified Size pyhf/__init__.py 2021-09-01 03:41:30 6148 pyhf/_version.py 2021-09-08 18:47:12 155 pyhf/compat.py 2021-09-01 03:41:30 4036 pyhf/constraints.py 2021-09-01 03:41:30 10274 pyhf/events.py 2021-09-01 03:41:30 4401 pyhf/mixins.py 2021-09-01 03:41:30 2048 pyhf/patchset.py 2021-09-01 03:41:30 11310 pyhf/pdf.py 2021-09-01 03:41:30 31029 pyhf/probability.py 2021-09-01 03:41:30 9537 pyhf/readxml.py 2021-09-07 03:51:54 13087 pyhf/simplemodels.py 2021-09-01 03:41:30 5764 pyhf/utils.py 2021-09-07 03:51:54 5061 pyhf/workspace.py 2021-09-07 03:51:54 33903 pyhf/writexml.py 2021-09-07 03:51:54 10807 pyhf/cli/__init__.py 2021-09-01 03:41:30 387 pyhf/cli/cli.py 2021-09-01 03:41:30 1299 pyhf/cli/complete.py 2021-08-18 14:06:12 982 pyhf/cli/infer.py 2021-09-01 03:41:30 6605 pyhf/cli/patchset.py 2021-09-01 03:41:30 4318 pyhf/cli/rootio.py 2021-09-01 03:41:30 3072 pyhf/cli/spec.py 2021-09-01 03:41:30 12266 pyhf/contrib/__init__.py 2021-08-18 14:06:12 89 pyhf/contrib/cli.py 2021-09-01 03:41:30 2051 pyhf/contrib/utils.py 2021-09-01 03:41:30 2674 pyhf/contrib/viz/__init__.py 2021-08-18 14:06:12 49 pyhf/contrib/viz/brazil.py 2021-09-01 03:41:30 13476 pyhf/data/citation.bib 2021-09-07 03:51:54 699 pyhf/exceptions/__init__.py 2021-09-01 03:41:30 4784 pyhf/infer/__init__.py 2021-09-01 03:41:30 8239 pyhf/infer/calculators.py 2021-09-01 03:41:30 35522 pyhf/infer/intervals.py 2021-09-01 03:41:30 2647 pyhf/infer/mle.py 2021-09-01 03:41:30 8281 pyhf/infer/test_statistics.py 2021-09-01 03:41:30 15867 pyhf/infer/utils.py 2021-09-01 03:41:30 3336 pyhf/interpolators/__init__.py 2021-09-01 03:41:30 1644 pyhf/interpolators/code0.py 2021-09-01 03:41:30 3847 pyhf/interpolators/code1.py 2021-09-01 03:41:30 4498 pyhf/interpolators/code2.py 2021-09-01 03:41:30 5349 pyhf/interpolators/code4.py 2021-09-01 03:41:30 16006 pyhf/interpolators/code4p.py 2021-09-01 03:41:30 4954 pyhf/modifiers/__init__.py 2021-09-01 03:41:30 7242 pyhf/modifiers/histosys.py 2021-09-01 03:41:30 3290 pyhf/modifiers/lumi.py 2021-09-01 03:41:30 2539 pyhf/modifiers/normfactor.py 2021-09-01 03:41:30 2727 pyhf/modifiers/normsys.py 2021-09-01 03:41:30 3312 pyhf/modifiers/shapefactor.py 2021-09-01 03:41:30 5889 pyhf/modifiers/shapesys.py 2021-09-07 14:57:40 7041 pyhf/modifiers/staterror.py 2021-09-01 03:41:30 5274 pyhf/optimize/__init__.py 2021-09-01 03:41:30 1493 pyhf/optimize/common.py 2021-09-01 03:41:30 5542 pyhf/optimize/mixins.py 2021-09-01 03:41:30 7237 pyhf/optimize/opt_jax.py 2021-09-01 03:41:30 2491 pyhf/optimize/opt_minuit.py 2021-09-01 03:41:30 5357 pyhf/optimize/opt_numpy.py 2021-09-01 03:41:30 1051 pyhf/optimize/opt_pytorch.py 2021-09-01 03:41:30 1353 pyhf/optimize/opt_scipy.py 2021-09-01 03:41:30 3434 pyhf/optimize/opt_tflow.py 2021-09-01 03:41:30 1583 pyhf/parameters/__init__.py 2021-09-01 03:41:30 447 pyhf/parameters/paramsets.py 2021-09-01 03:41:30 2018 pyhf/parameters/paramview.py 2021-09-01 03:41:30 2877 pyhf/parameters/utils.py 2021-09-01 03:41:30 2820 pyhf/schemas/HistFactorySchema.dtd 2021-01-21 19:23:20 6521 pyhf/schemas/1.0.0/defs.json 2021-09-01 03:41:30 13069 pyhf/schemas/1.0.0/jsonpatch.json 2021-08-18 14:06:12 180 pyhf/schemas/1.0.0/measurement.json 2021-08-18 14:06:12 184 pyhf/schemas/1.0.0/model.json 2021-08-18 14:06:12 172 pyhf/schemas/1.0.0/patchset.json 2021-08-18 14:06:12 178 pyhf/schemas/1.0.0/workspace.json 2021-08-18 14:06:12 180 pyhf/tensor/__init__.py 2021-09-01 03:41:30 2041 pyhf/tensor/common.py 2021-09-01 03:41:30 3278 pyhf/tensor/jax_backend.py 2021-09-07 03:51:54 18044 pyhf/tensor/numpy_backend.py 2021-09-02 17:02:22 17054 pyhf/tensor/pytorch_backend.py 2021-09-02 17:02:22 18220 pyhf/tensor/tensorflow_backend.py 2021-09-07 03:51:54 23064 pyhf-0.6.4.dev8.dist-info/LICENSE 2021-09-08 18:47:12 10760 pyhf-0.6.4.dev8.dist-info/METADATA 2021-09-08 18:47:12 21377 pyhf-0.6.4.dev8.dist-info/WHEEL 2021-09-08 18:47:12 92 pyhf-0.6.4.dev8.dist-info/entry_points.txt 2021-09-08 18:47:12 39 pyhf-0.6.4.dev8.dist-info/top_level.txt 2021-09-08 18:47:12 5 pyhf-0.6.4.dev8.dist-info/RECORD 2021-09-08 18:47:12 6457 ```

Given Issue #115, does this mean check-manifest is incompatible with this syntax and to use check-manifest users should manually add / exclude all top level files?

cc @alexander-held

mgedmin commented 3 years ago

This is a bug, or, more precisely, a missing feature: check-manifest currently processes exclude directives (such as prune) and ignores include directives.

I would like to fix it some day.

matthewfeickert commented 3 years ago

This is a bug, or, more precisely, a missing feature: check-manifest currently processes exclude directives (such as prune) and ignores include directives.

Thanks for the very responsive reply @mgedmin — truly appreciate it! This information is helpful, so thanks for clarifying the situation.

I would like to fix it some day.

I assume that this is nontrivial and not something that is an appropriate contribution for a first time contributor to the project. But, if I was selfishly interested in trying to help address this, do you have any advice on where to start working? I should of course also just start looking myself.

mgedmin commented 3 years ago

I assume that this is nontrivial and not something that is an appropriate contribution for a first time contributor to the project. But, if I was selfishly interested in trying to help address this, do you have any advice on where to start working? I should of course also just start looking myself.

I have this IgnoreList class that currently collects all the exclusion rules. It needs to be extended to handle include rules in proper order and renamed to something like RuleList.

Then read_manifest()'s inner helpers need to be changed (and renamed) so they handle all the include directives (there's an XXX comment) and return a RuleList instead of an IgnoreList.

Then I have to figure out how ignore rules specified in check-manifest's configuration interact with MANIFEST's include/exclude rules, order-wise, and maybe update the += statement in check_manifest().

And, of course, unit tests for all of the above. Oh dear, I haven't migrated to pytest yet, that's a shame. Well, the end-of-life of Python 3.5 makes things easier there -- the Windows console issue that was a blocker should no longer be a blocker. Eh, maybe unittest isn't that bad.

The highest risk is that I might be too burned out to review a PR in a timely fashion. (I'm currently failing to review three other PR's that are sitting there in open browser tabs since July.)