Summary

When using the PTB, then filters defined in file noxconfig.py seem to not work as expected.

Reproducing the Issue

In project saas-api-python In file noxconfig.py I added filters ".nox" and "openapi" as I had subdirectories .nox and exasol/saas/client/openapi/ which I wanted to filter as an experiment.

@dataclass(frozen=True)
class Config:
    root: Path = ROOT_DIR
    doc: Path = ROOT_DIR / "doc"
    version_file: Path = ROOT_DIR / "version.py"
    path_filters: Iterable[str] = (
        "dist",
        ".eggs",
        "venv",
        ".nox", 
        "openapi",
    )

When running poetry run nox -s lint pylint still reported files below directory .nox.

Expected Behaviour

Both files below directory .nox and files below directory exasol/saas/client/openapi/ are filtered out.

Actual Behaviour

Depending on the order of the filters in path_filters either only files below only one of the directories were filtered out.

Root Cause (optional)

Probably the nesting of generators in exasol/toolbox/project.py produces unexpected results:

def _deny_filter(files: Iterable[Path], deny_list: Iterable[str]) -> Iterable[Path]:
    for entry in deny_list:
        files = filter(lambda path: entry not in path.parts, files)
    return files

The following modified code is inefficient, but works:

files = list(filter(lambda path: entry not in path.parts, files))

Context

Git Repo https://github.com/exasol/saas-api-python/

System

Desktop:

OS: Linux (Fedora 38)
Python Version 3.11.8

tkilias commented 5 months ago

it seems it has something to do with generator expressions https://docs.python.org/3/library/functions.html#filter, see the note in the description

ckunki commented 5 months ago

An alternative implementation could be

def _matches(file: Path, c: str|Path|re.Pattern):
    if isinstance(c, re.Pattern):
        return c.match(str(file))
    if isinstance(c, Path):
        return str(c) in str(file)
    return c in file.parts

def _accept(file: Path, deny_list: Sequence[str|Path|re.Pattern]) -> Path | None:
    """
    Return ``file`` if none of the specified criteria in Sequence
    ``deny_list`` matches, otherwise return ``None``.
    """
    for criterion in deny_list:
        if _matches(file, criterion):
            return None
    return file

def _deny_filter(files: Iterable[Path], deny_list: Iterable[str]) -> Iterable[Path]:
    return (f for f in files if _accept(f, list(deny_list)))

exasol / python-toolbox

🐞 Path filtering doesn't work #159

Checklist