--output-file with end of argument (--) worked in 3.7.9 but no longer works in 6.0.0

[paisleyrob]

how did you install flake8?

$ cat requirements_hash.txt
GitPython==3.1.29 --hash=sha256:41eea0deec2deea139b459ac03656f0dd28fc4a3387240ec1d3c259a2c47850f2
PyYAML==6.0 --hash=sha256:68fb519c14306fec9720a2a5b45bc9f0c8d1b9c72adf45c37baedfcd949c35a23
bandit==1.7.4 --hash=sha256:412d3f259dab4077d0e7f0c11f50f650cc7d10db905d98f6520a95a18049658a4
coverage==7.0.0 --hash=sha256:b8f7cd942dda3795fc9eadf303cc53a422ac057e3b70c2ad6d4276ec6a83a5415
entrypoints==0.4 --hash=sha256:f174b5ff827504fd3cd97cc3f8649f3693f51538c7e4bdf3ef002c8429d42f9f6
flake8-bandit==4.1.1 --hash=sha256:4c8a53eb48f23d4ef1e59293657181a3c989d0077c9952717e98a0eace43e06d7
flake8-junit-report==2.1.0 --hash=sha256:f8890c1ebe0acb516fefacddec4b802bca9f89bb07db933e4ee3cd11ceaa1e8b8
flake8-polyfill==1.0.2 --hash=sha256:12be6a34ee3ab795b19ca73505e7b55826d5f6ad7230d31b18e106400169b9e99
flake8==6.0.0 --hash=sha256:3833794e27ff64ea4e9cf5d410082a8b97ff1a06c16aa3d2027339cd0f1195c710
gitdb==4.0.10 --hash=sha256:c286cf298426064079ed96a9e4a9d39e7f3e9bf15ba60701e95f5492f28415c711
mccabe==0.7.0 --hash=sha256:6c2d30ab6be0e4a46919781807b4f0d834ebdd6c6e3dca0bda5a15f863427b6e12
pbr==5.11.0 --hash=sha256:db2317ff07c84c4c63648c9064a79fe9d9f5c7ce85a9099d4b6258b3db83225a13
pycodestyle==2.10.0 --hash=sha256:8a4eaf0d0495c7395bdab3589ac2db602797d76207242c17d47018681570661014
pyflakes==3.0.1 --hash=sha256:ec55bf7fe21fff7f1ad2f7da62363d749e2a470500eab1b555334b67aa1ef8cf15
six==1.16.0 --hash=sha256:8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e025416
smmap==5.0.0 --hash=sha256:2aba19d6a040e78d8b09de5c57e96207b09ed71d8e55ce0959eeee6c8e190d9417
stevedore==4.1.1 --hash=sha256:aa6436565c069b2946fe4ebff07f5041e0c8bf18c7376dd29edf80cf7d524e4e18
typing-extensions==4.4.0 --hash=sha256:16fa4864408f655d35ec496218b85f79b3437c829e93320c7c9215ccfd92489e
$ PYTHONDONTWRITEBYTECODE=1 PYTHONHASHSEED=0 SOURCE_DATE_EPOCH=946702800 pip install --no-build-isolation --no-cache-dir --no-compile --require-hashes --requirement requirements_hash.txt

# Note: identical behavior with `pip install flake8` as the installation method.

unmodified output of flake8 --bug-report

{
  "platform": {
    "python_implementation": "CPython",
    "python_version": "3.11.1",
    "system": "Linux"
  },
  "plugins": [
    {
      "plugin": "flake8-bandit",
      "version": "4.1.1"
    },
    {
      "plugin": "mccabe",
      "version": "0.7.0"
    },
    {
      "plugin": "pycodestyle",
      "version": "2.10.0"
    },
    {
      "plugin": "pyflakes",
      "version": "3.0.1"
    }
  ],
  "version": "6.0.0"
}

describe the problem

I've been using flake8 in a CI environment called with something like:

find . -name '*.py' -print0 | xargs -0 flake --output-file output.txt --tee --

When upgrading to v6.0.0 I got errors from the above command. It appears to be some interaction between --output-file and --tee.

This sequence sums up the issue pretty well. If you use the -- which indicates no further arguments it fails to operate properly. I expected the last two commands to behave the same.

$ touch a
$ flake8 ./a
$ flake8 --output-file output.txt ./a
$ flake8 --output-file output.txt -- ./a
Unable to find qualified name for module: output.txt
--output-file:0:1: E902 FileNotFoundError: [Errno 2] No such file or directory: '--output-file'
$

[paisleyrob]

In the flake8 source options/parse_args.py:

args0, rest = prelim_parser.parse_known_args(argv)
# XXX (ericvw): Special case "forwarding" the output file option so
# that it can be reparsed again for the BaseFormatter.filename.
if args0.output_file:
    rest.extend(("--output-file", args0.output_file))

This basically captures the --output-file filename section for use locally, and re-appends it to the argument list so that it gets parsed later on by the main argument parser. Unfortunately, this puts it after the -- so the main argument parser sees it as a filename.

[asottile]

do you need -- ? I'm not sure that's documented and may have only worked by accident

[paisleyrob]

It's used to ensure that later on filenames that start with -- aren't accidentally used as options.

For example: rm -i * works perfectly to ask about each file unless one of those files is named -f. It's a very common idiom. Here's it's reference on Wikipedia: https://en.wikipedia.org/wiki/Command-line_interface#Arguments (It's down a bit from that section)

Two hyphen-minus characters without following letters (--) may indicate that the remaining arguments should not be treated as options, which is useful for example if a file name itself begins with a hyphen, or if further arguments are meant for an inner command (e.g., sudo).

[asottile]

do you have files starting with -? I'm well aware of the point of -- but I don't think you need it

[paisleyrob]

We have general purpose CI scripts that apply across projects company-wide. We harden those scripts against what projects might do to reduce the support burden.

As an aside: python3 -- --perfectly-valid-file-name.py works.

[asottile]

again, I'm well aware of the use of -- -- my assertion is that you don't need it and that you're defending against an impossible situation

[paisleyrob]

Will a Pull Request that fixes the issue be accepted?

[asottile]

I also can't reproduce your findings in 5.0.4 -- it also failed there too so there must be something you're not showing

[asottile]

I also kinda want to remove --output-file and --tee since you can just use > and / or tee directly

[paisleyrob]

I also can't reproduce your findings in 5.0.4 -- it also failed there too so there must be something you're not showing

My mistake, the previous CI was using 3.7.9

[paisleyrob]

I also kinda want to remove --output-file and --tee since you can just use > and / or tee directly

Removing the options seem fine to me, though it triggers a major version change. :shrug:

[sigmavirus24]

I also kinda want to remove --output-file and --tee since you can just use > and / or tee directly

I believe the justification was "Windows" and tox but with WSL and other options, I'm less and less worried about this working in cmd.exe etc. If context is worth anything, --output-file came first primarily because of tox and wanting to have a way to store output in a file since > doesn't work in tox. Then people wanted both to use that and see the output so --tee was introduced as a way to allow for that since it seemed reasonable.

I haven't ever needed flake8 in tox to put things in a file, but perhaps tox no longer has that limitation. :shrug:

[asottile]

cmd.exe has > so it was maybe just tox (which if I need shell things I'd usually just to {toxinidir}/program and put the more complicated logic there 🤷