Instagram / Fixit

Advanced Python linting framework with auto-fixes and hierarchical configuration that makes it easy to write custom in-repo lint rules.
https://fixit.rtfd.io/en/latest/
Other
666 stars 62 forks source link

Better handling around child process failures #381

Closed amyreese closed 10 months ago

amyreese commented 1 year ago

When running fixit with multiprocessing, failures can result in broken child processes that abort linting rather than generating usable error messages.

Add from missing_package import something to an enabled lint rule, and then run lint with multiple source files:

$ git diff
diff --git a/src/fixit/rules/use_fstring.py b/src/fixit/rules/use_fstring.py
index b56ebcf..223716c 100644
--- a/src/fixit/rules/use_fstring.py
+++ b/src/fixit/rules/use_fstring.py
@@ -10,6 +10,7 @@ import libcst as cst
 import libcst.matchers as m

 from fixit import Invalid, LintRule, Valid
+from frob import bar

 USE_FSTRING_SIMPLE_EXPRESSION_MAX_LENGTH = 30
$ hatch env run -- fixit lint src/fixit/*.py
concurrent.futures.process._RemoteTraceback:
'''
Traceback (most recent call last):
  File "/Users/amethyst/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/process.py", line 419, in wait_result_broken_or_wakeup
    result_item = result_reader.recv()
                  ^^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/.pyenv/versions/3.11.4/lib/python3.11/multiprocessing/connection.py", line 250, in recv
    return _ForkingPickler.loads(buf.getbuffer())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: CollectionError.__init__() missing 1 required positional argument: 'rule'
'''

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/bin/fixit", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/workspace/Fixit/src/fixit/cli.py", line 122, in lint
    for result in fixit_paths(paths, options=options):
  File "/Users/amethyst/workspace/Fixit/src/fixit/api.py", line 199, in fixit_paths
    for _, results in trailrunner.run_iter(expanded_paths, fn):
  File "/Users/amethyst/Library/Application Support/hatch/env/virtual/fixit/OQ5NT6ov/fixit/lib/python3.11/site-packages/trailrunner/core.py", line 220, in run_iter
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/Users/amethyst/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/amethyst/.pyenv/versions/3.11.4/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.