search

hackforla / peopledepot

A project to setup a datastore for people and projects at HackforLA. The link below takes you to the code documentation
https://hackforla.github.io/peopledepot/
GNU General Public License v2.0
6 stars 26 forks source link

flake8 plugin flake8-use-pathlib fails to load #247

Closed fyliu closed 8 months ago

fyliu commented 8 months ago

Overview

We need to fix the issue of flake8-use-pathlib plugin failing to load for some people, so they can run pre-commit locally to check their code before committing. @freaky4wrld and @ethanstrominger both encountered this problem on M1 mac.

Action Items

Resources/Instructions

Error message from @freaky4wrld

click to expand ``` Operations to perform: Apply all migrations: admin, auth, contenttypes, core, data, sessions Running migrations: No migrations to apply. + set +x + ./scripts/lint.sh + pre-commit run --all-files --show-diff-on-failure check python ast.........................................................Passed check yaml...............................................................Passed check json...............................................................Passed check toml...........................................(no files to check)Skipped check that executables have shebangs.....................................Passed check that scripts with shebangs are executable..........................Passed check for merge conflicts................................................Passed check for added large files..............................................Passed detect private key.......................................................Passed check for case conflicts.................................................Passed check docstring is first.................................................Passed check builtin type constructor use.......................................Passed debug statements (python)................................................Passed fix requirements.txt.....................................................Passed fix python encoding pragma...............................................Passed fix utf-8 byte order marker..............................................Passed python tests naming......................................................Passed mixed line ending........................................................Passed trim trailing whitespace.................................................Passed check that executables have shebangs.....................................Passed fix end of files.........................................................Failed - hook id: end-of-file-fixer - exit code: 1 - files were modified by this hook Fixing app/core/tests/test_models.py Hadolint.................................................................Passed pyupgrade................................................................Passed django-upgrade...........................................................Passed black....................................................................Failed - hook id: black - files were modified by this hook reformatted app/core/models.py All done! :sparkles: :cake: :sparkles: 1 file reformatted, 25 files left unchanged. blacken-docs.............................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 Traceback (most recent call last): File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/finder.py", line 291, in _load_plugin obj = plugin.entry_point.load() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/metadata/__init__.py", line 205, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/homebrew/Cellar/python@3.12/3.12.1_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 994, in exec_module File "", line 488, in _call_with_frames_removed File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8_use_pathlib/__init__.py", line 1, in from .checker import PathlibChecker File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8_use_pathlib/checker.py", line 6, in import pkg_resources ModuleNotFoundError: No module named 'pkg_resources' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/bin/flake8", line 8, in sys.exit(main()) ^^^^^^ File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/main/cli.py", line 23, in main app.run(argv) File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py", line 198, in run self._run(argv) File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py", line 186, in _run self.initialize(argv) File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/main/application.py", line 165, in initialize self.plugins, self.options = parse_args(argv) ^^^^^^^^^^^^^^^^ File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/options/parse_args.py", line 42, in parse_args plugins = finder.load_plugins(raw_plugins, plugin_opts) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/finder.py", line 365, in load_plugins return _classify_plugins(_import_plugins(plugins, opts), opts) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/finder.py", line 307, in _import_plugins return [_load_plugin(p) for p in plugins] ^^^^^^^^^^^^^^^ File "/Users/freaky4wrld/.cache/pre-commit/repoanuyys52/py_env-python3.12/lib/python3.12/site-packages/flake8/plugins/finder.py", line 293, in _load_plugin raise FailedToLoadPlugin(plugin.package, e) flake8.exceptions.FailedToLoadPlugin: Flake8 failed to load plugin "flake8-use-pathlib" due to No module named 'pkg_resources'. isort (python)...........................................................Failed - hook id: isort - files were modified by this hook Fixing /Users/freaky4wrld/hfla/peopledepot/app/core/tests/conftest.py Skipped 24 files bandit...................................................................Passed shellcheck...............................................................Passed sync-pre-commit-deps.....................................................Passed Check django migrations..................................................Passed pre-commit hook(s) made changes. If you are seeing this message in CI, reproduce locally with: `pre-commit run --all-files`. To run `pre-commit` as part of git workflow, use `pre-commit install`. All changes made by hooks: diff --git a/app/core/migrations/max_migration.txt b/app/core/migrations/max_migration.txt index 38228e2..67387ae 100644 --- a/app/core/migrations/max_migration.txt +++ b/app/core/migrations/max_migration.txt @@ -1 +1 @@ -0020_rename_is_sponsor_sponsorpartner_is_org_partner_and_more +0021_sdg diff --git a/app/core/models.py b/app/core/models.py index 1304487..f5c9b2b 100644 --- a/app/core/models.py +++ b/app/core/models.py @@ -314,3 +314,18 @@ class StackElementType(AbstractBaseModel): def __str__(self): return f"{self.name}" + + +class Sdg(AbstractBaseModel): + """ + Dictionary for SDG's + """ + + name = models.CharField(max_length=255) + description = models.TextField(blank=True) + image = models.URLField(blank=True) + + # PK of this model is the Foreignkey for sdg_target_indicator + + def __str__(self): + return f"{self.name}" diff --git a/app/core/tests/conftest.py b/app/core/tests/conftest.py index 3fbe27f..887ae6e 100644 --- a/app/core/tests/conftest.py +++ b/app/core/tests/conftest.py @@ -9,6 +9,7 @@ from ..models import PermissionType from ..models import PracticeArea from ..models import ProgramArea from ..models import Project +from ..models import Sdg from ..models import Skill from ..models import SponsorPartner from ..models import StackElementType @@ -133,3 +134,8 @@ def permission_type2(): @pytest.fixture def stack_element_type(): return StackElementType.objects.create(name="Test Stack Element Type") + + +@pytest.fixture +def sdg(): + return Sdg.objects.create(name="Test Sdg") diff --git a/app/core/tests/test_models.py b/app/core/tests/test_models.py index afca7ea..18024da 100644 --- a/app/core/tests/test_models.py +++ b/app/core/tests/test_models.py @@ -66,3 +66,7 @@ def test_permission_type2(permission_type2): def test_stack_element_type(stack_element_type): assert str(stack_element_type) == "Test Stack Element Type" + + +def test_sdg(sdg): + assert str(sdg) == "Test Sdg" ```

chatgpt suggestion from @freaky4wrld

click to expand ``` It seems like you have run some pre-commit hooks that automatically format and lint your code. The output shows changes made by various hooks, including black, isort, and flake8. Additionally, there is a module import issue related to flake8-use-pathlib, and it appears that pkg_resources is not available. Here are some suggestions to address the issues: 1. Resolve ModuleNotFoundError for pkg_resources: It looks like there is an issue with the flake8-use-pathlib plugin. You can try to install the pkg_resources package, which is part of setuptools: bash Copy code pip install setuptools This should resolve the ModuleNotFoundError. If the issue persists, you may want to review your flake8 and related plugins to ensure compatibility with your Python version and dependencies. 2. Resolve Black Formatting Issue: The black formatter seems to have modified the formatting of your code. If the changes made by black are acceptable, you can proceed. If you want to review and potentially adjust the changes, you may need to manually inspect the modified files and make any necessary adjustments. 3. Resolve Flake8 Linting Issue: The flake8 linting failed with an exit code of 1. It seems there's an issue with the flake8-use-pathlib plugin. You might want to either fix the plugin or exclude it from the linting process temporarily to proceed with other linting checks. 4. Check for Other Pre-commit Hooks: Inspect the output for any other failing hooks, and address them accordingly. The goal is to ensure that your code passes all linting and formatting checks before committing. 5. Update and Re-run Hooks: After making adjustments, run the pre-commit hooks again: bash Copy code pre-commit run --all-files --show-diff-on-failure This will run all pre-commit hooks on your files and show the diff for any changes made. Inspect the changes and ensure that they are acceptable. 6. Update pre-commit: Ensure that you are using the latest version of pre-commit: bash Copy code pip install --upgrade pre-commit After updating, you can run the hooks again. These steps should help you address the issues raised by the pre-commit hooks. If you encounter further issues or have specific questions about a particular hook, feel free to ask for more assistance. ```

flake8-use-pathlib repo

fyliu commented 8 months ago

I found a potential solution for the pathlib plugin.

The issue has the same sort of message. The fix was adding setuptools to the dependencies list like chatgpt suggested.

fyliu commented 8 months ago

The other thing that can be one is to switch to ruff, which implements a lot of the checks from other linters and is also a drop-in replacement for black. https://gitlab.com/RoPP/flake8-use-pathlib/-/issues/9

I've tried out ruff for this project before and it was very easy and worked well and fast. This is the easiest path rather than waiting for the plugin author to apply the fix.

fyliu commented 8 months ago

@ethanstrominger installed pre-commit using brew, which is different than how I installed it using pipx. I wonder if the different methods create different environments where one contains setuptools and the other one doesn't. It's just a guess.

fyliu commented 8 months ago

from slack session with @freaky4wrld , we found out that macOS does not come with setuptools installed while linux and wsl seem to have it.

Test by running this command within a python shell import pkg_resources and press enter If there's no error messages, then the system has setuptools

freaky4wrld commented 8 months ago

from slack session with @freaky4wrld , we found out that macOS does not come with setuptools installed while linux and wsl seem to have it.

Test by running this command within a python shell import pkg_resources and press enter If there's no error messages, then the system has setuptools

Running the command import pkg_resources in the python shell shows the warning that pkg_resources has been deprecated

Screenshot of the deprecation image

If there's a specific version of pipx to be used for the project

Also on trying to run ./scripts/precommit-check.sh in the local environment I guess the same error as @ethanstrominger occurred

The pre-commit log ### version information ``` pre-commit version: 3.6.0 git --version: git version 2.39.3 (Apple Git-145) sys.version: 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] sys.executable: /Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/bin/python os.name: posix sys.platform: darwin ``` ### error information ``` An unexpected error has occurred: CalledProcessError: command: ('/Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd/py_env-python3.12/bin/python', '-mpip', 'install', '.') return code: 1 stdout: Processing /Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Building wheels for collected packages: hadolint_py Building wheel for hadolint_py (pyproject.toml): started Building wheel for hadolint_py (pyproject.toml): finished with status 'error' Failed to build hadolint_py stderr: error: subprocess-exited-with-error × Building wheel for hadolint_py (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [5 lines of output] running bdist_wheel running build running build_py running fetch_binaries error: [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for hadolint_py ERROR: Could not build wheels for hadolint_py, which is required to install pyproject.toml-based projects ``` ``` Traceback (most recent call last): File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/error_handler.py", line 73, in error_handler yield File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/main.py", line 414, in main return run(args.config, store, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/commands/run.py", line 442, in run install_hook_envs(to_install, store) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/repository.py", line 238, in install_hook_envs _hook_install(hook) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/repository.py", line 94, in _hook_install lang.install_environment( File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/languages/python.py", line 214, in install_environment lang_base.setup_cmd(prefix, install_cmd) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/lang_base.py", line 86, in setup_cmd cmd_output_b(*cmd, cwd=prefix.prefix_dir, **kwargs) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/util.py", line 111, in cmd_output_b raise CalledProcessError(returncode, cmd, stdout_b, stderr_b) pre_commit.util.CalledProcessError: command: ('/Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd/py_env-python3.12/bin/python', '-mpip', 'install', '.') return code: 1 stdout: Processing /Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Building wheels for collected packages: hadolint_py Building wheel for hadolint_py (pyproject.toml): started Building wheel for hadolint_py (pyproject.toml): finished with status 'error' Failed to build hadolint_py stderr: error: subprocess-exited-with-error × Building wheel for hadolint_py (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [5 lines of output] running bdist_wheel running build running build_py running fetch_binaries error: [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for hadolint_py ERROR: Could not build wheels for hadolint_py, which is required to install pyproject.toml-based projects ``` ### version information ``` pre-commit version: 3.6.0 git --version: git version 2.39.3 (Apple Git-145) sys.version: 3.12.0 (v3.12.0:0fb18b02c8, Oct 2 2023, 09:45:56) [Clang 13.0.0 (clang-1300.0.29.30)] sys.executable: /Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/bin/python os.name: posix sys.platform: darwin ``` ### error information ``` An unexpected error has occurred: CalledProcessError: command: ('/Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd/py_env-python3.12/bin/python', '-mpip', 'install', '.') return code: 1 stdout: Processing /Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Building wheels for collected packages: hadolint_py Building wheel for hadolint_py (pyproject.toml): started Building wheel for hadolint_py (pyproject.toml): finished with status 'error' Failed to build hadolint_py stderr: error: subprocess-exited-with-error × Building wheel for hadolint_py (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [5 lines of output] running bdist_wheel running build running build_py running fetch_binaries error: [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for hadolint_py ERROR: Could not build wheels for hadolint_py, which is required to install pyproject.toml-based projects ``` ``` Traceback (most recent call last): File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/error_handler.py", line 73, in error_handler yield File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/main.py", line 414, in main return run(args.config, store, args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/commands/run.py", line 442, in run install_hook_envs(to_install, store) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/repository.py", line 238, in install_hook_envs _hook_install(hook) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/repository.py", line 94, in _hook_install lang.install_environment( File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/languages/python.py", line 214, in install_environment lang_base.setup_cmd(prefix, install_cmd) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/lang_base.py", line 86, in setup_cmd cmd_output_b(*cmd, cwd=prefix.prefix_dir, **kwargs) File "/Users/freaky4wrld/Library/Application Support/pipx/venvs/pre-commit/lib/python3.12/site-packages/pre_commit/util.py", line 111, in cmd_output_b raise CalledProcessError(returncode, cmd, stdout_b, stderr_b) pre_commit.util.CalledProcessError: command: ('/Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd/py_env-python3.12/bin/python', '-mpip', 'install', '.') return code: 1 stdout: Processing /Users/freaky4wrld/.cache/pre-commit/repoj1llzxyd Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Building wheels for collected packages: hadolint_py Building wheel for hadolint_py (pyproject.toml): started Building wheel for hadolint_py (pyproject.toml): finished with status 'error' Failed to build hadolint_py stderr: error: subprocess-exited-with-error × Building wheel for hadolint_py (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [5 lines of output] running bdist_wheel running build running build_py running fetch_binaries error: [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for hadolint_py ERROR: Could not build wheels for hadolint_py, which is required to install pyproject.toml-based projects ```
fyliu commented 8 months ago

Okay, in the interest of coding progress. Let's do workarounds for macOS.

  1. flake8-use-pathlib
    • disable flake8-use-pathlib
    • set up ruff and select the PTH checks
  2. hadolint
    • skip the check when running git and pre-commit 
      SKIP=hadolint git commit
SKIP=hadolint pre-commit run --all-files
    • or setup aliases (paste into ~/.bashrc) 
      alias git='SKIP=hadolint git'
alias pre-commit='SKIP=hadolint pre-commit'

      then use git and pre-commit normally

fyliu commented 8 months ago

Looks like there's no need to do the steps above to skip hadolint for macOS. Just installing the certificates in python will make the messages go away.

Python keeps its own set of certs rather than use the system ones in macOS.

Discussion on python fix reveals it's shown to users at python install time, which nobody notices. Clear instructions on fix options.