search

gridhead / nvidia-auto-installer-for-fedora-linux

A CLI tool which lets you install proprietary NVIDIA drivers and much more easily on Fedora Linux (32 or above and Rawhide)
https://ask.fedoraproject.org/t/nvidia-auto-installer-for-fedora-linux-now-works-with-fedora-linux-36/22310/
GNU General Public License v3.0
502 stars 28 forks source link

fix(ci): 🐞 pre-commit config checks and pip shim fix #248

Closed onuralpszr closed 1 year ago

onuralpszr commented 1 year ago

Fixes #246

I also found that there was a bug related to pre-commit and after a little bit of search I upgrade all pre-commit tools so now this one not showing up.

Previously error looks like this

[INFO] Installing environment for https://github.com/pycqa/isort.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/home/onuralp/.cache/pre-commit/repo7g5c_v3f/py_env-python3.11/bin/python', '-mpip', 'install', '.')
return code: 1
stdout:
    Processing /home/onuralp/.cache/pre-commit/repo7g5c_v3f
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
      Preparing metadata (pyproject.toml): started
      Preparing metadata (pyproject.toml): finished with status 'error'
stderr:
      error: subprocess-exited-with-error

      × Preparing metadata (pyproject.toml) did not run successfully.
      │ exit code: 1
      ╰─> [17 lines of output]
          Traceback (most recent call last):
            File "/home/onuralp/.cache/pre-commit/repo7g5c_v3f/py_env-python3.11/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
              main()
            File "/home/onuralp/.cache/pre-commit/repo7g5c_v3f/py_env-python3.11/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
              json_out['return_val'] = hook(**hook_input['kwargs'])
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            File "/home/onuralp/.cache/pre-commit/repo7g5c_v3f/py_env-python3.11/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
              return hook(metadata_directory, config_settings)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            File "/tmp/pip-build-env-4tp1yhlj/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 41, in prepare_metadata_for_build_wheel
              poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            File "/tmp/pip-build-env-4tp1yhlj/overlay/lib/python3.11/site-packages/poetry/core/factory.py", line 58, in create_poetry
              raise RuntimeError("The Poetry configuration is invalid:\n" + message)
          RuntimeError: The Poetry configuration is invalid:
            - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

          [end of output]

      note: This error originates from a subprocess, and is likely not a problem with pip.
    error: metadata-generation-failed

    × Encountered error while generating package metadata.
    ╰─> See above for output.

    note: This is an issue with the package mentioned above, not pip.
    hint: See above for details.
Check the log at /home/onuralp/.cache/pre-commit/pre-commit.log
onuralpszr commented 1 year ago

Let's add some real ruff rules and re-check all 

nvautoinstall/interfaces/__init__.py:20:71: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/__init__.py:21:101: E501 Line too long (258 > 100 characters)
nvautoinstall/interfaces/handle_compatibility_check.py:55:17: PLR5501 Use `elif` instead of `else` then `if`, to reduce indentation
nvautoinstall/interfaces/handle_drivers_installation.py:61:130: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_drivers_installation.py:71:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_drivers_installation.py:76:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_prime_support.py:38:9: C901 `__init__` is too complex (11 > 10)
nvautoinstall/interfaces/handle_prime_support.py:38:9: PLR0912 Too many branches (17 > 12)
nvautoinstall/interfaces/handle_prime_support.py:38:9: PLR0915 Too many statements (55 > 50)
nvautoinstall/interfaces/handle_prime_support.py:53:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_prime_support.py:93:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_prime_support.py:98:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_rpmfusion_repositories.py:42:120: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_rpmfusion_repositories.py:55:128: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_rpmfusion_repositories.py:65:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:36:9: C901 `__init__` is too complex (11 > 10)
nvautoinstall/interfaces/install_cuda_support.py:36:9: PLR0912 Too many branches (18 > 12)
nvautoinstall/interfaces/install_cuda_support.py:36:9: PLR0915 Too many statements (55 > 50)
nvautoinstall/interfaces/install_cuda_support.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:79:119: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:83:104: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:86:115: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:91:117: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:96:110: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:101:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:106:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:68:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:73:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:78:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:42:122: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:61:138: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:66:111: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:71:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:81:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:66:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:71:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:76:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:66:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:71:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:76:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_compatibility_check.py:30:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_compatibility_check.py:61:67: PLR2004 Magic value used in comparison, consider replacing 32 with a constant variable
nvautoinstall/operations/handle_cuda_installation.py:28:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_cuda_installation.py:35:138: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_cuda_installation.py:46:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:46:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_cuda_installation.py:50:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:50:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_cuda_installation.py:54:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:54:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_cuda_installation.py:58:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:58:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_drivers_installation.py:27:121: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_drivers_installation.py:34:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_prime_support.py:29:33: ISC003 Explicitly concatenated string should be implicitly concatenated
nvautoinstall/operations/handle_prime_support.py:33:16: BLE001 Do not catch blind exception: `Exception`
nvautoinstall/operations/handle_rpmfusion_repositories.py:28:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_rpmfusion_repositories.py:38:19: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_rpmfusion_repositories.py:38:19: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_rpmfusion_repositories.py:40:125: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_rpmfusion_repositories.py:43:131: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_rpmfusion_repositories.py:46:130: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/install_ffmpeg_support.py:25:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/install_ffmpeg_support.py:25:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/install_video_acceleration.py:25:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/install_video_acceleration.py:25:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/install_vulkan_support.py:25:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/install_vulkan_support.py:25:38: S607 Starting a process with a partial executable path
Found 101 errors (30 fixed, 71 remaining).

Now we can see actual problem of shell usages too. Also we need to handle noqa in a better way

onuralpszr commented 1 year ago

70 errors left 

nvautoinstall/interfaces/__init__.py:20:71: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_compatibility_check.py:55:17: PLR5501 Use `elif` instead of `else` then `if`, to reduce indentation
nvautoinstall/interfaces/handle_drivers_installation.py:61:130: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_drivers_installation.py:71:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_drivers_installation.py:76:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_prime_support.py:38:9: C901 `__init__` is too complex (11 > 10)
nvautoinstall/interfaces/handle_prime_support.py:38:9: PLR0912 Too many branches (17 > 12)
nvautoinstall/interfaces/handle_prime_support.py:38:9: PLR0915 Too many statements (55 > 50)
nvautoinstall/interfaces/handle_prime_support.py:53:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_prime_support.py:93:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_prime_support.py:98:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_rpmfusion_repositories.py:42:120: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_rpmfusion_repositories.py:55:128: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/handle_rpmfusion_repositories.py:65:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:36:9: C901 `__init__` is too complex (11 > 10)
nvautoinstall/interfaces/install_cuda_support.py:36:9: PLR0912 Too many branches (18 > 12)
nvautoinstall/interfaces/install_cuda_support.py:36:9: PLR0915 Too many statements (55 > 50)
nvautoinstall/interfaces/install_cuda_support.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:79:119: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:83:104: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:86:115: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:91:117: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:96:110: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:101:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_cuda_support.py:106:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:68:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:73:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_ffmpeg_support.py:78:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:42:122: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:61:138: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:66:111: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:71:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_nvidia_repositories.py:81:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:66:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:71:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_video_acceleration.py:76:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:51:113: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:66:107: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:71:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/interfaces/install_vulkan_support.py:76:105: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_compatibility_check.py:30:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_compatibility_check.py:61:67: PLR2004 Magic value used in comparison, consider replacing 32 with a constant variable
nvautoinstall/operations/handle_cuda_installation.py:28:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_cuda_installation.py:35:138: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_cuda_installation.py:46:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:46:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_cuda_installation.py:50:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:50:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_cuda_installation.py:54:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:54:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_cuda_installation.py:58:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_cuda_installation.py:58:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_drivers_installation.py:27:121: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_drivers_installation.py:34:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_prime_support.py:29:33: ISC003 Explicitly concatenated string should be implicitly concatenated
nvautoinstall/operations/handle_prime_support.py:33:16: BLE001 Do not catch blind exception: `Exception`
nvautoinstall/operations/handle_rpmfusion_repositories.py:28:21: S602 `subprocess` call with `shell=True` identified, security issue
nvautoinstall/operations/handle_rpmfusion_repositories.py:38:19: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/handle_rpmfusion_repositories.py:38:19: S607 Starting a process with a partial executable path
nvautoinstall/operations/handle_rpmfusion_repositories.py:40:125: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_rpmfusion_repositories.py:43:131: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/handle_rpmfusion_repositories.py:46:130: PGH004 Use specific rule codes when using `noqa`
nvautoinstall/operations/install_ffmpeg_support.py:25:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/install_ffmpeg_support.py:25:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/install_video_acceleration.py:25:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/install_video_acceleration.py:25:38: S607 Starting a process with a partial executable path
nvautoinstall/operations/install_vulkan_support.py:25:38: S605 Starting a process with a shell: seems safe, but may be changed in the future; consider rewriting without `shell`
nvautoinstall/operations/install_vulkan_support.py:25:38: S607 Starting a process with a partial executable path
Found 70 errors.
onuralpszr commented 1 year ago

I also ignore both of this for know but we need better exception handle and change that string

nvautoinstall/operations/handle_prime_support.py:29:33: ISC003 Explicitly concatenated string should be implicitly concatenated nvautoinstall/operations/handle_prime_support.py:33:16: BLE001 Do not catch blind exception: Exception

onuralpszr commented 1 year ago

These are the ignored list


# Ignores start with S means subprocess usage error catch
#
extend-ignore = ["S607","S605","S602","C901","PLR0912","PLR0915","PLR5501","PLR2004","BLE001","ISC003"]

For now pre-commit passes I also used experimental black string processing ability to handle those long lines. It looks okay but may need a tiny test other than me as well.

onuralpszr commented 1 year ago

Last status and isort removed yes because ruff handles that too.

check for added large files..............................................Passed
check toml...............................................................Passed
check yaml...............................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
pyupgrade................................................................Passed
ruff.....................................................................Passed
black....................................................................Passed
onuralpszr commented 1 year ago

cc @t0xic0der we can also remove this sections from toml file because we have different rule with ruff or we can do more adjustment we should stick to using one 

[tool.pytest.ini_options]
addopts = "--black --isort --flake8 --cov=nvautoinstall --cov-report=term-missing"
flake8-max-line-length = 100
filterwarnings = ["ignore::DeprecationWarning:flake8:254"]

[tool.isort]
line_length = 100
profile = "black"
onuralpszr commented 1 year ago 
>>> import nvautoinstall
>>> nvautoinstall.__all__
('decoration', 'operations', 'interfaces', 'CheckSuperuserPermissions', 'HandleCompatibilityCheck', 'HandleDriverInstallation', 'HandlePrimeSupport', 'HandleRPMFusionRepositories', 'InstallCudaSupport', 'InstallEverything', 'InstallFfmpegSupport', 'InstallNvidiaRepositories', 'InstallVideoAcceleration', 'InstallVulkanSupport')
>>>
onuralpszr commented 1 year ago

cc @t0xic0der all fixed and conflicts are resolved as well. CI also passed too (our check CI gonna be replace with pre-commit bot so this is okay)