Make ruff much stricter

[jherland]

Re-configure ruff to enable as many checks as feasible, and fix various issues throughout our codebase. The second PR in a series of 3.

Commits:

• Re-configure ruff to be much stricter
• Fix docstring issues
• Fix unused method arguments
• Fix set(<generator>) -> {<set comprehension>}
• Fix variable shadowing Python builtin
• Fix unused function arguments
• Fix assert False -> pytest.fail(...)
• Fix subprocess problems
• Fix magic values used in comparisons
• Fix use of non-tuples in @pytest.mark.parametrize
• Fix ambiguous character in comment
• Fix unnecessary use of .keys()
• Fix unnecessary use of str()
• Fix set([...]) -> {...}
• Fix unnecessary None argument to .get()
• Use ternary operator instead of if-else block
• Use @pytest.fixture() over @pytest.fixture
• Fix unecessary use of _ in parametrized args
• Use Path.* methods where available
• Avoid extraneous parentheses
• Use yield from instead of for-loop with yield inside
• Consolidate if-conditions
• Use [foo, *bars] instead of [foo] + bars
• Use elif instead of else + if where possible
• Ensure bool flag arguments to functions are always named
• Merge multiple comparisons
• Rename UnparseablePathException -> UnparseablePathError
• Ignore remaining/intentional violations of N818
• Fix PTH201: Do not pass the current directory explicitly to Path()

[jherland]

I really like how you logically structured the problem making ruff strict - each commit tackling a different type of error.

Thanks! 🙏🏻

I think we should avoid # noqa comment as much as possible, or explain why we use it, but this can be a subject of another PR.

Agreed. My philosophy here is roughly this:

• Enable all ruff checks by default, with the following exceptions:
• If there is a ruff check that we don't want, or that ruff itself discourages, then disable it in pyproject.toml with a comment explaining why.
• If there is a ruff check that we would like to follow, but that (a) requires a bigger effort to fix, or (b) touches a lot of our code, then disable it in pyproject.toml for now, and explain that we will get around to it later.
• Otherwise, add a # noqa comment for the issues that we will either get around to later, or that we find "acceptable" to live with indefinitely.

With these pytlint -> ruff PRs, we've gone from ~30 # noqa and # pylint comments before to ~60 # noqa comments after these PRs. Most of this is because ruff simply has more checks to push our code into a better shape, overall.

An important goal is of course to minimize the number of # noqa comments, and to that end, I look very critically at # noqa comments that I encounter while refactoring code. But trying to fix them all in this PR is not feasible. As long as there is a general trend towards decreasing these, I'm happy.