[python] CI `mypy` issue as of 2024-08-05

[johnkerl]

To be investigated:

• Example CI fail https://github.com/single-cell-data/TileDB-SOMA/pull/2834/checks (not just that PR)
  • Lint ok on python 3.11 + ubuntu
  • Lint not ok on python 3.11 + macos
  • Maybe related to mypy 1.11.1 just dropped a few days ago -- ? :thinking_face:
• Large-scale post-tag CI fail on 1.13.0rc0: https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10257321585/job/28378154410

On the one hand this might seem to be a minor annoyance; on the other, it's clouding CI just when we're trying to make the TileDB-SOMA 1.13.0 release. Thus we should solve this sooner than later.

See also #2838 (just created) which has significant narrative by @ryan-williams on it.

Some background info (something that's been in place for a year now, I am reading up on now):

• https://github.com/single-cell-data/TileDB-SOMA/blame/1.13.0rc0/.github/workflows/python-ci-single.yml#L45-L49
• https://github.com/single-cell-data/TileDB-SOMA/pull/756

[sc-52471]

[johnkerl]

Re-assigning to @ryan-williams

[ryan-williams]

At a minimum, actions/setup-python is doing something buggy, I filed https://github.com/actions/setup-python/issues/919.

This issue is live on #2853, until we find a work-around, so I'm reopening it.

[ryan-williams]

Issue persists with setup-python cache disabled (and no pre-commit cache found/restored), on both macos and ubuntu. Very strange.

[ryan-williams]

I believe I've root-caused this:

• Our pandas dep (in setup.py) is unpinned, as is our pandas-stub (in .pre-commit-config.yaml), meaning we can end up with Pandas 1.x or 2.x (and likewise for the "stub" types in mypy).
• When type-checking, some types must receive a type-arg in Pandas 2.x (pd.Series[Any], pd.Index[Any]), but must not in Pandas 1.x (pd.Series, pd.Index).
• Newly-created environments would pick up pandas{,-stub}>=2, and type-checking would pass, as HEAD is compatible with Pandas 2.x types.
• However, setup-python is (incorrectly, afaict, per setup-python#919) loading old cached environments containing Pandas 1.x, causing the 2.x-compatible types in the codebase to error, failing some GHAs (as well as local environments using.

I believe #2854 actually fixes it; we can keep an eye on setup-python caching behavior as well.

Here is a little script I wrote that mimics the GHA hashFiles helper (used in setup-python and in some of our actions/cache invocations):

#!/usr/bin/env python
import hashlib
import sys

sha256 = hashlib.sha256()

for path in sys.argv[1:]:
    file_sha256 = hashlib.sha256()
    with open(path, "rb") as f:
        for byte_block in iter(lambda: f.read(4096), b""):
            file_sha256.update(byte_block)
    sha256.update(file_sha256.digest())

print(sha256.hexdigest())