RNTuple split encoding tests

coveralls commented 1 year ago

Coverage: 62.736%. Remained the same when pulling 7d69dde565aa9615a3abaeb74e62118eaf8b0175 on rntuple_split_encoding into c2d0d776b2f9de0dc0e141bd2d1a1c8543c2d730 on main.

jpivarski commented 1 year ago

I can't fix the problems this is having with pre-commit, nor even reproduce them. When I try to do pre-commit on my computer, I get a different error.

In CI:

build attempt 1...
    => error
    [INFO] Installing environment for https://github.com/PyCQA/isort.
    [INFO] Once installed this environment will be reused.
    [INFO] This may take a few minutes...
    An unexpected error has occurred: CalledProcessError: command: ('/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/bin/python', '-mpip', 'install', '.')
    return code: 1
    stdout:
        Processing /pc/clone/k4obBRQdRaCnKfXwxABFkQ
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
          Getting requirements to build wheel: started
          Getting requirements to build wheel: finished with status 'done'
          Preparing metadata (pyproject.toml): started
          Preparing metadata (pyproject.toml): finished with status 'error'
    stderr:
          error: subprocess-exited-with-error

          × Preparing metadata (pyproject.toml) did not run successfully.
          │ exit code: 1
          ╰─> [17 lines of output]
              Traceback (most recent call last):
                File "/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
                  main()
                File "/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
                  json_out['return_val'] = hook(**hook_input['kwargs'])
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
                  return hook(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/tmp/pip-build-env-rnyfi_qx/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel
                  poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/tmp/pip-build-env-rnyfi_qx/overlay/lib/python3.11/site-packages/poetry/core/factory.py", line 57, in create_poetry
                  raise RuntimeError("The Poetry configuration is invalid:\n" + message)
              RuntimeError: The Poetry configuration is invalid:
                - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

              [end of output]

          note: This error originates from a subprocess, and is likely not a problem with pip.
        error: metadata-generation-failed

        × Encountered error while generating package metadata.
        ╰─> See above for output.

        note: This is an issue with the package mentioned above, not pip.
        hint: See above for details.
    Check the log at /pc/pre-commit.log
build attempt 2...
    => error
    [INFO] Installing environment for https://github.com/PyCQA/isort.
    [INFO] Once installed this environment will be reused.
    [INFO] This may take a few minutes...
    An unexpected error has occurred: CalledProcessError: command: ('/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/bin/python', '-mpip', 'install', '.')
    return code: 1
    stdout:
        Processing /pc/clone/k4obBRQdRaCnKfXwxABFkQ
          Installing build dependencies: started
          Installing build dependencies: finished with status 'done'
          Getting requirements to build wheel: started
          Getting requirements to build wheel: finished with status 'done'
          Preparing metadata (pyproject.toml): started
          Preparing metadata (pyproject.toml): finished with status 'error'
    stderr:
          error: subprocess-exited-with-error

          × Preparing metadata (pyproject.toml) did not run successfully.
          │ exit code: 1
          ╰─> [17 lines of output]
              Traceback (most recent call last):
                File "/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
                  main()
                File "/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
                  json_out['return_val'] = hook(**hook_input['kwargs'])
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/pc/clone/k4obBRQdRaCnKfXwxABFkQ/py_env-python3/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
                  return hook(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/tmp/pip-build-env-4ljolzlg/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 40, in prepare_metadata_for_build_wheel
                  poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/tmp/pip-build-env-4ljolzlg/overlay/lib/python3.11/site-packages/poetry/core/factory.py", line 57, in create_poetry
                  raise RuntimeError("The Poetry configuration is invalid:\n" + message)
              RuntimeError: The Poetry configuration is invalid:
                - [extras.pipfile_deprecated_finder.2] 'pip-shims<=0.3.4' does not match '^[a-zA-Z-_.0-9]+$'

              [end of output]

          note: This error originates from a subprocess, and is likely not a problem with pip.
        error: metadata-generation-failed

        × Encountered error while generating package metadata.
        ╰─> See above for output.

        note: This is an issue with the package mentioned above, not pip.
        hint: See above for details.
    Check the log at /pc/pre-commit.log

On my computer (Python 3.9):

% pre-commit run -a
check for added large files..............................................Passed
check for case conflicts.................................................Passed
check for merge conflicts................................................Passed
check for broken symlinks............................(no files to check)Skipped
check yaml...............................................................Passed
debug statements (python)................................................Passed
fix end of files.........................................................Passed
mixed line ending........................................................Passed
fix requirements.txt.................................(no files to check)Skipped
trim trailing whitespace.................................................Passed
pyupgrade................................................................Passed
black....................................................................Passed
isort....................................................................Passed
setup-cfg-fmt............................................................Passed
mypy 3.8.................................................................Failed
- hook id: mypy
- exit code: 1

src/skhep_testdata/local_files.py:72: error: Self? has no attribute "raise_for_status"  [attr-defined]
src/skhep_testdata/local_files.py:73: error: Self? has no attribute "iter_content"  [attr-defined]
Found 2 errors in 1 file (checked 5 source files)

flake8...................................................................Passed

But what's happening here is that we're checking in new data files with no changes to the code. The last code-change that I see in the codebase (not pre-commit version bumps, reformatting because of dropping Python 3.6, or added type annotations) are from April 16, 2021 (#51). That code is not breaking because it's not getting touched.

I'm going to merge this because we need these new files in order to do Uproot testing. It might mean that pre-commit will be broken in main. I think we should scale back the testing that pre-commit is doing.

eduardo-rodrigues commented 1 year ago

To be checked indeed what changed in pre-commit. Unless we move straight to Ruff and the issue may go away since the hooks aren't all the same as far as I remember? Maybe a more pragmatic approach. WDYT @henryiii?

henryiii commented 1 year ago

The poetry error is due to isort using poetry-core, and poetry-core breaking them in 1.5 and them telling them it was their fault. All older versions of isort will not build from source anymore. We can update it, or move to Ruff, which can't exhibit this problem, as it has no dependencies and doesn't build from source.

MyPy broke due to an update in a dependency I'd expect - maybe pytest. Should be an easy fix.

henryiii commented 1 year ago

Yes, something updates to start using Self from typing_extensions/Python 3.11, and that's only supported for mypy 1.0+. The root problem is pre-commit needs to be autoupdated (more often? At all?).

scikit-hep / scikit-hep-testdata

RNTuple split encoding tests #114