Upgrading jsonschema to 4.18+ to resolve security issues with older dependencies breaks starfish

shachafl commented 1 month ago

Description

To resolve a few security issues with older dependencies, I wish to upgrade jsonschema to 4.18+, but this breaks starfish and throws errors in testing as RefResolver was deprecated (https://python-jsonschema.readthedocs.io/en/latest/referencing/).

Steps/Code to Reproduce

git clone https://github.com/spacetx/starfish.git
cd starfish/
python -m venv .venv3.9
source .venv3.9/bin/activate
make install-dev
pip install -U jsonschema
make fast-test

Expected Results

Something like: ======== 370 passed, 3 skipped, 120 warnings in 42.88s ==================

Actual Results

================= short test summary info ======================= FAILED starfish/core/config/test/test_config.py::test_cache_merfish[enabled-expected0-config0] - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/config/test/test_config.py::test_cache_merfish[disabled-expected1-config1] - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/config/test/test_config.py::test_cache_merfish[limited-expected2-config2] - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_codebook.py::test_codebook_validate - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_experiment.py::test_fov - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_experiment.py::test_nuclei_must_be_present - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_fuzz.py::test_fuzz_experiment - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_field_of_view.py::test_dartfish_example_field_of_view - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_experiment.py::test_version_must_be_semantic - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted. FAILED starfish/core/spacetx_format/test_field_of_view.py::test_too_large - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted. FAILED starfish/core/spacetx_format/test_field_of_view.py::test_dartfish_nuclei_example_field_of_view - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_fov_manifest.py::test_fov_manifest - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_field_of_view.py::test_channel_must_be_present - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted. FAILED starfish/core/spacetx_format/test_experiment.py::test_dartfish_example_experiment - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_fuzz.py::test_fuzz_codebook - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_field_of_view.py::test_round_must_be_present - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted. FAILED starfish/core/spacetx_format/test_experiment.py::test_no_manifest_example_experiment - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_field_of_view.py::test_field_of_view - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_codebook.py::test_codebook - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_codebook.py::test_diagonal_codebook - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_codebook.py::test_diagonal_codebook_full_values - jsonschema.exceptions.RefResolutionError: <urlopen error [Errno 2] No such file or directory: '.../starfish/starfish/spacetx... FAILED starfish/core/spacetx_format/test_codebook.py::test_codebook_missing_channel_raises_validation_error - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted. ======= 22 failed, 348 passed, 3 skipped, 136 warnings in 41.19s ==============

Versions

Linux-5.15.153.1-microsoft-standard-WSL2-x86_64-with-glibc2.35 Python 3.9.18 (main, May 17 2024, 10:28:50) [GCC 11.4.0]

shachafl commented 1 month ago

@joshmoore can you please assist with this issue?

joshmoore commented 1 month ago

Do you have a commit where you have tried using a Registry rather than the RefResolver?

shachafl commented 1 month ago

No. I had difficulties solving the previous bug with jsonschema and was hoping someone more knowledgeable would carry the baton for this one.

spacetx / starfish