FynnBe
commented
1 month ago this reminds me that we should probably add the conda env the test was run in (output of conda env export --from-history) to the validation summary.
Should I just use the dev env?
yeah, for most pytorch and onnx models the dev env works (any env with an up-to-date pytorch/onnxruntime package gets you very far for their great backwardcompatibility...)
there is another extra bioimageio.core[tensorflow] that install tensorflow and keras. They are not included in the current dev env.
Getting the following issue. Noticed that it's because of the environment is not configured correctly, is there a universal conda env which can handle not all but most of the models? Should I just use the dev env?
cc @FynnBe
See error log
``` testing affable-shark... 2024-09-12 17:15:13.365 | INFO | bioimageio.spec._internal.io_utils:open_bioimageio_yaml:112 - loading affable-shark from https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/rdf.yaml Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/rdf.yaml' to file '/home/jovyan/.cache/bioimageio/c1d5091838f82f59738702f2d8a938ed-rdf.yaml'. 100%|█████████████████████████████████████| 4.22k/4.22k [00:00<00:00, 8.48MB/s] Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/zero_mean_unit_variance.ijm' to file '/home/jovyan/.cache/bioimageio/3c123c5162a87c48333c80f7389e99a7-zero_mean_unit_variance.ijm'. 100%|█████████████████████████████████████████| 845/845 [00:00<00:00, 1.29MB/s] computing SHA256 of 3c123c5162a87c48333c80f7389e99a7-zero_mean_unit_variance.ijm Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/documentation.md' to file '/home/jovyan/.cache/bioimageio/8c032152668b4aaa68f99fb56c067631-documentation.md'. 100%|█████████████████████████████████████| 1.57k/1.57k [00:00<00:00, 3.45MB/s] SHA256 hash of downloaded file: 9b8330e6fae8403efc7a44bb7365186c14fb6222b608791baab7a4e0352dbdcd Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future. Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/test_input_0.npy' to file '/home/jovyan/.cache/bioimageio/93e503596d7d0bbf5b3937cf1a80402b-test_input_0.npy'. 100%|████████████████████████████████████████| 262k/262k [00:00<00:00, 830MB/s] computing SHA256 of 93e503596d7d0bbf5b3937cf1a80402b-test_input_0.npy (result: c Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/sample_input_0.tif' to file '/home/jovyan/.cache/bioimageio/baeb9b3639733aa6ad5f8305149a3585-sample_input_0.tif'. 100%|██████████████████████████████████████| 65.8k/65.8k [00:00<00:00, 260MB/s] computing SHA256 of baeb9b3639733aa6ad5f8305149a3585-sample_input_0.tif (result: Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/test_output_0.npy' to file '/home/jovyan/.cache/bioimageio/858a63339bdb8f80ff310e077a5079bb-test_output_0.npy'. 100%|███████████████████████████████████████| 524k/524k [00:00<00:00, 2.06GB/s] computing SHA256 of 858a63339bdb8f80ff310e077a5079bb-test_output_0.npy (result: Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/sample_output_0.tif' to file '/home/jovyan/.cache/bioimageio/d22d18e9c3592839985cac2534ac4617-sample_output_0.tif'. 100%|███████████████████████████████████████| 525k/525k [00:00<00:00, 1.51GB/s] computing SHA256 of d22d18e9c3592839985cac2534ac4617-sample_output_0.tif (result Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights.onnx' to file '/home/jovyan/.cache/bioimageio/4f6534376602b6e094e1895b93c4ee74-weights.onnx'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 370GB/s] computing SHA256 of 4f6534376602b6e094e1895b93c4ee74-weights.onnx (result: df913 Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/unet.py' to file '/home/jovyan/.cache/bioimageio/760a77af34fd5e3e1eebeff7e0669585-unet.py'. 100%|█████████████████████████████████████| 20.9k/20.9k [00:00<00:00, 41.7MB/s] computing SHA256 of 760a77af34fd5e3e1eebeff7e0669585-unet.py (result: 7f5b15948e Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights.pt' to file '/home/jovyan/.cache/bioimageio/78e469078bf5116258ceaf6f41001dd7-weights.pt'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 128GB/s] computing SHA256 of 78e469078bf5116258ceaf6f41001dd7-weights.pt (result: 608f52c Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights-torchscript.pt' to file '/home/jovyan/.cache/bioimageio/d44de80414e1fd1414b97beaaed935f5-weights-torchscript.pt'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 252GB/s] computing SHA256 of d44de80414e1fd1414b97beaaed935f5-weights-torchscript.pt (res 2024-09-12 17:15:46.533 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' /srv/conda/envs/notebook/lib/python3.10/site-packages/bioimageio/core/_resource_tests.py:407: UserWarning: The argument `decimal` has been depricated in favour of `relative_tolerance` and `absolute_tolerance`, with different validation logic, using `numpy.testing.assert_allclose, see 'https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html'. Passing a value for `decimal` will cause validation to revert to the old behaviour. warnings.warn( 2024-09-12 17:15:46.540 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes 2024-09-12 17:15:46.544 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' 2024-09-12 17:15:46.555 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes 2024-09-12 17:15:46.557 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' 2024-09-12 17:15:46.566 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes ❌ bioimageio validation failed ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ source [https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.…](https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.%E2%80%A6) format version model 0.5.3 bioimageio.spec 0.5.3.2 bioimageio.core 0.6.8 ❓ location detail ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ✔️ initialized ModelDescr to describe model 0.5.3 ✔️ bioimageio.spec format validation model 0.5.3 🔍 context.perform_io_che… True 🔍 context.root [https://uk1s3.embassy.e…](https://uk1s3.embassy.e%E2%80%A6/) 🔍 context.known_files.ze… 767f2c3a50e36365c30b9e4… 🔍 context.known_files.te… c29bd6e16e3f7856217b407… 🔍 context.known_files.sa… a24b3c708b6ca6825494eb7… 🔍 context.known_files.te… 510181f38930e59e4fd8ecc… 🔍 context.known_files.sa… e8f99aabe8405427f515eba… 🔍 context.known_files.we… df913b85947f5132bcdaf81… 🔍 context.known_files.un… 7f5b15948e8e2c91f78dcff… 🔍 context.known_files.we… 608f52cd7f5119f7a7b8272… 🔍 context.known_files.we… 8410950508655a300793b38… 🔍 context.warning_level error ⚠ documentation No '# Validation' (sub)section found in documentation.md. ❌ Reproduce test outputs from test inputs ❌ weights.onnx The 'onnx' model adapter could not be created in this environment: ImportError(failed to import onnxruntime: No module named 'onnxruntime'). | | | | ]8;id=839366;_resource_tests.py\_resource_tests.py]8;;\, line 161, in _test_model_inference with create_prediction_pipeline( | | | | ]8;id=688587;_prediction_pipeline.py\_prediction_pipeline.py]8;;\, line 333, in create_prediction_pipeline model_adapter = model_adapter or create_model_adapter( | | | | ]8;id=670151;_model_adapter.py\_model_adapter.py]8;;\, line 138, in create raise ValueError( | | | | | | ❌ | | Run onnx inference for parametrized inputs | | ❌ | weights.onnx | The 'onnx' model adapter could not be created in this environment: ImportError(failed to import onnxruntime: No module named 'onnxruntime'). | | | | ]8;id=227383;_resource_tests.py\_resource_tests.py]8;;\, line 296, in _test_model_inference_parametrized with create_prediction_pipeline( | | | | ]8;id=394538;_prediction_pipeline.py\_prediction_pipeline.py]8;;\, line 333, in create_prediction_pipeline model_adapter = model_adapter or create_model_adapter( | | | | ]8;id=235797;_model_adapter.py\_model_adapter.py]8;;\, line 138, in create raise ValueError( | | | | | | ❌ | | Reproduce test outputs from test inputs | | ❌ | weights.pytorch_state_dict | The 'pytorch_state_dict' model adapter could not be created in this environment: ImportError(failed to import torch: No module named 'torch'). | | | | ]8;id=608868;_resource_tests.py\_resource_tests.py]8;;\, line 161, in _test_model_inference with create_prediction_pipeline( | | | | ]8;id=593174;_prediction_pipeline.py\_prediction_pipeline.py]8;;\, line 333, in create_prediction_pipeline model_adapter = model_adapter or create_model_adapter( | | | | ]8;id=396184;_model_adapter.py\_model_adapter.py]8;;\, line 138, in create raise ValueError( | | | | | | ❌ | | Run pytorch_state_dict inference for parametrized inputs | | ❌ | weights.pytorch_state_dict | The 'pytorch_state_dict' model adapter could not be created in this environment: ImportError(failed to import torch: No module named 'torch'). | | | | ]8;id=13371;_resource_tests.py\_resource_tests.py]8;;\, line 296, in _test_model_inference_parametrized with create_prediction_pipeline( | | | | ]8;id=137629;_prediction_pipeline.py\_prediction_pipeline.py]8;;\, line 333, in create_prediction_pipeline model_adapter = model_adapter or create_model_adapter( | | | | ]8;id=975301;_model_adapter.py\_model_adapter.py]8;;\, line 138, in create raise ValueError( | | | | | | ❌ | | Reproduce test outputs from test inputs | | ❌ | weights.torchscript | The 'torchscript' model adapter could not be created in this environment: ImportError(failed to import torch: No module named 'torch'). | | | | ]8;id=429218;_resource_tests.py\_resource_tests.py]8;;\, line 161, in _test_model_inference with create_prediction_pipeline( | | | | ]8;id=163549;_prediction_pipeline.py\_prediction_pipeline.py]8;;\, line 333, in create_prediction_pipeline model_adapter = model_adapter or create_model_adapter( | | | | ]8;id=43384;_model_adapter.py\_model_adapter.py]8;;\, line 138, in create raise ValueError( | | | | | | ❌ | | Run torchscript inference for parametrized inputs | | ❌ | weights.torchscript | The 'torchscript' model adapter could not be created in this environment: ImportError(failed to import torch: No module named 'torch'). | | | | ]8;id=935828;_resource_tests.py\_resource_tests.py]8;;\, line 296, in _test_model_inference_parametrized with create_prediction_pipeline( | | | | ]8;id=672681;_prediction_pipeline.py\_prediction_pipeline.py]8;;\, line 333, in create_prediction_pipeline model_adapter = model_adapter or create_model_adapter( | | | | ]8;id=379847;_model_adapter.py\_model_adapter.py]8;;\, line 138, in create raise ValueError( | | | | ```