Open RobBW opened 2 months ago
chaofengc
commented
2 months ago The code uses PyTorch for GPU acceleration, allowing you to set the compute device in the same way as in PyTorch. For instance, you can use metric.to(torch.device("mps")) to specify the device.
Please note, I haven't tested the code on Mac yet. Let me know if it works for you!
RobBW
commented
2 months ago pyiqa runs on my Apple Silicon M2, Sonoma. I've only run the hyperiqa metric on 2 images and got results more or less in line with my own judgement. If you post some test images that you have already measured and advise which metrics you would like me to run I can then verify that pyiqa is working correctly on MacOS. I will test and confirm the different device modes soon. Its pretty fast anyway.
chaofengc
commented
2 months ago Thank you for providing information about macOS. I greatly appreciate your willingness to help verify the functionality of pyiqa on macOS.
To run the tests, please follow these steps:
git.make test_cal. This will verify the results against official values and save them in ResultsCalibra/calibration_summary.csv.RobBW
commented
2 months ago I attach two files recording the results of my tests of pyIQA on Apple Silicon running Sonoma
I used the Warp terminal which is rather smart and helpful. (https://www.warp.dev/) I've copied its output into text files produced by BBEdit.
The 'MPS only' test report documents my installation process and the adjustments I had to make to get any test results at all.
I'll comment by referring to line numbers in the report:
L603 showed that there was a timm module missing. This was corrected by installing the most recently available version. L693 This allowed me to run the -ls command successfully
Running make test_cal texposed the fact that pytest was needed. L714
L735 make test_cal ran all the tests with a mixture of passes and fails. See L744 to 771
The test summaries L3539 onward give the reasons for most of the failures.
The problem of MPS being unable to run the aten operator is well known and has a simple fix which I made.
It is to put set the environment variable PYTORCH_ENABLE_MPS_FALLBACK=1 ahead of the command.
Re-ran the test, this time yielding a higher pass rate . The log for all this is attached in the file labelled CPU/MPS. I dont have time to comment on it but it seems to me that a simple code change in one or two places will eliminate the remaining float64 errors
I hope all of this is helpful to you .
Happy to test any changes that you make.
Regards Rob
@.*** οΏΌοΏΌ
On 16 Sep 2024, at 18:11, Chaofeng Chen @.***> wrote:
make test_cal
cd ~
pip uninstall pyiqa Found existing installation: pyiqa 0.1.12 Uninstalling pyiqa-0.1.12: Would remove: /Users/robbw/miniforge3/bin/pyiqa /Users/robbw/miniforge3/lib/python3.10/site-packages/pyiqa.egg-link Proceed (Y/n)? y Successfully uninstalled pyiqa-0.1.12
git clone https://github.com/chaofengc/IQA-PyTorch.git Cloning into 'IQA-PyTorch'... remote: Enumerating objects: 4019, done. remote: Counting objects: 100% (1805/1805), done. remote: Compressing objects: 100% (575/575), done. remote: Total 4019 (delta 1326), reused 1567 (delta 1213), pack-reused 2214 (from 1) Receiving objects: 100% (4019/4019), 26.83 MiB | 14.01 MiB/s, done. Resolving deltas: 100% (2812/2812), done. cd /Users/robbw/XPLAY/IQA-PyTorch
pip install -r requirements.txt Requirement already satisfied: addict in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (2.4.0) Requirement already satisfied: future in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 2)) (0.16.0) Requirement already satisfied: lmdb in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (1.5.1) Requirement already satisfied: numpy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 4)) (2.0.2) Requirement already satisfied: opencv-python-headless in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (4.10.0.84) Requirement already satisfied: pandas in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 6)) (2.2.2) Requirement already satisfied: Pillow in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (9.2.0) Requirement already satisfied: pyyaml in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 8)) (6.0.1) Requirement already satisfied: requests in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (2.20.0) Requirement already satisfied: scikit-image in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (0.24.0) Requirement already satisfied: scipy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (1.13.1) Requirement already satisfied: tensorboard in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (2.17.1) Requirement already satisfied: timm in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 13)) (0.4.12) Requirement already satisfied: torch>=1.12 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (2.3.1) Requirement already satisfied: torchvision>=0.13 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (0.18.1) Requirement already satisfied: tqdm in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 16)) (4.64.0) Requirement already satisfied: yapf in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 17)) (0.40.2) Requirement already satisfied: einops in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 18)) (0.4.1) Requirement already satisfied: openai-clip in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 19)) (1.0.1) Requirement already satisfied: facexlib in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 20)) (0.3.0) Requirement already satisfied: transformers==4.37.2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 21)) (4.37.2) Requirement already satisfied: accelerate in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 22)) (0.34.2) Requirement already satisfied: bitsandbytes in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 23)) (0.42.0) Requirement already satisfied: icecream in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 24)) (2.1.3) Requirement already satisfied: sentencepiece in /Users/robbw/miniforge3/lib/python3.10/site-packages (from -r requirements.txt (line 25)) (0.2.0) Requirement already satisfied: filelock in /Users/robbw/miniforge3/lib/python3.10/site-packages (from transformers==4.37.2->-r requirements.txt (line 21)) (3.15.4) Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from transformers==4.37.2->-r requirements.txt (line 21)) (0.24.7) Requirement already satisfied: packaging>=20.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from transformers==4.37.2->-r requirements.txt (line 21)) (24.1) Requirement already satisfied: regex!=2019.12.17 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from transformers==4.37.2->-r requirements.txt (line 21)) (2024.9.11) Requirement already satisfied: tokenizers<0.19,>=0.14 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from transformers==4.37.2->-r requirements.txt (line 21)) (0.15.2) Requirement already satisfied: safetensors>=0.4.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from transformers==4.37.2->-r requirements.txt (line 21)) (0.4.5) Requirement already satisfied: python-dateutil>=2.8.2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pandas->-r requirements.txt (line 6)) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pandas->-r requirements.txt (line 6)) (2024.1) Requirement already satisfied: tzdata>=2022.7 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pandas->-r requirements.txt (line 6)) (2024.1) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->-r requirements.txt (line 9)) (3.0.4) Requirement already satisfied: idna<2.8,>=2.5 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->-r requirements.txt (line 9)) (2.7) Requirement already satisfied: urllib3<1.25,>=1.21.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->-r requirements.txt (line 9)) (1.24.3) Requirement already satisfied: certifi>=2017.4.17 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->-r requirements.txt (line 9)) (2024.8.30) Requirement already satisfied: networkx>=2.8 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from scikit-image->-r requirements.txt (line 10)) (3.3) Requirement already satisfied: imageio>=2.33 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from scikit-image->-r requirements.txt (line 10)) (2.35.1) Requirement already satisfied: tifffile>=2022.8.12 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from scikit-image->-r requirements.txt (line 10)) (2024.8.30) Requirement already satisfied: lazy-loader>=0.4 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from scikit-image->-r requirements.txt (line 10)) (0.4) Requirement already satisfied: absl-py>=0.4 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (2.1.0) Requirement already satisfied: grpcio>=1.48.2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (1.66.1) Requirement already satisfied: markdown>=2.6.8 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (3.7) Requirement already satisfied: protobuf!=4.24.0,>=3.19.6 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (3.20.1) Requirement already satisfied: setuptools>=41.0.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (75.1.0) Requirement already satisfied: six>1.9 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (1.16.0) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from tensorboard->-r requirements.txt (line 12)) (3.0.4) Requirement already satisfied: typing-extensions>=4.8.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.12->-r requirements.txt (line 14)) (4.12.2) Requirement already satisfied: sympy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.12->-r requirements.txt (line 14)) (1.13.0) Requirement already satisfied: jinja2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.12->-r requirements.txt (line 14)) (3.1.4) Requirement already satisfied: fsspec in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.12->-r requirements.txt (line 14)) (2024.6.1) Requirement already satisfied: importlib-metadata>=6.6.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from yapf->-r requirements.txt (line 17)) (8.0.0) Requirement already satisfied: platformdirs>=3.5.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from yapf->-r requirements.txt (line 17)) (4.2.2) Requirement already satisfied: tomli>=2.0.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from yapf->-r requirements.txt (line 17)) (2.0.1) Requirement already satisfied: ftfy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from openai-clip->-r requirements.txt (line 19)) (6.2.3) Requirement already satisfied: filterpy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from facexlib->-r requirements.txt (line 20)) (1.4.5) Requirement already satisfied: numba in /Users/robbw/miniforge3/lib/python3.10/site-packages (from facexlib->-r requirements.txt (line 20)) (0.60.0) Requirement already satisfied: opencv-python in /Users/robbw/miniforge3/lib/python3.10/site-packages (from facexlib->-r requirements.txt (line 20)) (4.6.0.66) Requirement already satisfied: psutil in /Users/robbw/miniforge3/lib/python3.10/site-packages (from accelerate->-r requirements.txt (line 22)) (6.0.0) Requirement already satisfied: colorama>=0.3.9 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from icecream->-r requirements.txt (line 24)) (0.4.6) Requirement already satisfied: pygments>=2.2.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from icecream->-r requirements.txt (line 24)) (2.18.0) Requirement already satisfied: executing>=0.3.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from icecream->-r requirements.txt (line 24)) (2.0.1) Requirement already satisfied: asttokens>=2.0.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from icecream->-r requirements.txt (line 24)) (2.4.1) Requirement already satisfied: zipp>=0.5 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from importlib-metadata>=6.6.0->yapf->-r requirements.txt (line 17)) (3.19.2) Requirement already satisfied: MarkupSafe>=2.1.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard->-r requirements.txt (line 12)) (2.1.5) Requirement already satisfied: matplotlib in /Users/robbw/miniforge3/lib/python3.10/site-packages (from filterpy->facexlib->-r requirements.txt (line 20)) (3.9.2) Requirement already satisfied: wcwidth<0.3.0,>=0.2.12 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from ftfy->openai-clip->-r requirements.txt (line 19)) (0.2.13) Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from numba->facexlib->-r requirements.txt (line 20)) (0.43.0) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from sympy->torch>=1.12->-r requirements.txt (line 14)) (1.3.0) Requirement already satisfied: contourpy>=1.0.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from matplotlib->filterpy->facexlib->-r requirements.txt (line 20)) (1.3.0) Requirement already satisfied: cycler>=0.10 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from matplotlib->filterpy->facexlib->-r requirements.txt (line 20)) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from matplotlib->filterpy->facexlib->-r requirements.txt (line 20)) (4.53.1) Requirement already satisfied: kiwisolver>=1.3.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from matplotlib->filterpy->facexlib->-r requirements.txt (line 20)) (1.4.7) Requirement already satisfied: pyparsing>=2.3.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from matplotlib->filterpy->facexlib->-r requirements.txt (line 20)) (3.1.4)
python setup.py develop running develop /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/command/develop.py:41: EasyInstallDeprecationWarning: easy_install command is deprecated. !!
********************************************************************************
Please avoid running ``setup.py`` and ``easy_install``.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://github.com/pypa/setuptools/issues/917 for details.
********************************************************************************!! easy_install.initialize_options(self) /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated. !!
********************************************************************************
Please avoid running ``setup.py`` directly.
Instead, use pypa/build, pypa/installer or other
standards-based tools.
See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
********************************************************************************!! self.initialize_options() running egg_info creating pyiqa.egg-info writing pyiqa.egg-info/PKG-INFO writing dependency_links to pyiqa.egg-info/dependency_links.txt writing entry points to pyiqa.egg-info/entry_points.txt writing requirements to pyiqa.egg-info/requires.txt writing top-level names to pyiqa.egg-info/top_level.txt writing manifest file 'pyiqa.egg-info/SOURCES.txt' reading manifest file 'pyiqa.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' adding license file 'LICENSE-S-Lab' writing manifest file 'pyiqa.egg-info/SOURCES.txt' running build_ext Creating /Users/robbw/miniforge3/lib/python3.10/site-packages/pyiqa.egg-link (link to .) Adding pyiqa 0.1.12 to easy-install.pth file Installing pyiqa script to /Users/robbw/miniforge3/bin
Installed /Users/robbw/XPLAY/IQA-PyTorch Processing dependencies for pyiqa==0.1.12 Searching for sentencepiece==0.2.0 Best match: sentencepiece 0.2.0 Adding sentencepiece 0.2.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for icecream==2.1.3 Best match: icecream 2.1.3 Adding icecream 2.1.3 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for bitsandbytes==0.42.0 Best match: bitsandbytes 0.42.0 Adding bitsandbytes 0.42.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for accelerate==0.34.2 Best match: accelerate 0.34.2 Adding accelerate 0.34.2 to easy-install.pth file Installing accelerate script to /Users/robbw/miniforge3/bin Installing accelerate-config script to /Users/robbw/miniforge3/bin Installing accelerate-estimate-memory script to /Users/robbw/miniforge3/bin Installing accelerate-launch script to /Users/robbw/miniforge3/bin Installing accelerate-merge-weights script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for transformers==4.37.2 Best match: transformers 4.37.2 Adding transformers 4.37.2 to easy-install.pth file Installing transformers-cli script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for facexlib==0.3.0 Best match: facexlib 0.3.0 Adding facexlib 0.3.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for openai-clip==1.0.1 Best match: openai-clip 1.0.1 Adding openai-clip 1.0.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for einops==0.4.1 Best match: einops 0.4.1 Adding einops 0.4.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for yapf==0.40.2 Best match: yapf 0.40.2 Adding yapf 0.40.2 to easy-install.pth file Installing yapf script to /Users/robbw/miniforge3/bin Installing yapf-diff script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for tqdm==4.66.1 Best match: tqdm 4.66.1 Adding tqdm 4.66.1 to easy-install.pth file Installing tqdm script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for torchvision==0.18.1 Best match: torchvision 0.18.1 Adding torchvision 0.18.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for torch==2.3.1 Best match: torch 2.3.1 Adding torch 2.3.1 to easy-install.pth file Installing convert-caffe2-to-onnx script to /Users/robbw/miniforge3/bin Installing convert-onnx-to-caffe2 script to /Users/robbw/miniforge3/bin Installing torchrun script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for timm==0.4.12 Best match: timm 0.4.12 Adding timm 0.4.12 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for tensorboard==2.17.1 Best match: tensorboard 2.17.1 Adding tensorboard 2.17.1 to easy-install.pth file Installing tensorboard script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for scipy==1.13.1 Best match: scipy 1.13.1 Adding scipy 1.13.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for scikit-image==0.24.0 Best match: scikit-image 0.24.0 Adding scikit-image 0.24.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for requests==2.31.0 Best match: requests 2.31.0 Adding requests 2.31.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for PyYAML==6.0.1 Best match: PyYAML 6.0.1 Adding PyYAML 6.0.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for Pillow==9.2.0 Best match: Pillow 9.2.0 Adding Pillow 9.2.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for pandas==2.2.2 Best match: pandas 2.2.2 Adding pandas 2.2.2 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for opencv-python-headless==4.10.0.84 Best match: opencv-python-headless 4.10.0.84 Adding opencv-python-headless 4.10.0.84 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for numpy==2.0.2 Best match: numpy 2.0.2 Adding numpy 2.0.2 to easy-install.pth file Installing f2py script to /Users/robbw/miniforge3/bin Installing numpy-config script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for lmdb==1.5.1 Best match: lmdb 1.5.1 Adding lmdb 1.5.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for future==0.16.0 Best match: future 0.16.0 Adding future 0.16.0 to easy-install.pth file Installing futurize script to /Users/robbw/miniforge3/bin Installing pasteurize script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for addict==2.4.0 Best match: addict 2.4.0 Adding addict 2.4.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for asttokens==2.4.1 Best match: asttokens 2.4.1 Adding asttokens 2.4.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for executing==2.0.1 Best match: executing 2.0.1 Adding executing 2.0.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for pygments==2.18.0 Best match: pygments 2.18.0 Adding pygments 2.18.0 to easy-install.pth file Installing pygmentize script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for colorama==0.4.6 Best match: colorama 0.4.6 Adding colorama 0.4.6 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for safetensors==0.4.5 Best match: safetensors 0.4.5 Adding safetensors 0.4.5 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for huggingface-hub==0.24.7 Best match: huggingface-hub 0.24.7 Adding huggingface-hub 0.24.7 to easy-install.pth file Installing huggingface-cli script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for psutil==6.0.0 Best match: psutil 6.0.0 Adding psutil 6.0.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for packaging==24.1 Best match: packaging 24.1 Adding packaging 24.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_vendor Searching for tokenizers==0.15.2 Best match: tokenizers 0.15.2 Adding tokenizers 0.15.2 to easy-install.pth file detected new path './setuptools/_vendor'
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for regex==2024.9.11 Best match: regex 2024.9.11 Adding regex 2024.9.11 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for filelock==3.15.4 Best match: filelock 3.15.4 Adding filelock 3.15.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for opencv-python==4.6.0.66 Best match: opencv-python 4.6.0.66 Adding opencv-python 4.6.0.66 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for numba==0.60.0 Best match: numba 0.60.0 Adding numba 0.60.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for filterpy==1.4.5 Best match: filterpy 1.4.5 Adding filterpy 1.4.5 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for ftfy==6.2.3 Best match: ftfy 6.2.3 Adding ftfy 6.2.3 to easy-install.pth file Installing ftfy script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for tomli==2.0.1 Best match: tomli 2.0.1 tomli 2.0.1 is already the active version in easy-install.pth
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_vendor Searching for platformdirs==4.2.2 Best match: platformdirs 4.2.2 platformdirs 4.2.2 is already the active version in easy-install.pth
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_vendor Searching for importlib-metadata==8.0.0 Best match: importlib-metadata 8.0.0 importlib-metadata 8.0.0 is already the active version in easy-install.pth
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_vendor Searching for fsspec==2024.6.1 Best match: fsspec 2024.6.1 Adding fsspec 2024.6.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for jinja2==3.1.4 Best match: jinja2 3.1.4 Adding jinja2 3.1.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for networkx==3.3 Best match: networkx 3.3 Adding networkx 3.3 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for sympy==1.13.0 Best match: sympy 1.13.0 Adding sympy 1.13.0 to easy-install.pth file Installing isympy script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for typing-extensions==4.12.2 Best match: typing-extensions 4.12.2 typing-extensions 4.12.2 is already the active version in easy-install.pth
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_vendor Searching for werkzeug==3.0.4 Best match: werkzeug 3.0.4 Adding werkzeug 3.0.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for tensorboard-data-server==0.7.2 Best match: tensorboard-data-server 0.7.2 Adding tensorboard-data-server 0.7.2 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for six==1.16.0 Best match: six 1.16.0 Adding six 1.16.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for setuptools==75.1.0 Best match: setuptools 75.1.0 Processing setuptools-75.1.0-py3.10.egg Adding setuptools 75.1.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools-75.1.0-py3.10.egg Searching for protobuf==3.20.1 Best match: protobuf 3.20.1 Adding protobuf 3.20.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for Markdown==3.7 Best match: Markdown 3.7 Adding Markdown 3.7 to easy-install.pth file Installing markdown_py script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for grpcio==1.66.1 Best match: grpcio 1.66.1 Adding grpcio 1.66.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for absl-py==2.1.0 Best match: absl-py 2.1.0 Adding absl-py 2.1.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for lazy-loader==0.4 Best match: lazy-loader 0.4 Adding lazy-loader 0.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for tifffile==2024.8.30 Best match: tifffile 2024.8.30 Adding tifffile 2024.8.30 to easy-install.pth file Installing lsm2bin script to /Users/robbw/miniforge3/bin Installing tiff2fsspec script to /Users/robbw/miniforge3/bin Installing tiffcomment script to /Users/robbw/miniforge3/bin Installing tifffile script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for imageio==2.35.1 Best match: imageio 2.35.1 Adding imageio 2.35.1 to easy-install.pth file Installing imageio_download_bin script to /Users/robbw/miniforge3/bin Installing imageio_remove_bin script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for certifi==2024.8.30 Best match: certifi 2024.8.30 Adding certifi 2024.8.30 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for urllib3==2.0.4 Best match: urllib3 2.0.4 Adding urllib3 2.0.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for idna==3.4 Best match: idna 3.4 Adding idna 3.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for charset-normalizer==3.3.2 Best match: charset-normalizer 3.3.2 Adding charset-normalizer 3.3.2 to easy-install.pth file Installing normalizer script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for tzdata==2024.1 Best match: tzdata 2024.1 Adding tzdata 2024.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for pytz==2024.1 Best match: pytz 2024.1 Adding pytz 2024.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for python-dateutil==2.8.2 Best match: python-dateutil 2.8.2 Adding python-dateutil 2.8.2 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for llvmlite==0.43.0 Best match: llvmlite 0.43.0 Adding llvmlite 0.43.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for matplotlib==3.9.2 Best match: matplotlib 3.9.2 Adding matplotlib 3.9.2 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for wcwidth==0.2.13 Best match: wcwidth 0.2.13 Adding wcwidth 0.2.13 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for zipp==3.19.2 Best match: zipp 3.19.2 zipp 3.19.2 is already the active version in easy-install.pth
Using /Users/robbw/miniforge3/lib/python3.10/site-packages/setuptools/_vendor Searching for MarkupSafe==2.1.5 Best match: MarkupSafe 2.1.5 Adding MarkupSafe 2.1.5 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for mpmath==1.3.0 Best match: mpmath 1.3.0 Adding mpmath 1.3.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for pyparsing==3.1.4 Best match: pyparsing 3.1.4 Adding pyparsing 3.1.4 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for kiwisolver==1.4.7 Best match: kiwisolver 1.4.7 Adding kiwisolver 1.4.7 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for fonttools==4.53.1 Best match: fonttools 4.53.1 Adding fonttools 4.53.1 to easy-install.pth file Installing fonttools script to /Users/robbw/miniforge3/bin Installing pyftmerge script to /Users/robbw/miniforge3/bin Installing pyftsubset script to /Users/robbw/miniforge3/bin Installing ttx script to /Users/robbw/miniforge3/bin
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for cycler==0.12.1 Best match: cycler 0.12.1 Adding cycler 0.12.1 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages Searching for contourpy==1.3.0 Best match: contourpy 1.3.0 Adding contourpy 1.3.0 to easy-install.pth file
Using /Users/robbw/miniforge3/lib/python3.10/site-packages
Finished processing dependencies for pyiqa==0.1.12
pyiqa -ls
Traceback (most recent call last):
File "/Users/robbw/miniforge3/bin/pyiqa", line 33, in
pip install timm Requirement already satisfied: timm in /Users/robbw/miniforge3/lib/python3.10/site-packages (0.4.12) Requirement already satisfied: torch>=1.4 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (2.3.1) Requirement already satisfied: torchvision in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (0.18.1) Requirement already satisfied: filelock in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.4->timm) (3.15.4) Requirement already satisfied: typing-extensions>=4.8.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.4->timm) (4.12.2) Requirement already satisfied: sympy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.4->timm) (1.13.0) Requirement already satisfied: networkx in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.4->timm) (3.3) Requirement already satisfied: jinja2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.4->timm) (3.1.4) Requirement already satisfied: fsspec in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch>=1.4->timm) (2024.6.1) Requirement already satisfied: numpy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torchvision->timm) (2.0.2) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torchvision->timm) (9.2.0) Requirement already satisfied: MarkupSafe>=2.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from jinja2->torch>=1.4->timm) (2.1.5) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from sympy->torch>=1.4->timm) (1.3.0)
pyiqa -ls
Traceback (most recent call last):
File "/Users/robbw/miniforge3/bin/pyiqa", line 33, in
pip uninstall timm Found existing installation: timm 0.4.12 Uninstalling timm-0.4.12: Would remove: /Users/robbw/miniforge3/lib/python3.10/site-packages/timm-0.4.12.dist-info/ /Users/robbw/miniforge3/lib/python3.10/site-packages/timm/ Proceed (Y/n)? y Successfully uninstalled timm-0.4.12
pip install timm -i https:??pypi.tuna.tsinghua.edu.cn/simple zsh: no matches found: https:??pypi.tuna.tsinghua.edu.cn/simple pip install timm -i https://pypi.tuna.tsinghua.edu.cn/simple Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting timm Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e7/0e/ef97f6d8c399bf5842af0dd5a4f5ac55b2f169d62e29ecbf7663e1cb1438/timm-1.0.9-py3-none-any.whl (2.3 MB) ββββββββββββββββββββββββββββββββββββββββ 2.3/2.3 MB 248.0 kB/s eta 0:00:00 Requirement already satisfied: torch in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (2.3.1) Requirement already satisfied: torchvision in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (0.18.1) Requirement already satisfied: pyyaml in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (6.0.1) Requirement already satisfied: huggingface_hub in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (0.24.7) Requirement already satisfied: safetensors in /Users/robbw/miniforge3/lib/python3.10/site-packages (from timm) (0.4.5) Requirement already satisfied: filelock in /Users/robbw/miniforge3/lib/python3.10/site-packages (from huggingface_hub->timm) (3.15.4) Requirement already satisfied: fsspec>=2023.5.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from huggingface_hub->timm) (2024.6.1) Requirement already satisfied: packaging>=20.9 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from huggingface_hub->timm) (24.1) Requirement already satisfied: requests in /Users/robbw/miniforge3/lib/python3.10/site-packages (from huggingface_hub->timm) (2.20.0) Requirement already satisfied: tqdm>=4.42.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from huggingface_hub->timm) (4.64.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from huggingface_hub->timm) (4.12.2) Requirement already satisfied: sympy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch->timm) (1.13.0) Requirement already satisfied: networkx in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch->timm) (3.3) Requirement already satisfied: jinja2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torch->timm) (3.1.4) Requirement already satisfied: numpy in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torchvision->timm) (2.0.2) Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from torchvision->timm) (9.2.0) Requirement already satisfied: MarkupSafe>=2.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from jinja2->torch->timm) (2.1.5) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->huggingface_hub->timm) (3.0.4) Requirement already satisfied: idna<2.8,>=2.5 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->huggingface_hub->timm) (2.7) Requirement already satisfied: urllib3<1.25,>=1.21.1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->huggingface_hub->timm) (1.24.3) Requirement already satisfied: certifi>=2017.4.17 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from requests->huggingface_hub->timm) (2024.8.30) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from sympy->torch->timm) (1.3.0) Installing collected packages: timm Successfully installed timm-1.0.9
pyiqa -ls ['ahiq', 'arniqa', 'arniqa-clive', 'arniqa-csiq', 'arniqa-flive', 'arniqa-kadid', 'arniqa-live', 'arniqa-spaq', 'arniqa-tid', 'brisque', 'brisque_matlab', 'ckdn', 'clipiqa', 'clipiqa+', 'clipiqa+_rn50_512', 'clipiqa+_vitL14_512', 'clipscore', 'cnniqa', 'cw_ssim', 'dbcnn', 'dists', 'entropy', 'fid', 'fsim', 'gmsd', 'hyperiqa', 'ilniqe', 'inception_score', 'laion_aes', 'liqe', 'liqe_mix', 'lpips', 'lpips+', 'lpips-vgg', 'lpips-vgg+', 'mad', 'maniqa', 'maniqa-kadid', 'maniqa-pipal', 'ms_ssim', 'musiq', 'musiq-ava', 'musiq-paq2piq', 'musiq-spaq', 'nima', 'nima-koniq', 'nima-spaq', 'nima-vgg16-ava', 'niqe', 'niqe_matlab', 'nlpd', 'nrqm', 'paq2piq', 'pi', 'pieapp', 'piqe', 'psnr', 'psnry', 'qalign', 'qalign_4bit', 'qalign_8bit', 'ssim', 'ssimc', 'stlpips', 'stlpips-vgg', 'topiq_fr', 'topiq_fr-pipal', 'topiq_iaa', 'topiq_iaa_res50', 'topiq_nr', 'topiq_nr-face', 'topiq_nr-flive', 'topiq_nr-spaq', 'tres', 'tres-flive', 'unique', 'uranker', 'vif', 'vsi', 'wadiqam_fr', 'wadiqam_nr']
make test_cal pytest tests/ -m calibration -v make: pytest: No such file or directory make: *** [test_cal] Error 1 pip find pytests ERROR: unknown command "find" - maybe you meant "index" pytest --help zsh: command not found: pytest
pip install pytest Collecting pytest Using cached pytest-8.3.3-py3-none-any.whl.metadata (7.5 kB) Collecting iniconfig (from pytest) Using cached iniconfig-2.0.0-py3-none-any.whl.metadata (2.6 kB) Requirement already satisfied: packaging in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pytest) (24.1) Requirement already satisfied: pluggy<2,>=1.5 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pytest) (1.5.0) Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pytest) (1.2.2) Requirement already satisfied: tomli>=1 in /Users/robbw/miniforge3/lib/python3.10/site-packages (from pytest) (2.0.1) Using cached pytest-8.3.3-py3-none-any.whl (342 kB) Using cached iniconfig-2.0.0-py3-none-any.whl (5.9 kB) Installing collected packages: iniconfig, pytest Successfully installed iniconfig-2.0.0 pytest-8.3.3
make test_cal pytest tests/ -m calibration -v =========================================================== test session starts =========================================================== platform darwin -- Python 3.10.14, pytest-8.3.3, pluggy-1.5.0 -- /Users/robbw/miniforge3/bin/python3.10 cachedir: .pytest_cache rootdir: /Users/robbw/XPLAY/IQA-PyTorch plugins: anyio-4.1.0, typeguard-4.3.0 collected 268 items / 240 deselected / 28 selected
tests/test_metric_general.py::test_match_official_with_given_cases[brisque] PASSED [ 3%] tests/test_metric_general.py::test_match_official_with_given_cases[brisque_matlab] PASSED [ 7%] tests/test_metric_general.py::test_match_official_with_given_cases[ckdn] FAILED [ 10%] tests/test_metric_general.py::test_match_official_with_given_cases[cw_ssim] FAILED [ 14%] tests/test_metric_general.py::test_match_official_with_given_cases[dists] PASSED [ 17%] tests/test_metric_general.py::test_match_official_with_given_cases[fsim] PASSED [ 21%] tests/test_metric_general.py::test_match_official_with_given_cases[gmsd] PASSED [ 25%] tests/test_metric_general.py::test_match_official_with_given_cases[lpips] PASSED [ 28%] tests/test_metric_general.py::test_match_official_with_given_cases[mad] FAILED [ 32%] tests/test_metric_general.py::test_match_official_with_given_cases[ms_ssim] FAILED [ 35%] tests/test_metric_general.py::test_match_official_with_given_cases[niqe] FAILED [ 39%] tests/test_metric_general.py::test_match_official_with_given_cases[niqe_matlab] FAILED [ 42%] tests/test_metric_general.py::test_match_official_with_given_cases[piqe] PASSED [ 46%] tests/test_metric_general.py::test_match_official_with_given_cases[nlpd] PASSED [ 50%] tests/test_metric_general.py::test_match_official_with_given_cases[psnr] PASSED [ 53%] tests/test_metric_general.py::test_match_official_with_given_cases[ssim] FAILED [ 57%] tests/test_metric_general.py::test_match_official_with_given_cases[vif] FAILED [ 60%] tests/test_metric_general.py::test_match_official_with_given_cases[vsi] FAILED [ 64%] tests/test_metric_general.py::test_match_official_with_given_cases[nrqm] FAILED [ 67%] tests/test_metric_general.py::test_match_official_with_given_cases[pi] FAILED [ 71%] tests/test_metric_general.py::test_match_official_with_given_cases[ilniqe] FAILED [ 75%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq] FAILED [ 78%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq-ava] FAILED [ 82%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq-paq2piq] FAILED [ 85%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq-spaq] FAILED [ 89%] tests/test_metric_general.py::test_match_official_with_given_cases[paq2piq] PASSED [ 92%] tests/test_metric_general.py::test_match_official_with_given_cases[entropy] PASSED [ 96%] tests/test_metric_general.py::test_match_official_with_given_cases[laion_aes] FAILED [100%]
================================================================ FAILURES ================================================================= _ test_match_official_with_given_cases[ckdn] __
ref_img = tensor([[[[0.5882, 0.5882, 0.6431, ..., 0.4157, 0.4157, 0.4039], [0.5725, 0.5686, 0.5686, ..., 0.4078, 0.4...78, 0.6275, 0.6353, ..., 0.1804, 0.1059, 0.0941], [0.6157, 0.6392, 0.6549, ..., 0.2706, 0.2627, 0.1765]]]]) dist_img = tensor([[[[0.6314, 0.6314, 0.6275, ..., 0.3961, 0.3961, 0.3961], [0.6314, 0.6314, 0.6275, ..., 0.3961, 0.3...59, 0.7216, 0.6941, ..., 0.1176, 0.1176, 0.1176], [0.5059, 0.7216, 0.6941, ..., 0.1176, 0.1176, 0.1176]]]]) metric_name = 'ckdn', device = device(type='mps')
@pytest.mark.calibration
@pytest.mark.parametrize(
("metric_name"),
[(k) for k in metrics_with_official_results().keys()]
)
def test_match_official_with_given_cases(ref_img, dist_img, metric_name, device):
official_result = metrics_with_official_results()[metric_name]
metric = pyiqa.create_metric(metric_name, device=device)score = metric(dist_img, ref_img)
tests/test_metric_general.py:73:
../../miniforge3/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl return self._call_impl(*args, kwargs) ../../miniforge3/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl return forward_call(args, kwargs) pyiqa/models/inference_model.py:104: in forward output = self.net(target.to(device), ref.to(device), kwargs) ../../miniforge3/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl return self._call_impl(args, kwargs) ../../miniforge3/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl return forward_call(*args, **kwargs) pyiqa/archs/ckdn_arch.py:359: in forward x, y = self._default_preprocess(x, y) pyiqa/archs/ckdn_arch.py:337: in _default_preprocess x = tv.transforms.functional.resize(x, scaled_size, tv.transforms.InterpolationMode.BICUBIC) ../../miniforge3/lib/python3.10/site-packages/torchvision/transforms/functional.py:470: in resize return F_t.resize(img, size=output_size, interpolation=interpolation.value, antialias=antialias) ../../miniforge3/lib/python3.10/site-packages/torchvision/transforms/_functional_tensor.py:465: in resize img = interpolate(img, size=size, mode=interpolation, align_corners=align_corners, antialias=antialias)
input = tensor([[[[0.6314, 0.6314, 0.6275, ..., 0.3961, 0.3961, 0.3961], [0.6314, 0.6314, 0.6275, ..., 0.3961, 0.3...., 0.1176, 0.1176, 0.1176], [0.5059, 0.7216, 0.6941, ..., 0.1176, 0.1176, 0.1176]]]], device='mps:0') size = [329, 438], scale_factor = None, mode = 'bicubic', align_corners = False, recompute_scale_factor = None, antialias = True
def interpolate(input: Tensor, size: Optional[int] = None, scale_factor: Optional[List[float]] = None, mode: str = 'nearest', align_corners: Optional[bool] = None, recompute_scale_factor: Optional[bool] = None, antialias: bool = False) -> Tensor: # noqa: F811,B950
r"""Down/up samples the input.
Tensor interpolated to either the given :attr:`size` or the given
:attr:`scale_factor`
The algorithm used for interpolation is determined by :attr:`mode`.
Currently temporal, spatial and volumetric sampling are supported, i.e.
expected inputs are 3-D, 4-D or 5-D in shape.
The input dimensions are interpreted in the form:
`mini-batch x channels x [optional depth] x [optional height] x width`.
The modes available for resizing are: `nearest`, `linear` (3D-only),
`bilinear`, `bicubic` (4D-only), `trilinear` (5D-only), `area`, `nearest-exact`
Args:
input (Tensor): the input tensor
size (int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int]):
output spatial size.
scale_factor (float or Tuple[float]): multiplier for spatial size. If `scale_factor` is a tuple,
its length has to match the number of spatial dimensions; `input.dim() - 2`.
mode (str): algorithm used for upsampling:
``'nearest'`` | ``'linear'`` | ``'bilinear'`` | ``'bicubic'`` |
``'trilinear'`` | ``'area'`` | ``'nearest-exact'``. Default: ``'nearest'``
align_corners (bool, optional): Geometrically, we consider the pixels of the
input and output as squares rather than points.
If set to ``True``, the input and output tensors are aligned by the
center points of their corner pixels, preserving the values at the corner pixels.
If set to ``False``, the input and output tensors are aligned by the corner
points of their corner pixels, and the interpolation uses edge value padding
for out-of-boundary values, making this operation *independent* of input size
when :attr:`scale_factor` is kept the same. This only has an effect when :attr:`mode`
is ``'linear'``, ``'bilinear'``, ``'bicubic'`` or ``'trilinear'``.
Default: ``False``
recompute_scale_factor (bool, optional): recompute the scale_factor for use in the
interpolation calculation. If `recompute_scale_factor` is ``True``, then
`scale_factor` must be passed in and `scale_factor` is used to compute the
output `size`. The computed output `size` will be used to infer new scales for
the interpolation. Note that when `scale_factor` is floating-point, it may differ
from the recomputed `scale_factor` due to rounding and precision issues.
If `recompute_scale_factor` is ``False``, then `size` or `scale_factor` will
be used directly for interpolation. Default: ``None``.
antialias (bool, optional): flag to apply anti-aliasing. Default: ``False``. Using anti-alias
option together with ``align_corners=False``, interpolation result would match Pillow
result for downsampling operation. Supported modes: ``'bilinear'``, ``'bicubic'``.
.. note::
With ``mode='bicubic'``, it's possible to cause overshoot, in other words it can produce
negative values or values greater than 255 for images.
Explicitly call ``result.clamp(min=0, max=255)`` if you want to reduce the overshoot
when displaying the image.
.. note::
Mode ``mode='nearest-exact'`` matches Scikit-Image and PIL nearest neighbours interpolation
algorithms and fixes known issues with ``mode='nearest'``. This mode is introduced to keep
backward compatibility.
Mode ``mode='nearest'`` matches buggy OpenCV's ``INTER_NEAREST`` interpolation algorithm.
.. note::
The gradients for the dtype ``float16`` on CUDA may be inaccurate in the upsample operation
when using modes ``['linear', 'bilinear', 'bicubic', 'trilinear', 'area']``.
For more details, please refer to the discussion in
`issue#104157 <https://github.com/pytorch/pytorch/issues/104157>`_.
Note:
{backward_reproducibility_note}
"""
if has_torch_function_unary(input):
return handle_torch_function(
interpolate,
(input,),
input,
size=size,
scale_factor=scale_factor,
mode=mode,
align_corners=align_corners,
recompute_scale_factor=recompute_scale_factor,
antialias=antialias
)
if mode in ("nearest", "area", "nearest-exact"):
if align_corners is not None:
raise ValueError(
"align_corners option can only be set with the "
"interpolating modes: linear | bilinear | bicubic | trilinear"
)
else:
if align_corners is None:
align_corners = False
dim = input.dim() - 2 # Number of spatial dimensions.
# Process size and scale_factor. Validate that exactly one is set.
# Validate its length if it is a list, or expand it if it is a scalar.
# After this block, exactly one of output_size and scale_factors will
# be non-None, and it will be a list (or tuple).
if size is not None and scale_factor is not None:
raise ValueError("only one of size or scale_factor should be defined")
elif size is not None:
assert scale_factor is None
scale_factors = None
if isinstance(size, (list, tuple)):
if len(size) != dim:
raise ValueError(
"Input and output must have the same number of spatial dimensions, but got "
f"input with spatial dimensions of {list(input.shape[2:])} and output size of {size}. "
"Please provide input tensor in (N, C, d1, d2, ...,dK) format and "
"output size in (o1, o2, ...,oK) format."
)
if not torch.jit.is_scripting():
if not all(_is_integer(x) for x in size):
raise TypeError(
"expected size to be one of int or Tuple[int] or Tuple[int, int] or "
f"Tuple[int, int, int], but got size with types {[type(x) for x in size]}"
)
output_size = size
else:
output_size = [size for _ in range(dim)]
elif scale_factor is not None:
assert size is None
output_size = None
if isinstance(scale_factor, (list, tuple)):
if len(scale_factor) != dim:
raise ValueError(
"Input and scale_factor must have the same number of spatial dimensions, but "
f"got input with spatial dimensions of {list(input.shape[2:])} and "
f"scale_factor of shape {scale_factor}. "
"Please provide input tensor in (N, C, d1, d2, ...,dK) format and "
"scale_factor in (s1, s2, ...,sK) format."
)
scale_factors = scale_factor
else:
scale_factors = [scale_factor for _ in range(dim)]
else:
raise ValueError("either size or scale_factor should be defined")
if recompute_scale_factor is not None and recompute_scale_factor and size is not None:
raise ValueError("recompute_scale_factor is not meaningful with an explicit size.")
# "area" mode always requires an explicit size rather than scale factor.
# Re-use the recompute_scale_factor code path.
if mode == "area" and output_size is None:
recompute_scale_factor = True
if recompute_scale_factor is not None and recompute_scale_factor:
# We compute output_size here, then un-set scale_factors.
# The C++ code will recompute it based on the (integer) output size.
assert scale_factors is not None
if not torch.jit.is_scripting() and torch._C._get_tracing_state():
# make scale_factor a tensor in tracing so constant doesn't get baked in
output_size = [
(torch.floor((input.size(i + 2).float() * torch.tensor(scale_factors[i], dtype=torch.float32)).float()))
for i in range(dim)
]
elif torch.jit.is_scripting():
output_size = [int(math.floor(float(input.size(i + 2)) * scale_factors[i]))
for i in range(dim)]
else:
output_size = [
_sym_int(input.size(i + 2) * scale_factors[i])
for i in range(dim)
]
scale_factors = None
if antialias and not (mode in ("bilinear", "bicubic") and input.ndim == 4):
raise ValueError("Anti-alias option is restricted to bilinear and bicubic modes and requires a 4-D tensor as input")
if input.dim() == 3 and mode == "nearest":
return torch._C._nn.upsample_nearest1d(input, output_size, scale_factors)
if input.dim() == 4 and mode == "nearest":
return torch._C._nn.upsample_nearest2d(input, output_size, scale_factors)
if input.dim() == 5 and mode == "nearest":
return torch._C._nn.upsample_nearest3d(input, output_size, scale_factors)
if input.dim() == 3 and mode == "nearest-exact":
return torch._C._nn._upsample_nearest_exact1d(input, output_size, scale_factors)
if input.dim() == 4 and mode == "nearest-exact":
return torch._C._nn._upsample_nearest_exact2d(input, output_size, scale_factors)
if input.dim() == 5 and mode == "nearest-exact":
return torch._C._nn._upsample_nearest_exact3d(input, output_size, scale_factors)
if input.dim() == 3 and mode == "area":
assert output_size is not None
return adaptive_avg_pool1d(input, output_size)
if input.dim() == 4 and mode == "area":
assert output_size is not None
return adaptive_avg_pool2d(input, output_size)
if input.dim() == 5 and mode == "area":
assert output_size is not None
return adaptive_avg_pool3d(input, output_size)
if input.dim() == 3 and mode == "linear":
assert align_corners is not None
return torch._C._nn.upsample_linear1d(input, output_size, align_corners, scale_factors)
if input.dim() == 4 and mode == "bilinear":
assert align_corners is not None
if antialias:
return torch._C._nn._upsample_bilinear2d_aa(input, output_size, align_corners, scale_factors)
# Two levels are necessary to prevent TorchScript from touching
# are_deterministic_algorithms_enabled.
if not torch.jit.is_scripting():
if torch.are_deterministic_algorithms_enabled() and input.is_cuda:
# Use slow decomp whose backward will be in terms of index_put
# importlib is required because the import cannot be top level
# (cycle) and cannot be nested (TS doesn't support)
return importlib.import_module('torch._decomp.decompositions')._upsample_linear_vec(
input, output_size, align_corners, scale_factors)
return torch._C._nn.upsample_bilinear2d(input, output_size, align_corners, scale_factors)
if input.dim() == 5 and mode == "trilinear":
assert align_corners is not None
return torch._C._nn.upsample_trilinear3d(input, output_size, align_corners, scale_factors)
if input.dim() == 4 and mode == "bicubic":
assert align_corners is not None
if antialias:return torch._C._nn._upsample_bicubic2d_aa(input, output_size, align_corners, scale_factors)E NotImplementedError: The operator 'aten::_upsample_bicubic2d_aa.out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/iss
chaofengc
commented
2 months ago Thank you so much for testing on macOS. I will update the requirements accordingly.
At the moment, it's not feasible for me to resolve the interpolation issue on the mps backend. Additionally, downgrading to fp32 isn't a suitable option, as it could lead to incorrect results for certain metrics. As a result, we will need to wait for PyTorch to fully support the upsample operation on the mps device.
In the meantime, we can work with the M-series CPU. Since the M2 chip is significantly more powerful, it would be fantastic if you could help test the efficiency on it using the following command:
git pull
python tests/test_efficiency.py -d cpu -c cpu_m2Results will be saved in tests/Efficiency_benchmark.csv. Thanks again for your support!
RobBW
commented
2 months ago Done Terminal efficiency report attached.
Also, for completeness a CPU run of test_cal
FYI Have you spotted this paper and code posted 2 weeks ago: LAR-IQA: A Lightweight, Accurate, and Robust No-Reference Image Quality Assessment Model https://arxiv.org/abs/2408.17057 https://github.com/nasimjamshidi/LAR-IQA/tree/mainοΏΌ nasimjamshidi/LAR-IQA github.com
Regards Rob
@.***
On 18 Sep 2024, at 01:25, Chaofeng Chen @.***> wrote:
Thank you so much for testing on macOS. I will update the requirements accordingly.
At the moment, it's not feasible for me to resolve the interpolation issue on the mps backend. Additionally, downgrading to fp32 isn't a suitable option, as it could lead to incorrect results for certain metrics. As a result, we will need to wait for PyTorch to fully support the upsample operation on the mps device.
In the meantime, we can work with the M-series CPU. Since the M2 chip is significantly more powerful, it would be fantastic if you could help test the efficiency on it using the following command:
git pull python tests/test_efficiency.py -d cpu -c cpu_m2 Thanks again for your support!
β
οΏΌ οΏΌ Efficiency Tests M2
(base) @.*** IQA-PyTorch % git pull python tests/test_efficiency.py -d cpu -c cpu_m2 Already up to date. Testing arniqa-kadid on cpu: 8%|βββββββββ | 6/74 [01:53<09:29, 8.37s/it]Downloading: "https://github.com/miccunifi/ARNIQA/releases/download/weights/regressor_live.pth" to /Users/robbw/.cache/torch/hub/checkpoints/regressor_live.pth 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 19.2k/19.2k [00:00<00:00, 4.41MB/s] Testing arniqa-live on cpu: 9%|ββββββββββ | 7/74 [01:57<07:55, 7.10s/it]Downloading: "https://github.com/miccunifi/ARNIQA/releases/download/weights/regressor_spaq.pth" to /Users/robbw/.cache/torch/hub/checkpoints/regressor_spaq.pth 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 19.2k/19.2k [00:00<00:00, 5.12MB/s] Testing arniqa-spaq on cpu: 11%|ββββββββββββ | 8/74 [02:01<06:50, 6.22s/it]Downloading: "https://github.com/miccunifi/ARNIQA/releases/download/weights/regressor_tid2013.pth" to /Users/robbw/.cache/torch/hub/checkpoints/regressor_tid2013.pth 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 19.2k/19.2k [00:00<00:00, 5.09MB/s] Testing brisque_matlab on cpu: 15%|βββββββββββββββ | 11/74 [02:07<03:09, 3.01s/it]Loading pretrained model CKDN from /Users/robbw/.cache/pyiqa/CKDN_model_best-38b27dc6.pth 100%|βββββββββββββββββββββββββββββββββββββββ| 244M/244M [00:08<00:00, 29.6MiB/s] | 12/74 [02:08<02:22, 2.29s/it] Testing clipiqa on cpu: 18%|βββββββββββββββββββ | 13/74 [02:21<05:49, 5.73s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/CLIP-IQA+_learned_prompts-603f3273.pth" to /Users/robbw/.cache/pyiqa/CLIP-IQA+_learned_prompts-603f3273.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 16.7k/16.7k [00:00<00:00, 4.40MB/s] Testing clipiqa+ on cpu: 19%|ββββββββββββββββββββ | 14/74 [02:27<05:47, 5.78s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/CLIPIQA+_RN50_512-89f5d940.pth" to /Users/robbw/.cache/pyiqa/CLIPIQA+_RN50_512-89f5d940.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 309k/309k [00:00<00:00, 7.80MB/s] Loading pretrained model CLIPIQA from /Users/robbw/.cache/pyiqa/CLIPIQA+_RN50_512-89f5d940.pth | 0.00/309k [00:00<?, ?B/s] Testing clipiqa+_rn50_512 on cpu: 20%|ββββββββββββββββββββ | 15/74 [02:33<05:45, 5.86s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/CLIPIQA+_ViTL14_512-e66488f2.pth" to /Users/robbw/.cache/pyiqa/CLIPIQA+_ViTL14_512-e66488f2.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 463k/463k [00:00<00:00, 10.8MB/s] Loading pretrained model CLIPIQA from /Users/robbw/.cache/pyiqa/CLIPIQA+_ViTL14_512-e66488f2.pth | 0.00/463k [00:00<?, ?B/s] Testing clipiqa+_vitL14_512 on cpu: 22%|βββββββββββββββββββββ | 16/74 [03:17<16:49, 17.40s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/CNNIQA_koniq10k-e6f14c91.pth" to /Users/robbw/.cache/pyiqa/CNNIQA_koniq10k-e6f14c91.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2.79M/2.79M [00:00<00:00, 26.1MB/s] Loading pretrained model CNNIQA from /Users/robbw/.cache/pyiqa/CNNIQA_koniq10k-e6f14c91.pthββββββββββββββββββββββββββββββββ | 2.62M/2.79M [00:00<00:00, 26.6MB/s] Testing cw_ssim on cpu: 24%|ββββββββββββββββββββββββββ | 18/74 [03:24<09:34, 10.26s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/DBCNN_scnn-7ea73d75.pth" to /Users/robbw/.cache/pyiqa/DBCNN_scnn-7ea73d75.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2.47M/2.47M [00:00<00:00, 25.8MB/s] Loading pretrained model SCNN from /Users/robbw/.cache/pyiqa/DBCNN_scnn-7ea73d75.pth | 0.00/2.47M [00:00<?, ?B/s] Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/DBCNN_KonIQ10k-2de81c0a.pth" to /Users/robbw/.cache/pyiqa/DBCNN_KonIQ10k-2de81c0a.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 58.4M/58.4M [00:04<00:00, 12.3MB/s] Loading pretrained model DBCNN from /Users/robbw/.cache/pyiqa/DBCNN_KonIQ10k-2de81c0a.pthββββββββββββββββββββββββββββββββββββββββ | 57.8M/58.4M [00:04<00:00, 19.3MB/s] Testing dbcnn on cpu: 26%|ββββββββββββββββββββββββββββ | 19/74 [03:40<10:58, 11.98s/it]Loading pretrained model DISTS from /Users/robbw/.cache/pyiqa/DISTS_weights-f5e65c96.pth Testing gmsd on cpu: 31%|ββββββββββββββββββββββββββββββββββ | 23/74 [03:57<04:01, 4.74s/it]Loading pretrained model HyperNet from /Users/robbw/.cache/pyiqa/HyperIQA-resnet50-koniq10k-c96c41b1.pth Testing ilniqe on cpu: 34%|βββββββββββββββββββββββββββββββββββββ | 25/74 [04:19<06:29, 7.94s/it]Loading pretrained model MLP from /Users/robbw/.cache/pyiqa/sac%2Blogos%2Bava1-l14-linearMSE.pth 100%|βββββββββββββββββββββββββββββββββββββββ| 338M/338M [01:48<00:00, 3.27MiB/s] | 26/74 [04:25<05:49, 7.28s/it] Downloading: "https://github.com/zwx8981/IQA-PyTorch/releases/download/Weights/liqe_koniq.pt" to /Users/robbw/.cache/pyiqa/liqe_koniq.pt
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 337M/337M [00:26<00:00, 13.2MB/s] Loading pretrained model LIQE from /Users/robbw/.cache/pyiqa/liqe_koniq.ptβββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 334M/337M [00:26<00:00, 9.21MB/s] Generating text features for LIQE model, will be cached at /Users/robbw/.cache/pyiqa/liqe_text_feat.pt. Testing liqe on cpu: 36%|ββββββββββββββββββββββββββββββββββββββββ | 27/74 [06:47<37:26, 47.79s/it]Downloading: "https://github.com/zwx8981/IQA-PyTorch/releases/download/Weights/liqe_mix.pt" to /Users/robbw/.cache/pyiqa/liqe_mix.pt
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 337M/337M [00:26<00:00, 13.1MB/s] Loading pretrained model LIQE from /Users/robbw/.cache/pyiqa/liqe_mix.ptβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 334M/337M [00:26<00:00, 15.1MB/s] Generating text features for LIQE model, will be cached at /Users/robbw/.cache/pyiqa/liqe_text_feat_mix.pt. Testing liqe_mix on cpu: 38%|ββββββββββββββββββββββββββββββββββββββββ | 28/74 [07:33<36:09, 47.16s/it]Loading pretrained model LPIPS from /Users/robbw/.cache/pyiqa/LPIPS_v0.1_alex-df73285e.pth Testing lpips on cpu: 39%|βββββββββββββββββββββββββββββββββββββββββββ | 29/74 [07:35<25:14, 33.66s/it]Loading pretrained model LPIPS from /Users/robbw/.cache/pyiqa/LPIPS_v0.1_alex-df73285e.pth Testing lpips+ on cpu: 41%|ββββββββββββββββββββββββββββββββββββββββββββ | 30/74 [07:36<17:32, 23.93s/it]Downloading: "https://github.com/chaofengc/IQA-Toolbox-Python/releases/download/v0.1-weights/LPIPS_v0.1_vgg-a78928a0.pth" to /Users/robbw/.cache/pyiqa/LPIPS_v0.1_vgg-a78928a0.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 7.12k/7.12k [00:00<00:00, 2.75MB/s] Loading pretrained model LPIPS from /Users/robbw/.cache/pyiqa/LPIPS_v0.1_vgg-a78928a0.pth | 0.00/7.12k [00:00<?, ?B/s] Testing lpips-vgg on cpu: 42%|ββββββββββββββββββββββββββββββββββββββββββββ | 31/74 [07:51<15:12, 21.22s/it]Loading pretrained model LPIPS from /Users/robbw/.cache/pyiqa/LPIPS_v0.1_vgg-a78928a0.pth Testing mad on cpu: 45%|βββββββββββββββββββββββββββββββββββββββββββββββββ | 33/74 [08:59<20:11, 29.54s/it]Downloading: "https://github.com/IIGROUP/MANIQA/releases/download/Koniq10k/ckpt_koniq10k.pt" to /Users/robbw/.cache/pyiqa/ckpt_koniq10k.pt
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 518M/518M [00:41<00:00, 13.0MB/s] Loading pretrained model MANIQA from /Users/robbw/.cache/pyiqa/ckpt_koniq10k.ptβββββββββββββββββββββββββββββββββββββββββββββββββββββ| 517M/518M [00:41<00:00, 16.8MB/s] Testing maniqa on cpu: 46%|ββββββββββββββββββββββββββββββββββββββββββββββββββ | 34/74 [10:26<31:12, 46.81s/it]Downloading: "https://github.com/IIGROUP/MANIQA/releases/download/Kadid10k/ckpt_kadid10k.pt" to /Users/robbw/.cache/pyiqa/ckpt_kadid10k.pt
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 518M/518M [00:42<00:00, 12.9MB/s] Loading pretrained model MANIQA from /Users/robbw/.cache/pyiqa/ckpt_kadid10k.ptβββββββββββββββββββββββββββββββββββββββββββββββββββββ| 518M/518M [00:42<00:00, 19.8MB/s] Testing maniqa-kadid on cpu: 47%|ββββββββββββββββββββββββββββββββββββββββββββββββ | 35/74 [11:53<38:15, 58.86s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/MANIQA_PIPAL-ae6d356b.pth" to /Users/robbw/.cache/pyiqa/MANIQA_PIPAL-ae6d356b.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 518M/518M [00:38<00:00, 14.0MB/s] Loading pretrained model MANIQA from /Users/robbw/.cache/pyiqa/MANIQA_PIPAL-ae6d356b.pthββββββββββββββββββββββββββββββββββββββββββββ| 518M/518M [00:38<00:00, 16.1MB/s] Testing ms_ssim on cpu: 50%|βββββββββββββββββββββββββββββββββββββββββββββββββββββ | 37/74 [13:19<29:06, 47.20s/it]Loading pretrained model MUSIQ from /Users/robbw/.cache/pyiqa/musiq_koniq_ckpt-e95806b9.pth Testing musiq on cpu: 51%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 38/74 [13:23<20:27, 34.09s/it]Loading pretrained model MUSIQ from /Users/robbw/.cache/pyiqa/musiq_ava_ckpt-e8d3f067.pth Testing musiq-ava on cpu: 53%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 39/74 [13:27<14:34, 24.99s/it]Loading pretrained model MUSIQ from /Users/robbw/.cache/pyiqa/musiq_paq2piq_ckpt-364c0c84.pth Testing musiq-paq2piq on cpu: 54%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 40/74 [13:30<10:29, 18.52s/it]Loading pretrained model MUSIQ from /Users/robbw/.cache/pyiqa/musiq_spaq_ckpt-358bb6af.pth model.safetensors: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 224M/224M [00:10<00:00, 21.1MB/s] Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/NIMA_InceptionV2_ava-b0c77c00.pth" to /Users/robbw/.cache/pyiqa/NIMA_InceptionV2_ava-b0c77c00.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 208M/208M [00:14<00:00, 15.0MB/s] Loading pretrained model NIMA from /Users/robbw/.cache/pyiqa/NIMA_InceptionV2_ava-b0c77c00.pthββββββββββββββββββββββββββββββββββββββ| 207M/208M [00:14<00:00, 16.1MB/s] Testing nima on cpu: 57%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 42/74 [14:08<10:45, 20.18s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/NIMA_koniq-250367ae.pth" to /Users/robbw/.cache/pyiqa/NIMA_koniq-250367ae.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 208M/208M [00:15<00:00, 14.1MB/s] Loading pretrained model NIMA from /Users/robbw/.cache/pyiqa/NIMA_koniq-250367ae.pthββββββββββββββββββββββββββββββββββββββββββββββββ| 208M/208M [00:15<00:00, 17.0MB/s] Testing nima-koniq on cpu: 58%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 43/74 [14:26<10:07, 19.59s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/NIMA-spaq-46a7fcb7.pth" to /Users/robbw/.cache/pyiqa/NIMA-spaq-46a7fcb7.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 208M/208M [00:16<00:00, 13.0MB/s] Loading pretrained model NIMA from /Users/robbw/.cache/pyiqa/NIMA-spaq-46a7fcb7.pthββββββββββββββββββββββββββββββββββββββββββββββββ | 206M/208M [00:16<00:00, 13.2MB/s] model.safetensors: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 553M/553M [00:24<00:00, 23.0MB/s] Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/NIMA_VGG16_ava-dc4e8265.pth" to /Users/robbw/.cache/pyiqa/NIMA_VGG16_ava-dc4e8265.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 56.2M/56.2M [00:04<00:00, 13.0MB/s] Loading pretrained model NIMA from /Users/robbw/.cache/pyiqa/NIMA_VGG16_ava-dc4e8265.pthβββββββββββββββββββββββββββββββββββββββββ | 55.4M/56.2M [00:04<00:00, 14.2MB/s] Testing nrqm on cpu: 66%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 49/74 [16:20<09:15, 22.21s/it]Loading pretrained model PAQ2PIQ from /Users/robbw/.cache/pyiqa/P2P_RoIPoolModel-fit.10.bs.120-ca69882e.pth Testing pi on cpu: 69%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 51/74 [17:16<10:37, 27.73s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/PieAPPv0.1-0937b014.pth" to /Users/robbw/.cache/pyiqa/PieAPPv0.1-0937b014.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 261M/261M [00:22<00:00, 12.3MB/s] Loading pretrained model PieAPP from /Users/robbw/.cache/pyiqa/PieAPPv0.1-0937b014.pthββββββββββββββββββββββββββββββββββββββββββββββ| 260M/261M [00:22<00:00, 13.7MB/s] Testing ssimc on cpu: 77%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 57/74 [18:20<02:16, 8.02s/it]Downloading: "https://github.com/abhijay9/ShiftTolerant-LPIPS/raw/main/stlpips/weights/vST0.0/alex_shift_tolerant.pth" to /Users/robbw/.cache/pyiqa/alex_shift_tolerant.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 9.44M/9.44M [00:00<00:00, 28.1MB/s] Loading pretrained model STLPIPS from /Users/robbw/.cache/pyiqa/alex_shift_tolerant.pthββββββββββββββββββββββββββ | 8.12M/9.44M [00:00<00:00, 28.7MB/s] Testing stlpips on cpu: 78%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 58/74 [18:30<02:14, 8.43s/it]Downloading: "https://github.com/abhijay9/ShiftTolerant-LPIPS/raw/main/stlpips/weights/vST0.0/vgg_shift_tolerant.pth" to /Users/robbw/.cache/pyiqa/vgg_shift_tolerant.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 56.2M/56.2M [00:01<00:00, 31.7MB/s] Loading pretrained model STLPIPS from /Users/robbw/.cache/pyiqa/vgg_shift_tolerant.pthβββββββββββββββββββββββββββββββββββββββββββ | 55.8M/56.2M [00:01<00:00, 32.4MB/s] Testing stlpips-vgg on cpu: 80%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 59/74 [18:57<03:31, 14.08s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_fr_kadid_res50-2c4cc61d.pth" to /Users/robbw/.cache/pyiqa/cfanet_fr_kadid_res50-2c4cc61d.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 138M/138M [00:09<00:00, 16.0MB/s] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_fr_kadid_res50-2c4cc61d.pthββββββββββββββββββββββββββββββββββ | 136M/138M [00:08<00:00, 14.9MB/s] Testing topiq_fr on cpu: 81%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 60/74 [19:15<03:33, 15.28s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_fr_pipal_res50-69bbe5ba.pth" to /Users/robbw/.cache/pyiqa/cfanet_fr_pipal_res50-69bbe5ba.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 138M/138M [00:11<00:00, 13.1MB/s] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_fr_pipal_res50-69bbe5ba.pthβββββββββββββββββββββββββββββββββββ| 137M/138M [00:10<00:00, 21.1MB/s] Testing topiq_fr-pipal on cpu: 82%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 61/74 [19:34<03:34, 16.49s/it]/Users/robbw/miniforge3/lib/python3.10/site-packages/torch/functional.py:512: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:3588.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_iaa_ava_swin-393b41b4.pth Testing topiq_iaa on cpu: 84%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 62/74 [19:37<02:27, 12.29s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_iaa_ava_res50-3cd62bb3.pth" to /Users/robbw/.cache/pyiqa/cfanet_iaa_ava_res50-3cd62bb3.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 280M/280M [00:20<00:00, 14.1MB/s] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_iaa_ava_res50-3cd62bb3.pthββββββββββββββββββββββββββββββββββββ| 280M/280M [00:20<00:00, 13.7MB/s] Testing topiq_iaa_res50 on cpu: 85%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 63/74 [20:01<02:55, 15.93s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_nr_koniq_res50-9a73138b.pth" to /Users/robbw/.cache/pyiqa/cfanet_nr_koniq_res50-9a73138b.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 173M/173M [00:11<00:00, 15.5MB/s] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_nr_koniq_res50-9a73138b.pthββββββββββββββββββββββββββββββββββ | 171M/173M [00:11<00:00, 15.5MB/s] Testing topiq_nr on cpu: 86%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 64/74 [20:19<02:45, 16.55s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_nr_flive_res50-ded1c74e.pth" to /Users/robbw/.cache/pyiqa/cfanet_nr_flive_res50-ded1c74e.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 173M/173M [00:12<00:00, 14.1MB/s] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_nr_flive_res50-ded1c74e.pthβββββββββββββββββββββββββββββββββββ| 172M/173M [00:12<00:00, 13.0MB/s] Testing topiq_nr-flive on cpu: 88%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 65/74 [20:38<02:35, 17.30s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/cfanet_nr_spaq_res50-a7f799ac.pth" to /Users/robbw/.cache/pyiqa/cfanet_nr_spaq_res50-a7f799ac.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 173M/173M [00:14<00:00, 12.8MB/s] Loading pretrained model CFANet from /Users/robbw/.cache/pyiqa/cfanet_nr_spaq_res50-a7f799ac.pthββββββββββββββββββββββββββββββββββ | 170M/173M [00:14<00:00, 10.7MB/s] Testing topiq_nr-spaq on cpu: 89%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 66/74 [20:59<02:26, 18.27s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/tres_koniq-f0502926.pth" to /Users/robbw/.cache/pyiqa/tres_koniq-f0502926.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 582M/582M [00:41<00:00, 14.8MB/s] Loading pretrained model TReS from /Users/robbw/.cache/pyiqa/tres_koniq-f0502926.pthββββββββββββββββββββββββββββββββββββββββββββββββ| 581M/582M [00:41<00:00, 14.6MB/s] Testing tres on cpu: 91%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 67/74 [22:26<04:32, 38.96s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/tres_flive-09b0de5b.pth" to /Users/robbw/.cache/pyiqa/tres_flive-09b0de5b.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 582M/582M [00:43<00:00, 14.1MB/s]
Loading pretrained model TReS from /Users/robbw/.cache/pyiqa/tres_flive-09b0de5b.pthββββββββββββββββββββββββββββββββββββββββββββββββ| 582M/582M [00:43<00:00, 12.7MB/s]
Testing tres-flive on cpu: 92%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 68/74 [23:55<05:24, 54.05s/it]/Users/robbw/miniforge3/lib/python3.10/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
warnings.warn(
/Users/robbw/miniforge3/lib/python3.10/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=ResNet34_Weights.IMAGENET1K_V1. You can also use weights=ResNet34_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
Downloading: "https://download.pytorch.org/models/resnet34-b627a593.pth" to /Users/robbw/.cache/torch/hub/checkpoints/resnet34-b627a593.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 83.3M/83.3M [00:03<00:00, 26.6MB/s]
Downloading: "https://github.com/zwx8981/IQA-PyTorch/releases/download/Weights/UNIQUE.pt" to /Users/robbw/.cache/pyiqa/UNIQUE.ptββ| 83.0M/83.3M [00:03<00:00, 29.1MB/s]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 85.3M/85.3M [00:06<00:00, 13.3MB/s] Loading pretrained model UNIQUE from /Users/robbw/.cache/pyiqa/UNIQUE.ptβββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 84.1M/85.3M [00:06<00:00, 11.9MB/s] Testing unique on cpu: 93%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 69/74 [24:10<03:31, 42.27s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/URanker_ckpt-450eb36d.pth" to /Users/robbw/.cache/pyiqa/URanker_ckpt-450eb36d.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 82.5M/82.5M [00:09<00:00, 9.44MB/s] Loading pretrained model URanker from /Users/robbw/.cache/pyiqa/URanker_ckpt-450eb36d.pthβββββββββββββββββββββββββββββββββββββββββ| 82.1M/82.5M [00:09<00:00, 9.38MB/s] Testing vsi on cpu: 97%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 72/74 [24:39<00:38, 19.18s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/WaDIQaM-kadid-f7541ea5.pth" to /Users/robbw/.cache/pyiqa/WaDIQaM-kadid-f7541ea5.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 24.0M/24.0M [00:01<00:00, 14.1MB/s] Loading pretrained model WaDIQaM from /Users/robbw/.cache/pyiqa/WaDIQaM-kadid-f7541ea5.pthββββββββββββββββββββββββββββββ | 22.0M/24.0M [00:01<00:00, 11.4MB/s] Testing wadiqam_fr on cpu: 99%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 73/74 [24:48<00:16, 16.22s/it]Downloading: "https://github.com/chaofengc/IQA-PyTorch/releases/download/v0.1-weights/WaDIQaM-NR-koniq-aaffea29.pth" to /Users/robbw/.cache/pyiqa/WaDIQaM-NR-koniq-aaffea29.pth
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 20.0M/20.0M [00:01<00:00, 16.2MB/s] Loading pretrained model WaDIQaM from /Users/robbw/.cache/pyiqa/WaDIQaM-NR-koniq-aaffea29.pthβββββββββββββββββββββ | 17.4M/20.0M [00:01<00:00, 14.5MB/s] Testing wadiqam_nr on cpu: 100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 74/74 [24:54<00:00, 20.20s/it] CPU only pytest tests/ -m calibration -v ============================================================================================================================= test session starts ============================================================================================================================== platform darwin -- Python 3.10.14, pytest-8.3.3, pluggy-1.5.0 -- /Users/robbw/miniforge3/bin/python3.10 cachedir: .pytest_cache rootdir: /Users/robbw/XPLAY/IQA-PyTorch plugins: anyio-4.1.0, typeguard-4.3.0 collected 268 items / 240 deselected / 28 selected
tests/test_metric_general.py::test_match_official_with_given_cases[brisque] FAILED [ 3%] tests/test_metric_general.py::test_match_official_with_given_cases[brisque_matlab] PASSED [ 7%] tests/test_metric_general.py::test_match_official_with_given_cases[ckdn] PASSED [ 10%] tests/test_metric_general.py::test_match_official_with_given_cases[cw_ssim] PASSED [ 14%] tests/test_metric_general.py::test_match_official_with_given_cases[dists] PASSED [ 17%] tests/test_metric_general.py::test_match_official_with_given_cases[fsim] PASSED [ 21%] tests/test_metric_general.py::test_match_official_with_given_cases[gmsd] PASSED [ 25%] tests/test_metric_general.py::test_match_official_with_given_cases[lpips] PASSED [ 28%] tests/test_metric_general.py::test_match_official_with_given_cases[mad] PASSED [ 32%] tests/test_metric_general.py::test_match_official_with_given_cases[ms_ssim] PASSED [ 35%] tests/test_metric_general.py::test_match_official_with_given_cases[niqe] PASSED [ 39%] tests/test_metric_general.py::test_match_official_with_given_cases[niqe_matlab] PASSED [ 42%] tests/test_metric_general.py::test_match_official_with_given_cases[piqe] PASSED [ 46%] tests/test_metric_general.py::test_match_official_with_given_cases[nlpd] PASSED [ 50%] tests/test_metric_general.py::test_match_official_with_given_cases[psnr] PASSED [ 53%] tests/test_metric_general.py::test_match_official_with_given_cases[ssim] PASSED [ 57%] tests/test_metric_general.py::test_match_official_with_given_cases[vif] PASSED [ 60%] tests/test_metric_general.py::test_match_official_with_given_cases[vsi] PASSED [ 64%] tests/test_metric_general.py::test_match_official_with_given_cases[nrqm] PASSED [ 67%] tests/test_metric_general.py::test_match_official_with_given_cases[pi] PASSED [ 71%] tests/test_metric_general.py::test_match_official_with_given_cases[ilniqe] PASSED [ 75%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq] PASSED [ 78%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq-ava] PASSED [ 82%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq-paq2piq] PASSED [ 85%] tests/test_metric_general.py::test_match_official_with_given_cases[musiq-spaq] PASSED [ 89%] tests/test_metric_general.py::test_match_official_with_given_cases[paq2piq] PASSED [ 92%] tests/test_metric_general.py::test_match_official_with_given_cases[entropy] PASSED [ 96%] tests/test_metric_general.py::test_match_official_with_given_cases[laion_aes] PASSED [100%]
=================================================================================================================================== FAILURES =================================================================================================================================== ____ test_match_official_with_givencases[brisque] ____
ref_img = tensor([[[[0.5882, 0.5882, 0.6431, ..., 0.4157, 0.4157, 0.4039], [0.5725, 0.5686, 0.5686, ..., 0.4078, 0.4...78, 0.6275, 0.6353, ..., 0.1804, 0.1059, 0.0941], [0.6157, 0.6392, 0.6549, ..., 0.2706, 0.2627, 0.1765]]]]) dist_img = tensor([[[[0.6314, 0.6314, 0.6275, ..., 0.3961, 0.3961, 0.3961], [0.6314, 0.6314, 0.6275, ..., 0.3961, 0.3...59, 0.7216, 0.6941, ..., 0.1176, 0.1176, 0.1176], [0.5059, 0.7216, 0.6941, ..., 0.1176, 0.1176, 0.1176]]]]) metric_name = 'brisque', device = device(type='cpu')
@pytest.mark.calibration
@pytest.mark.parametrize(
("metric_name"),
[(k) for k in metrics_with_official_results().keys()]
)
def test_match_official_with_given_cases(ref_img, dist_img, metric_name, device):
official_result = metrics_with_official_results()[metric_name]
metric = pyiqa.create_metric(metric_name, device=device)
score = metric(dist_img, ref_img)
# save results
cal_sum = pd.read_csv(CALBR_SUMMARY_FILE, index_col='Method')
cal_sum.loc[metric_name] = [f'{item:.4f}' for item in official_result.tolist()]
cal_sum.loc[metric_name + '(ours)'] = [f'{item:.4f}' for item in score.squeeze().cpu().numpy().tolist()]
cal_sum = cal_sum.sort_values(by=['Method'], ascending=True)
cal_sum.to_csv(CALBR_SUMMARY_FILE)
if metric_name in TOL_DICT.keys():
atol, rtol = TOL_DICT[metric_name]
else:
atol, rtol = ATOL, RTOLassert torch.allclose(score.squeeze(), torch.from_numpy(official_result).to(score), atol=atol, rtol=rtol), \f"Metric {metric_name} results mismatch with official results." E AssertionError: Metric brisque results mismatch with official results. E assert False E + where False = <built-in method allclose of type object at 0x10614b778>(tensor([94.6418, -0.1131, 1.0842, 5.1447, 66.8351]), tensor([94.6421, -0.1076, 0.9929, 5.3583, 72.2617]), atol=0.01, rtol=0.08) E + where <built-in method allclose of type object at 0x10614b778> = torch.allclose E + and tensor([94.6418, -0.1131, 1.0842, 5.1447, 66.8351]) = <built-in method squeeze of Tensor object at 0x16c40cf40>() E + where <built-in method squeeze of Tensor object at 0x16c40cf40> = tensor([94.6418, -0.1131, 1.0842, 5.1447, 66.8351]).squeeze E + and tensor([94.6421, -0.1076, 0.9929, 5.3583, 72.2617]) = <built-in method to of Tensor object at 0x16c40ee30>(tensor([94.6418, -0.1131, 1.0842, 5.1447, 66.8351])) E + where <built-in method to of Tensor object at 0x16c40ee30> = tensor([94.6421, -0.1076, 0.9929, 5.3583, 72.2617], dtype=torch.float64).to E + where tensor([94.6421, -0.1076, 0.9929, 5.3583, 72.2617], dtype=torch.float64) = <built-in method from_numpy of type object at 0x10614b778>(array([94.6421 , -0.107618, 0.992889, 5.35827 , 72.2617 ])) E + where <built-in method from_numpy of type object at 0x10614b778> = torch.from_numpy
tests/test_metric_general.py:86: AssertionError
tests/test_metric_general.py::test_match_official_with_given_cases[laion_aes] /Users/robbw/XPLAY/IQA-PyTorch/tests/test_metric_general.py:77: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '4.6458' has dtype incompatible with float64, please explicitly cast to a compatible dtype first. cal_sum.loc[metric_name] = [f'{item:.4f}' for item in official_result.tolist()]
tests/test_metric_general.py::test_match_official_with_given_cases[laion_aes] /Users/robbw/XPLAY/IQA-PyTorch/tests/test_metric_general.py:77: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value '3.0889' has dtype incompatible with float64, please explicitly cast to a compatible dtype first. cal_sum.loc[metric_name] = [f'{item:.4f}' for item in official_result.tolist()]
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html =========================================================================================================================== short test summary info ============================================================================================================================ FAILED tests/test_metric_general.py::test_match_official_with_given_cases[brisque] - AssertionError: Metric brisque results mismatch with official results. ==================================================================================================== 1 failed, 27 passed, 240 deselected, 149 warnings in 77.60s (0:01:17) ===================================================================================================== make: *** [test_cal] Error 1
chaofengc
commented
2 months ago Thank you for your help! Could you please attach the results file? It should be located at tests/Efficiency_benchmark.csv.
Regarding the new work you mentioned, I'll review it when I have the time.
RobBW
commented
2 months ago Herewith. Regards Rob
@.***
οΏΌ
On 18 Sep 2024, at 16:46, Chaofeng Chen @.***> wrote:
Thank you for your help! Could you please attach the results file? It should be located at ResultsCalibra/calibration_summary.csv.
Regarding the new work you mentioned, I'll review it when I have the time.
β Reply to this email directly, view it on GitHub https://github.com/chaofengc/IQA-PyTorch/issues/190#issuecomment-2357486382, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRNQJS6O4RZGVF2ZMFZVHLZXEARRAVCNFSM6AAAAABOGPFB56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJXGQ4DMMZYGI. You are receiving this because you authored the thread.
chaofengc
commented
2 months ago Thank you for your help! However, Iβm unable to see the attached file. Could you please upload the result directly here?
RobBW
commented
2 months ago This time?
chaofengc
commented
2 months ago Apologies for providing the incorrect file path earlier, this is the results calibration file. The time benchmark results can be found in tests/Efficiency_benchmark.csv
RobBW
commented
2 months ago No problem: Efficiency_benchmark.csv
chaofengc
commented
2 months ago That's it! Thank you so much!
RobBW
commented
2 months ago I've been looking at: https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html Are there any bottleneck points you can pinpoint that would benefit from using torch.compile? If yes I'd be interested to test them for you.
chaofengc
commented
2 months ago Thank you for your suggestions. It is indeed possible to further accelerate certain pure deep neural network metrics using torch.compile. However, this process may take some time, as different metrics could require distinct modifications. Since our goal is not to provide production-level code, the current GPU performance (all metrics <1s for a $1080\times800$ image on a V100) is largely sufficient for research purposes.
RobBW
commented
2 months ago Thanks for your explanation.
On Fri, 20 Sep 2024 at 19:32, Chaofeng Chen @.***> wrote:
Thank you for your suggestions. It is indeed possible to further accelerate certain pure deep neural network metrics using torch.compile. However, this process may take some time, as different metrics could require distinct modifications. Since our goal is not to provide production-level code, the current GPU performance (all metrics <1s for a $1080\times800$ image on a V100) is largely sufficient for research purposes.
β Reply to this email directly, view it on GitHub https://github.com/chaofengc/IQA-PyTorch/issues/190#issuecomment-2363024022, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABRNQJUXTYCHM4QZA6T3VSDZXPFRFAVCNFSM6AAAAABOGPFB56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRTGAZDIMBSGI . You are receiving this because you authored the thread.Message ID: @.***>
I have just installed IQA-PyTorch on my M2 Mac mini and subjectively it seems to run quite slowly. On examining your code I note that there are no device specification for "mps". Before I launch any experimentation I am asking you whether there is likely to be any performance benefit in adding .device("mps") to these files: