Open jiehuang165 opened 9 months ago
Try to reproduce the HM benchmark, but "psnr-rgb" is quite lower than reported.
Steps to reproduce the behavior:
./examples/run-benchmarks.sh ~/dataset/kodak hm
Expect to reproduct kodak/hm.json
However, I get following results:
{ "name": "my-HM", "description": "HM", "results": { "bpp": [ 3.845564948187935, 2.4760615030924478, 1.559600830078125, 0.9503741794162326, 0.5295282999674479, 0.26666259765625, 0.12086995442708333, 0.05143822564019097 ], "encoding_time": [ 14.452731589476267, 10.646501451730728, 14.093859483798346, 12.758554518222809, 11.680685818195343, 12.660903205474218, 8.8968399365743, 8.146473407745361 ], "decoding_time": [ 0.7784863809744517, 0.3728034794330597, 0.3049040138721466, 0.38336031635602313, 0.4734913905461629, 0.19524875283241272, 0.3037620981534322, 0.24185975392659506 ], "psnr-rgb": [ 42.922397886745436, 41.28368055742206, 39.05052577559535, 36.308070045031265, 33.27506657838453, 30.333590592627463, 27.68549982125182, 25.38004455197402 ], "ms-ssim-rgb": [ 0.9981328224142393, 0.9964238107204437, 0.9932830135027567, 0.9872708593805631, 0.974710909028848, 0.949935294687748, 0.9075854445497195, 0.8457145740588506 ] } }
After plotting bitrate-psnr curve of my reproduce and official results, the difference is obvious especially when bitrate is large.
- PyTorch / CompressAI Version (1.10.1+cu111 / 1.2.4.dev0): - OS (e.g., Linux): Ubuntu 20.04.3 - How you installed PyTorch / CompressAI (`pip`, source): pip install -e . - Build command you used (if compiling from source): - Python version: 3.8 - CUDA/cuDNN version: 11.1
Bug
Try to reproduce the HM benchmark, but "psnr-rgb" is quite lower than reported.
To Reproduce
Steps to reproduce the behavior:
./examples/run-benchmarks.sh ~/dataset/kodak hm
Expected behavior
Expect to reproduct kodak/hm.json
However, I get following results:
After plotting bitrate-psnr curve of my reproduce and official results, the difference is obvious especially when bitrate is large.![hm-bitrate-curve](https://github.com/InterDigitalInc/CompressAI/assets/82852936/68b13869-0aa2-4d68-ad9f-e524d3aba1b0)
Environment