InterDigitalInc / CompressAI

A PyTorch library and evaluation platform for end-to-end compression research
https://interdigitalinc.github.io/CompressAI/
BSD 3-Clause Clear License
1.21k stars 232 forks source link

Can't reproduce HM benchmark, lower than reported. #263

Open jiehuang165 opened 12 months ago

jiehuang165 commented 12 months ago

Bug

Try to reproduce the HM benchmark, but "psnr-rgb" is quite lower than reported.

To Reproduce

Steps to reproduce the behavior:

  1. git clone this repo
  2. dowload and compile HM, the version is 'HM-16.20+SCM-8.8' same as https://github.com/InterDigitalInc/CompressAI/blob/a4ae2eeef7bdb1b84ba076ac0d650b523f3fa882/examples/run-benchmarks.sh#L88
  3. run benchmark script ./examples/run-benchmarks.sh ~/dataset/kodak hm

Expected behavior

Expect to reproduct kodak/hm.json

However, I get following results:

{
  "name": "my-HM",
  "description": "HM",
  "results": {
    "bpp": [
      3.845564948187935,
      2.4760615030924478,
      1.559600830078125,
      0.9503741794162326,
      0.5295282999674479,
      0.26666259765625,
      0.12086995442708333,
      0.05143822564019097
    ],
    "encoding_time": [
      14.452731589476267,
      10.646501451730728,
      14.093859483798346,
      12.758554518222809,
      11.680685818195343,
      12.660903205474218,
      8.8968399365743,
      8.146473407745361
    ],
    "decoding_time": [
      0.7784863809744517,
      0.3728034794330597,
      0.3049040138721466,
      0.38336031635602313,
      0.4734913905461629,
      0.19524875283241272,
      0.3037620981534322,
      0.24185975392659506
    ],
    "psnr-rgb": [
      42.922397886745436,
      41.28368055742206,
      39.05052577559535,
      36.308070045031265,
      33.27506657838453,
      30.333590592627463,
      27.68549982125182,
      25.38004455197402
    ],
    "ms-ssim-rgb": [
      0.9981328224142393,
      0.9964238107204437,
      0.9932830135027567,
      0.9872708593805631,
      0.974710909028848,
      0.949935294687748,
      0.9075854445497195,
      0.8457145740588506
    ]
  }
}

After plotting bitrate-psnr curve of my reproduce and official results, the difference is obvious especially when bitrate is large. hm-bitrate-curve

Environment

- PyTorch / CompressAI Version (1.10.1+cu111 / 1.2.4.dev0):
- OS (e.g., Linux): Ubuntu 20.04.3
- How you installed PyTorch / CompressAI (`pip`, source): pip install -e .
- Build command you used (if compiling from source):
- Python version: 3.8
- CUDA/cuDNN version: 11.1