InterDigitalInc / CompressAI

A PyTorch library and evaluation platform for end-to-end compression research
https://interdigitalinc.github.io/CompressAI/
BSD 3-Clause Clear License
1.19k stars 232 forks source link

fix some bugs in bench #214

Closed Do1e closed 1 year ago

Do1e commented 1 year ago

When I used bash examples/run-benchmarks.sh {dataset} jpeg2000 to run benchmark, I found there are two questions in bench.

  1. run-benchmarks.sh uses seq to set qps, so args.qps should be a list in compressai/utils/bench/__main__.py
  2. __main__.py raises TypeError: Can't instantiate abstract class JPEG2000 with abstract method setup_args in line 160: codec = codec_cls(args). It means I was trying to create an instance of an abstract class that has abstract methods not implemented by its subclass. I checked the code and found that the function setup_args was not implemented.
  3. Type of quality in bpg codec is str. We must convert it to int before comparing.
YodaEmbedding commented 1 year ago

Looks like the current --qps should work. It's a comma separated list:

--qps=1,2,3,4

Perhaps we can add this to the help string or update any incorrect documentation, and update any usages of qps in run_benchmarks.sh.

IIRC, the behavior was changed to avoid issues w.r.t. argument order, e.g. --qps 42 /dataset/path. https://github.com/InterDigitalInc/CompressAI/issues/171

Do1e commented 1 year ago

Perhaps we can add this to the help string or update any incorrect documentation, and update any usages of qps in run_benchmarks.sh.

You are right. I revert __main__.py and fix it in run_benchmarks.sh.