search

undertherain / benchmarker

modular framework for [not only] deep learning performance benchmarking
http://blackbird.pw/performance
Mozilla Public License 2.0
9 stars 5 forks source link

Add flop per joule #140

Closed vatai closed 3 years ago

vatai commented 3 years ago

Not sure where should gflop_per_joule be put. Now it is in the "root" of result i.e. result["gflop_per_second"], but not sure if it should be in result["power"]["gflop_per_joule"]???

thinking

vatai commented 3 years ago

And I'm soooo happy I found out I can put images and gifs in the PRs!!!! Does it show???

And I also found this marvellous FF gif extension! https://addons.mozilla.org/en-US/firefox/addon/gifs-for-github/

undertherain commented 3 years ago

what's up?

undertherain commented 3 years ago

copying from telegram

  1. Most important is to get flop_per_sec

  2. let's keep measured flop in int problem - to keep in consistent with estimated flop

  3. let's check if astimated flop are available - well, here it is confusing anit, because if we have both estimated and measure flop - we have to decide which flop/sec to report

  4. you assume poewr measurement is always there - so it fails when power is not available 

[0]blackbird@sugi:~/Projects_heavy/performance/benchmarker$  [add-flop-per-joule|…13]$ python3 -m benchmarker --framework=pytorch --problem=resnet50 --problem_size=32 --batch_size=8 --nb_epoch=1 --precision=FP32 --flops
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/blackbird/Projects_heavy/performance/benchmarker/benchmarker/__main__.py", line 77, in <module>
    main()
  File "/home/blackbird/Projects_heavy/performance/benchmarker/benchmarker/__main__.py", line 71, in main
    result["gflop_per_joule"] /= float(result["power"]["joules_total"])
ZeroDivisionError: float division by zero