search

NeuroBench / neurobench

Benchmark harness and baseline results for the NeuroBench algorithm track.
https://neurobench.readthedocs.io
Apache License 2.0
47 stars 12 forks source link

LSTMs are not supported by `single_layer_MACs` function #147

Closed V0XNIHILI closed 2 months ago

V0XNIHILI commented 7 months ago

See description! Error that I received:

Traceback (most recent call last):
  File "/space/ddenblanken/Projects/neurobench/neurobench/examples/mswc_fscil/mswc_fscil.py", line 222, in <module>
    pre_train_results = benchmark_all_test.run(postprocessors=[out_mask, out2pred, torch.squeeze])
  File "/space/ddenblanken/Projects/neurobench/neurobench/benchmarks/benchmark.py", line 85, in run
    batch_results[m] = self.data_metrics[m](self.model, preds, data)
  File "/space/ddenblanken/Projects/neurobench/neurobench/benchmarks/data_metrics.py", line 206, in __call__
    operations, spiking = single_layer_MACs(spikes, hook.layer)
  File "/space/ddenblanken/Projects/neurobench/neurobench/benchmarks/utils/metric_utils.py", line 208, in single_layer_MACs
    raise 'This layer is not yet supported by NeuroBench.'
TypeError: exceptions must derive from BaseException
V0XNIHILI commented 7 months ago

Possibly we can either depend on (but is not really maintained anymore) or copy code from: https://github.com/Lyken17/pytorch-OpCounter for some common missing modules.

korneelf1 commented 4 months ago

Currently, LSTMs can be benchmarked by using lstmcells rather than lstm blocks