search

NeuroBench / neurobench

Benchmark harness and baseline results for the NeuroBench algorithm track.
https://neurobench.readthedocs.io
Apache License 2.0
52 stars 12 forks source link

Model size metric #109

Closed jasonlyik closed 11 months ago

jasonlyik commented 11 months ago

Model size should be renamed to footprint.

This metric should not count any parameters with zero value, because on deployment to neuromorphic hardware these would simply not be added.

Side note, but maybe we could calculate connection sparsity via num zero params / total number of params, without looping through the model's layers? --> what is included in params which is not included in the connection layers?

jasonlyik commented 11 months ago

This metric should instead by model footprint on disk, in the particular pytorch format.

If we were to say that the ESN is "smaller", point to the connection sparsity, it would be better supported when deployed to hardware that can take advantage of sparse connection.