search

NeuroBench / neurobench

Benchmark harness and baseline results for the NeuroBench algorithm track.
https://neurobench.readthedocs.io
Apache License 2.0
53 stars 12 forks source link

Staging area for new tasks #68

Closed jasonlyik closed 4 months ago

jasonlyik commented 1 year ago

Organization - add staging area / documentation for new tasks which are built by community, but not necessarily part of a NeuroBench version (which will be the officially reported benchmarks). Staged tasks can make their way into a version through some process, maybe some vote from the community or some committee review.

New tasks should be implemented just like the existing tasks.

korneelf1 commented 12 months ago

Implement as last task, will change file structure

jasonlyik commented 4 months ago

Going forward, there won't be any difference in the harness between "official" and "unofficial" tasks. If you want to submit a new benchmark, i.e. dataset / metrics, etc., just submit a PR and it will be incorporated.