Remove two-level directories for benchmarks and add the dwarf that a benchmark belongs to a meta-file inside the benchmark

[sujaykathrotia]

+1

We already have a README files for each benchmark. We can mention the dwarf there.

[elavoie]

Hey Sujay! I suggest we put it in a JSON file so that as the suite grows, we can run tools to automatically filter on characteristics (dwarf, language supported, etc.) of benchmarks.

See https://github.com/Sable/Ostrich/issues/27

[sujaykathrotia]

Okay so the meta-file will be at the top level directory, not inside benchmark.

[elavoie]

Why? I believe every benchmark should have one.

[sujaykathrotia]

A top level JSON containing all the benchmarks makes more sense than individual JSON in each benchmark directory. You can read the JSON file initially and create tasks to compile and run benchmarks in one go.

[gnuvince]

So what's the plan exactly? Just have directories like bfs, crc, etc. and either through an information at the top level or within that directory say that it's a graph-traversal or combinatorics dwarf?

[elavoie]

@gnuvince, yes my initial proposal is to have 1) stable: directory for stable fully integrated benchmarks that are up to the quality standard of the suite (self-checking test, multiple coherent language implementations, fast running times, etc.), starting with the current 12 dwarfs 2) unstable: staging directory for benchmarks that are not fully integrated yet but have shown useful in the past or maybe useful in the future (other various benchmarks lab members have collected in the past, COMP 621 benchmarks, etc.) but would benefit from all being in a single place

Each benchmark will have its own directory and may draw from common stuff provided by the suite. To minimize redundancy between each benchmark implementation and minimize the number of implicit assumptions behind the structure of benchmark implementations, I will factorizing common parts away.

I may or may not decide to introduce a dependency system on common modules to simplify extending the suite with new benchmarks. I will do a first pass on the current Ostrich benchmarks and if I end up doing tedious work, I'll do it.

@sujaykathrotia I would like people to have each benchmark stand on its own so that people can cherrypick the benchmarks they like simply by copying the common Makefile/drivers/utilities, and keep only the directories of the benchmarks they care about, without having to edit the top-level JSON file. It means other tools will have to scan through the directories to assemble the information for the entire collection. I want to prioritize convenience for users of the suite, rather than the tool writers.

[elavoie]

Thinking through it, I am considering migrating benchmarks such that they each have their own github repository and adhere to a set of conventions set by the suite. That should make the toolchain and the basic infrastructure of the suite reusable beyond Ostrich as well as allowing users to configure the collection of benchmarks to their needs simply by changing the list of references to github repos.

Think of it like each benchmark is a node module.

[sujaykathrotia]

I think a repo for each benchmark would be very complicated. Users will have to download all the repos before starting the benchmarking. In case of node modules, you can require them just by using npm install command. You will have to create a tool which does the same for the benchmarks but is it worth the trouble?

[elavoie]

Git has a notion of sub repos. I'll look into it this afternoon and evening.

On Monday, November 9, 2015, Sujay Kathrotia notifications@github.com wrote:

I think a repo for each benchmark would be very complicated. Users will have to download all the repos before starting the benchmarking. In case of node modules, you can require them just by using npm install command. You will have to create a tool which does the same for the benchmarks but is it worth the trouble?

— Reply to this email directly or view it on GitHub https://github.com/Sable/Ostrich/issues/10#issuecomment-155165865.

[gnuvince]

I don't like the idea of splitting Ostrich into a bunch of loosely coupled repositories. To me, Ostrich is not a meta-framework for aggregating and running benchmarks, it is a very specific selection of benchmarks which happens to include a little infrastructure for running them. I feel strongly that splitting the benchmarks into their own repositories would increase the difficulty of running the benchmarks for users, which goes against the goal of Erick of prioritizing the convenience of the user (i.e. they'll need to know to git clone 12 extra repos or put URLs of those repos in some config file, which is objectively harder than just doing git clone ostrich && cd ostrich && make), but it would also make modifications harder than necessary for us.

If I had to describe my own vision for the organization of Ostrich, it would be a simple directory structure with a global Makefile, a directory containing the necessary dependencies (e.g. a runner script for JavaScript) and the benchmarks reduced to an information file (containing information such as dwarf, but also default run-time parameters) and very simple code structure. Of course, I haven't implemented it, so issues would surely be found, but the core idea is that once somebody checks out the repository, running make is the only step necessary for running the benchmarks.

    Makefile
    dependencies/
     |-- runner.py
    bfs/
     | info.json
     |-- c
         |-- main.c
         |-- core.c
     |-- matlab
         |-- main.m
         |-- core.m
    crc/
     | info.json
     |-- c
         |-- main.c
         |-- core.c
     |-- matlab
         |-- main.m
         |-- core.m
    nqueens/
     | info.json
     |-- c
         |-- main.c
         |-- core.c
     |-- matlab
         |-- main.m
         |-- core.m

[elavoie]

I am not going to argue different visions of organizing things here as they are to some extend somewhat arbitrary, the most important thing is that a particular vision is applied consistently. I will do a first pass on a couple of benchmarks, extract the key ideas behind the organization, do a presentation in front of the group to gather feedback, and improve it.

Otherwise, I am keeping the final call on decisions since nobody else has decided to bite the bullet until now (in 25 years Laurie tells me...) and I will be the one maintaining the thing for the next 2 years. Once I have a working proposal, I will check with other people and potential users whether its simple enough and will refine it from there.

I do not want to get stuck on bike shed design discussions... You guys are encouraged to provide comments on the ideas I am presenting here but I would like you to focus these comments on things I might have overlooked in terms of use cases, especially regarding making your own life simple as a compiler writer.

However, please refrain from arguing a particular organization over another one unless you intend to fully validate your idea by doing the work of refactoring the benchmark suite and support everybody else's use cases. Instead, please extract from your idea the underlying need you need to be met and I'll try my best to accomodate it while meeting all other requirements for the suite and other people use cases.

[elavoie]

See the current version of the 'matlab-version' branch to see how it works. The benchmarks are part of the suite but each can also have their own repository in the future as long as the directory structure and the configuration file 'benchmark.json' structure is followed. The dwarf category is now a property in the configuration file. See the lud benchmark as an example.