Add reference output - Githubissues

frankmcsherry / dynamic-datalog

Engines, queries, and data for dynamic Datalog computation

121 stars 8 forks source link

Add reference output #1

Open ryzhyk opened 5 years ago

ryzhyk commented 5 years ago

Any chance we could ask benchmark developers to add reference output for each of the three benchmarks? If this is difficult, then at least knowing the size of output relations would help with debugging.

frankmcsherry commented 5 years ago

Can do. The doop and galen ones have clear output sizes. The crdt one is more awkward in that it only produces the "winning edit locations" not the characters that are there, but I'll put something together.

In the cases that Souffle runs (doop and galen) I confirmed that it produced the same numbers of output tuples.

frankmcsherry commented 5 years ago

I think this is at least partially addressed with https://github.com/frankmcsherry/dynamic-datalog/commit/17a8b1f83451014ff4505d0685ed338808b54147. Let me know what you think.

frankmcsherry commented 5 years ago

More detail:

CRDT: Martin, Michael, and Mistral went over the outputs with me in person in Cambridge, and we validated that it was producing the right results on smaller problems. I don't think we have a reference output for the provided problem. DD numbers reported, and independent reproduction (or not) would teach us something.
DOOP: Soufflé and DD produce the same cardinality outputs.
GALEN: Soufflé and DD produce the same cardinality outputs, and the same cardinality for p as a reference solution provided by the John Liagouris.