New benchmark idea (low from benchmark harness authors)

[lpereira]

Here's an idea for a new benchmark:

• Require frameworks to implement JSON, Queries, Fortunes, and Plaintext.
• Make requests in an arbitrary order for each endpoint (e.g. /json, /queries?queries=10, /plaintext, /fortunes, /queries?queries=500, /json, ...).

This should mimic pretty well a normal workload and provide a more "balanced" result.

There's no work to be done by any of the frameworks. (Bonus point: this should reduce some of the "smoke and mirrors" some frameworks perform to game each individual benchmark.)

[NateBrady23]

Thinking about it now, there might be some work to be done by frameworks. Some frameworks separate tests such that plaintext and json might be in one configuration, while database tests might be in another. We'd want them to be in the same test. And then the framework would have to add the test key to those configurations so we'd know we can run the "all_tests" or whatever we'd call it if we did do this.

[lpereira]

Indeed. I ran this script:

#!/usr/bin/python

import os
import json

for root, _, files in os.walk('.'):
  for name in files:
    if name != 'benchmark_config.json':
      continue

    path = os.path.join(root, name)
    with open(path, "r") as f:
      j = json.load(f)
      for test_level0 in j.get('tests', []):
        for k, v in test_level0.items():
          if not 'json_url' in v: break
          if not 'plaintext_url' in v: break
          if not 'query_url' in v: break
          if not 'fortune_url' in v: break

          print(path)

And it found 144 frameworks that would be able to run such benchmark without any effort from their part.

So maybe call this "low-effort" instead of "no-effort"? :)