Shootout benchmarks

[krono]

These are the shootout benches

• [x] ackermann
• [x] ary
• [x] binarytrees
• chameneos samth/pycket#116 (threads)
• cheapconcurrency samth/pycket#116 (threads)
• echo needs threads (threads)
• [ ] except
  • samth/pycket#122 (call-with-current-continuation-prompt)
• [x] fannkuch
• [x] fannkuch-redux
• [x] fasta samth/pycket#112
• [x] fibo
• [x] hash samth/pycket#90
• [x] hash2 samth/pycket#90
• [x] heapsort samth/pycket#120
• [ ] k-nucleotide
  • samth/pycket#90
  • samth/pycket#134
• [x] lists
• [x] mandelbrot
• [x] mandelbrot-generic
• [x] matrix
• [x] meteor
• [x] moments
• [x] nbody
• [x] nbody-generic
• [x] nbody-vec
• [x] nbody-vec-generic
• [x] nestedloop
• [x] nsieve
• [x] nsievebits
• [x] partialsums
• [x] pidigits
• [x] pidigits1
• [x] random samth/pycket#115
• [x] recursive
• [x] /regexmatch/
  • samth/pycket#121(regex)
• [ ] /regexpdna/
  • samth/pycket#117 samth/pycket#121 (regex)
  • samth/pycket#135
• [ ] /reversecomplement/
  • samth/pycket#121
• [x] reversefile
• [x] sieve
• [x] spellcheck samth/pycket#90
• [x] spectralnorm
• [x] spectralnorm-generic
• [x] strcat samth/pycket#119
• [x] sumcol
• thread-ring no output
• [x] wc samth/pycket#118
• [x] wordfreq

[cfbolz]

Should we focus on the current shootout benchmark set? In the above benchmarks a lot of older, currently phased out benchmarks are listed. the current set is:

binary-trees chameneos-redux fannkuch-redux fasta k-nucleotide mandelbrot meteor-contest n-body pidigits regex-dna reverse-complement spectral-norm thread-ring

[samth]

An unfortunately high percentage of these involve regular expressions. If they just need something simple that's not performance critical, we could maybe support that, but not the others.

The set I'd most like to focus on is those in my pldi 11 paper, which Racket has previously been compared on.

[krono]

The set from this paper is (shootout part):

• [x] binarytrees
• [x] fannkuch
• [x] fasta
• [ ] k-nucleotide
• [x] mandelbrot
• [x] nbody
• [x] pidigits
• [ ] regex- dna
• [ ] rev.comp
• [x] spectralnorm

[cfbolz]

Mandelbrot* seem to not really work for me, I get primitive regexp-match? is not yet implemented

[samth]

I think @krono changed that to error instead of always returning true.

[krono]

Yes, thanks for the catch. I’ll try to patch the benches where regexes are not actually necessare

[samth]

I think the benchmark "uses" regexps by requiring a library that loads them. We just need to put that hack back in. ​

[krono]

well, loading does not trigger the not-implemented-error, calling does. Probably someone in racket/port does this, so, yes for regex-match? the hack may be justified

[samth]

@krono can you update the status here?

[krono]

will do

[krono]

status is up-to-date