Developer utilities: Graven Image `benchmark*` macro.

[aartaka]

This mentions graven-image:benchmark* macro as a more portable alternative to trivial-benchmark. However, adding this line feels... questionable for a variety of reasons:

• This might be perceived as antagonistic to trivial-benchmark or squatting the space in the awesome-cl at someone else's cost. I don't want to convey such an image:
  • I've asked about my (time-gutting) approach in trivial-benchmark two weeks ago with no reply left as of PR creation moment.
  • There's a clear mention of what differentiates benchmark* from trivial-benchmark.
  • I'm also CC-ing @Shinmera for transparency purposes.
• Mentioning a single macro instead of the whole library seems quite atypical to awesome-cl. Should it be a library mention instead?
  • While I consider benchmark* macro production-ready, I don't perceive Graven Image as a library ready enough to be mentioned.
  • I'll be glad to accept an advice on what to do in such a situation.
  • I'm okay with either of
    • Not including this line here.
    • Rewriting it to mention Graven Image as a library in the Developer utilities section.
    • Or leaving it be as a macro mention.

Anyway, huge thanks for maintaining this list :heart:

[aartaka]

Your link to benchmark* actually links to trivial-benchmark.

Oh, right 😅 Fixed now!

[fstamour]

@aartaka Also, I think 2 weeks is not that long to wait for an answer. Especially that @Shinmera maintains a shit-ton of libraries. Maybe your message just got lost in their inbox.

EDIT: I don't mean that this shouldn't be added, I really just meant that "2 weeks is not that long" :P

[fstamour]

Mentioning a single macro instead of the whole library seems quite atypical to awesome-cl. Should it be a library mention instead?

That's a good question... I think it happens often that there are gems hidden in random projects... Maybe it's not a bad idea

[Shinmera]

I did read the issue ticket when it was originally made, but didn't really know what to do with it, so I didn't reply. I also don't think that CL is the ecosystem where we really have to care about different libraries duplicating functionality lmao.

[fstamour]

I also don't think that CL is the ecosystem where we really have to care about different libraries duplicating functionality lmao.

I totally agree, but I won't get too off-topic here

[aartaka]

@aartaka Also, I think 2 weeks is not that long to wait for an answer. Especially that @Shinmera maintains a shit-ton of libraries. Maybe your message just got lost in their inbox.

My sample is too small, but I ofter found her responding to most issues/PRs I opened in under a waek.

But yes, let's wait for at least some comnent, I guess.

[fstamour]

But yes, let's wait for at least some comnent, I guess.

Lol, Shinmera already answered less than 20m ago

[aartaka]

So, merging then?

[vindarel]

The added line says that trivial-benchmark is limited to SBCL, that's not really so is it?

Also I'd like to understand better what this macro adds, and the readme doesn't really say, could this be improved?

Someone tried this macro? More feedback is welcome; also don't hesitate to write a little something / inform the community about this macro.

BTW did you reference this macro on Cliki? That would be the first place to edit, before this list ;)

[aartaka]

The added line says that trivial-benchmark is limited to SBCL, that's not really so is it?

See https://github.com/Shinmera/trivial-benchmark/blob/1fbc8d15f09ed8aa426bc73956b8b7c9b2668802/samples.lisp#L78 and https://github.com/Shinmera/trivial-benchmark/blob/1fbc8d15f09ed8aa426bc73956b8b7c9b2668802/samples.lisp#L134:

• Most metrics are only implemented on SBCL,
• There's also allocated bytes metric for ECL.
• All the other implementations only measure real and (user-space) run time using standard-based stubs.

Even though technically all implementations are supported, trivial-benchmark is only useful and matching the README-advertized output on SBCL. benchmark* is more consistent. Here's the output per every non-SBCL implementation I can test:

CCL:

(gimage::benchmark* (20) ;; Repeat count.
  (loop for i below 1000 collect (make-list i) finally (return 1)))
;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000 COLLECT (MAKE-LIST I) FINALLY (RETURN 1))
;; -                   UNIT      MINIMUM        AVERAGE        MAXIMUM        TOTAL
;; REAL-TIME           seconds   0.011919       0.0220453      0.045781       0.440906
;; USER-RUN-TIME       seconds   0.011753       0.02176555     0.045594       0.435311
;; SYSTEM-RUN-TIME     seconds   0.0            0.0002107      0.004036       0.004214
;; GC-RUN-TIME         seconds   0.007174       0.01677975     0.036468       0.335595
;; ALLOCATED           bytes     8008016.0      8008016.0      8008016.0      160160320.0
;; 1

ECL:

(gimage::benchmark* (20) ;; Repeat count.
  (loop for i below 1000 collect (make-list i) finally (return 1)))

;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000 COLLECT (MAKE-LIST I) FINALLY (RETURN 1))
;; -                   UNIT      MINIMUM        AVERAGE        MAXIMUM        TOTAL
;; REAL-TIME           seconds   0.012          0.01255        0.02           0.251
;; USER-RUN-TIME       seconds   0.017          0.01795        0.029          0.359
;; ALLOCATED           bytes     8003152.0      8008181.5      8013728.0      160163630.0
;; 1

ABCL:

(gimage::benchmark* (20) ;; Repeat count.
  (loop for i below 1000 collect (make-list i) finally (return 1)))
;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000 COLLECT (MAKE-LIST I) FINALLY (RETURN 1))
;; -                   UNIT      MINIMUM        AVERAGE        MAXIMUM        TOTAL
;; REAL-TIME           seconds   0.004          0.0063         0.029          0.126
;; USER-RUN-TIME       seconds   0.004          0.0063         0.029          0.126
;; ALLOCATED           bytes     10476024.0     14143264.0     20952048.0     282865280.0
;; 1

CLISP:

(gimage::benchmark* (20) ;; Repeat count.
  (loop for i below 1000 collect (make-list i) finally (return 1)))
;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000 COLLECT (MAKE-LIST I) FINALLY (RETURN 1))
;; -                   UNIT      MINIMUM        AVERAGE        MAXIMUM        TOTAL
;; REAL-TIME           seconds   0.022303       0.04416385     0.084759       0.883277
;; USER-RUN-TIME       seconds   0.022303       0.0439359      0.084509       0.878718
;; GC-RUN-TIME         seconds   0.010194       0.03094095     0.060914       0.618819
;; ALLOCATED           bytes     8008016.0      8008016.0      8008016.0      160160320.0
;; 1

Allegro:

(gimage::benchmark* (20) ;; Repeat count.
  (loop for i below 1000 collect (make-list i) finally (return 1)))
;; Benchmark for 20 runs of
;; (LOOP FOR I BELOW 1000 COLLECT (MAKE-LIST I) FINALLY (RETURN 1))
;; -                   UNIT      MINIMUM        AVERAGE        MAXIMUM        TOTAL
;; REAL-TIME           seconds   0.00875        0.01228025     0.075848       0.245605
;; USER-RUN-TIME       seconds   0.008733       0.0120648      0.07159        0.241296
;; SYSTEM-RUN-TIME     seconds   0.0            2.1565e-4      0.004254       0.004313
;; GC-RUN-TIME         seconds   0.003996       0.00712505     0.063914       0.142501
;; ALLOCATED           bytes     521699.0       521699.0       521699.0       1.043398e+7
;; 1

Also I'd like to understand better what this macro adds, and the readme doesn't really say, could this be improved?

The main source of documentation is the macro docstring (https://github.com/aartaka/graven-image/blob/cc47baf21d639dadc7b4226a8f1d3498b206c094/source/time.lisp#L282, also intertwined with documentation for other macros/functions that provide more details). But I've updated the README in https://github.com/aartaka/graven-image/commit/cc47baf21d639dadc7b4226a8f1d3498b206c094 too.

also don't hesitate to write a little something / inform the community about this macro.

I'm postponing it until Graven Image is in the stable state.

BTW did you reference this macro on Cliki? That would be the first place to edit, before this list ;)

No, I don't know how to do that :cold_sweat:

[fstamour]

BTW did you reference this macro on Cliki? That would be the first place to edit, before this list ;)

No, I don't know how to do that 😰

Next issue: document how to use the cliki xD

[aartaka]

Okay, I figured it out. Will probably document it on CLiki on the weekend. Is that a hard requirement?

[vindarel]

Is that a hard requirement?

no no, not yet at least, but way better© IMO, as the goal of Cliki is to reference everything CL and one has space to write explanations.