search

smarr / are-we-fast-yet

Are We Fast Yet? Comparing Language Implementations with Objects, Closures, and Arrays
Other
336 stars 36 forks source link

Common Lisp implementation questions #75

Open q3cpma opened 2 years ago

q3cpma commented 2 years ago

Hello, I'm starting a Common Lisp version at https://github.com/q3cpma/are-we-fast-yet/tree/common-lisp and I have some questions about it.

Any comment on the code (https://github.com/q3cpma/are-we-fast-yet/tree/common-lisp/benchmarks/Common-Lisp) or on which implementations may be interesting? Couldn't make clisp work, but I intend to try Clasp.

smarr commented 2 years ago

Hi @q3cpma very interesting :)

Am I supposed to set the maximum optimization flags?

For execution and results, we usually run things in the "production" settings, which I naively would assume are usually the highest optimizations. So, my guess would be yes, except if there's a reason not to do so.

How about the code itself? Since CL has gradual typing, adding some types here and there can have some significant impact.

This is very good question, and there's no strict answer.

We have the general guideline: ""Use available language constructs to indicate immutability, or visibility of methods, fields, and other constructs. As long as it does not change the behavior of a benchmark, it is desirable to benefit from potential optimizations.""

So, from that, I'd say it's fair game to use types.

However, one may want to consider a second variant, without types. I am saying that, because most of the time, we use these benchmarks when working on the performance of language implementations, and here it's really a question of what we are interested in in optimizing. If you think that your Common Lisp implementation should figure it out from the code structure itself, you may want to have a version that tests it.

Assuming that it's easy to remove types, I'd probably start with the typed version.

I am using CLOS for now (except for som:random) but I may try to use structs at some point, for optimisation purposes and since the dynamic features of CLOS don't seem to apply. Does it make sense to you to have another version or something?

Yeah, it might make sense. As a language implementer as mentioned earlier, I'd consider having different versions when comparing/optimizing different things.

Just as an example: the benchmarks were originally designed to compare compiler effectiveness, but sometimes you want to look at different details. So, originally, the one main constraint was that all languages do exactly the same thing. But over time, I also wanted to see how the dynamic languages fair when using the builtin collections, which unfortunately does not guarantee that they do exactly the same things. But it may still be interesting, so there's a branch for that here where we do not use the SOM containers/collections: https://github.com/smarr/are-we-fast-yet/tree/awfy-dynamic-languages

q3cpma commented 2 years ago

So, I now have 2 or 3 versions (class, struct and struct with type declarations) of each micro benchmark and I want to know what you'd want in a PR before going all the way; especially considering that the numbers show CLOS as massively slow (reasons: 1. Class slot access is slow 2. Generic functions are slow (cf https://github.com/marcoheisig/fast-generic-functions) 3. You can't type class slots for optimization purposes (cf https://www.sbcl.org/manual/#Declarations-as-Assertions) unless you fiddle with the slot writer), which would make the most optimized defstruct version more appropriate, maybe with a link to my repo for more detailed numbers.

Don't worry about the README, bench.sh and other unnecessary files, I plan on making a clean branch for the PR and keeping this branch for more experiments.

I also don't really know how I'm supposed to wire all the testing part.

smarr commented 2 years ago

My intuition would be to go with the programming style that is most "idiomatic" for application code.

For me, the intention of these benchmarks is not to showcase the fastest possible performance, we have the Computer Language Benchmark Game for that. Instead the goal for me is to identify optimization opportunities in widely used language implementations by comparing with other implementation techniques available.

Does this sound useful to you?

In terms of testing, I usually try to have a community-accepted linter to have some confidence in coding style, and beyond that, I only run the benchmarks for 1 iteration as "test" to see that they yield the expected result.