exercism / v3

The work-in-progress project for developing v3 tracks
https://v3.exercism.io
Other
170 stars 163 forks source link

[Common Lisp] Call for Discussion Regarding Common Lisp Tooling #497

Closed TheLostLambda closed 4 years ago

TheLostLambda commented 4 years ago

The aim of this thread is to provide a place where the tooling for the Common Lisp track can be discussed. The end goal is to agree upon a standard environment that can comfortably be recommended to students and used by the Common Lisp test runner.

Some preliminary tasks (please feel free to add to this):

That's just a place to start, if you think of something additional that needs discussion, it can be added to this list!

verdammelt commented 4 years ago

RE: testing libraries - not just lisp-unit but lisp-unit2!

RE: students using external packages - perhaps this is my 'old exercism' mind talking - but we are trying to teach Common Lisp, its idioms and concepts - not what the library ecosystem is like IMNSHO. Perhaps I need to change this mindset for v3?

TheLostLambda commented 4 years ago

A good call with lisp-unit2! I've added it to the list. There seem to be an endless number of options...

Regarding external packages, I'd tend to agree actually – at least for the beginning. The language standard is massive as is. With that being said, should something like UIOP (built into anything with ASDF, as I understand it) be fair game?

verdammelt commented 4 years ago

Just having a discussion with a student (not you right?) about using UIOP. I say no if we limit students to using only things in the language. The fact that we may have the student install ASDF doesn't, to me, relax the constraint of only using the features of the language to solve the problems.

TheLostLambda commented 4 years ago

Okay, I find that reasonable. I've used it once or twice just for the split-string function, but aside from that, it's mostly OS level compatibility stuff which probably won't be necessary anyways. I would be alright with only allowing the standard.

We can wait and see what others have to say regarding the topic (as I've encountered some Exercism tracks that encourage / require external libraries – namely the "Reverse Graphemes" challenge on the Rust track).

If we aren't allowing Quicklisp, then we will probably need to manually bundle a testing library in the test runner, but that shouldn't be too difficult.

Speaking of which, I've not played much with CL testing libraries, but I know lisp-unit is being used currently and FiveAM and rove are recommended here. Does anyone have a particularly strong opinion here? It's probably something that needs to be sorted out before any real exercises are written.

timotheosh commented 4 years ago

I think Portacle is the easiest thing to have a student install on their system. And it is available on Windows, MacOS, and Linux.

Atom editor with SLIME is another option. It is easier to set up than Emacs+SLIME, but not as easy as Portacle.

roswell seems like a no-brainer for the test runner. The only real other viable option, in my mind, is cl-launch. I've used both, and have even created docker images using cl-launch before roswell became a thing.

I think Quicklisp should be allowed. It is such a fundamental part of the Common Lisp infrastructure, today, I do not think we are promoting the use of Common Lisp very well, by eliminating it. Portacle comes with Quicklisp, and so does roswell. I think we just need to decide "when" it is permissable, not "if", and write the test cases accordingly.

TheLostLambda commented 4 years ago

I think that might be a good compromise, to build the infrastructure with support for Quicklisp and to allow it selectively in different exercises. I still agree with the overall sentiment that I think @verdammelt was conveying, that most of the exercises (particularly the foundational concept ones) should stick as close to the standard as possible, but Quicklisp could allow for later Regex or multi-threaded challenges.

I think that the analyzer would be more than capable of automatically identifying solutions that use "disallowed" libraries for a given exercise.

Additionally, I wasn't able to get Roswell to even install a CL implementation until a little while ago, so I'm not super familiar with it. What benefit does it provide over just loading a test file into SBCL? I only ask because I don't have a ton of experience with it and, generally, the fewer moving parts, the better.

I think Portacle is a pretty sound recommendation. I've not tested it much on Windows, but it seems like a really quick way to get started (minus the Emacs learning curve for total newcomers). I'll have to look into this a bit more myself, but it looks good at the moment. I wasn't aware that there was a SLIME for Atom, so I may give that a look as well.

@timotheosh Do you have much experience with CL unit testing libraries? Do you have one that you're a fan of?

timotheosh commented 4 years ago

@TheLostLambda roswell also simplifies installing any common CL implementation onto Docker, complete with Quicklisp, and any unit testing framework we want.

I've never had an issue installing a Common Lisp implementation using roswell. I have some automation setup that downloads, installs and runs roswell automatically on my laptop. Automation is sorta my profession/expertise. It's my day job.

I do not have much experience with CL testing libraries and like the idea of using what we are using, already.

TheLostLambda commented 4 years ago

Okay, I think that's good reasoning for using Roswell. I can definitely see it simplifying CL implementation installs, particularly if we want to test on several implementations. I'd be fine with using roswell for the test runner.

Additionally, it had been a while since I'd checked out Portacle, but I've just taken another look an that seems like an excellent recommendation for students. Less than 100MB to download everything that you'll need. The emacs configuration it comes with is pretty slick too (and comes with Sly <3).

We could continue with lisp-unit, but I've been playing a bit with rove and am quite liking it. I think the output is a bit nicer, the testing writing much cleaner, and it's actively maintained.

Here is the old lisp-unit:

(ql:quickload "lisp-unit")
#-xlisp-test (load "roman")

(defpackage #:roman-test
  (:use #:cl #:lisp-unit))

(in-package #:roman-test)

(define-test test-1
  (assert-equal "I" (roman:romanize 1)))

(define-test test-2
  (assert-equal "II" (roman:romanize 2)))

(define-test test-3
  (assert-equal "III" (roman:romanize 3)))

(define-test test-4
  (assert-equal "IV" (roman:romanize 4)))

(define-test test-5
  (assert-equal "V" (roman:romanize 5)))

#-xlisp-test
(let ((*print-errors* t)
      (*print-failures* t))
  (run-tests :all :roman-test))
TEST-1: 1 assertions passed, 0 failed.

TEST-2: 1 assertions passed, 0 failed.

TEST-3: 1 assertions passed, 0 failed.

TEST-4: 1 assertions passed, 0 failed.

TEST-5: 1 assertions passed, 0 failed.

Unit Test Summary
 | 5 assertions total
 | 5 passed
 | 0 failed
 | 0 execution errors
 | 0 missing tests

The new rove:

(ql:quickload "rove")
(load "roman")

(defpackage #:roman-test
  (:use #:cl #:rove))

(in-package #:roman-test)

(deftest test-romanize
  (ok (string= "I" (roman:romanize 1)))
  (ok (string= "II" (roman:romanize 2)))
  (ok (string= "III" (roman:romanize 3)))
  (ok (string= "IV" (roman:romanize 4)))
  (ok (string= "V" (roman:romanize 5))))

(run-suite *package*)
test-romanize
  ✓ Expect (STRING= "I" (ROMAN:ROMANIZE 1)) to be true.
  ✓ Expect (STRING= "II" (ROMAN:ROMANIZE 2)) to be true.
  ✓ Expect (STRING= "III" (ROMAN:ROMANIZE 3)) to be true.
  ✓ Expect (STRING= "IV" (ROMAN:ROMANIZE 4)) to be true.
  ✓ Expect (STRING= "V" (ROMAN:ROMANIZE 5)) to be true.

✓ 5 tests completed

Personally, I'm a fan, and there is a lot more here: https://github.com/fukamachi/rove

Let me know what you all think!

TheLostLambda commented 4 years ago

Additionally, I'm not sure what I'm doing wrong, but the breakage problem I've always had with Roswell persists. ros install clisp crashes on both my Arch machines as well as a Void Linux container, and Ubuntu container.

sbcl-bin works fine, but for a tool made to install other lisp distributions, the inability to install clisp, clasp, ecl, and mkcl seems pretty breaking... Not even sbcl is working.

It's a bit demoralizing when Ubuntu and Arch have one-line installs for SBCL and CLISP (without the extra Roswell cruft).

I'm likely just doing something silly? But I'm at a loss for the moment.

timotheosh commented 4 years ago

It installs Clisp from source. One major issue I have about using Clisp is that it is barely maintained any more. GNU's CL hack, gcl, gets more love judging from the Savanah repos, than Clisp does.

FWIW, on Docker, I would not use Arch or Void Linux. They are just fine for hobbyists, but very poor choices for general consumption. Stick with Debian/Ubuntu/CentOS (stable releases only, do not use betas). Exercism is promoting the use of Alpine. Mostly for the smaller footprint, I am sure, but Alpine is just barely better tested than Arch or Void, and many arch detection scripts mistake 64bit Alpine for a 32bit system.

Also, I don't think we will be testing with Clasp. It takes several hours to compile, even on a fast modern computer, and is serious overkill for our purposes. Clasp has a very specialized application.

This works, but installs more than is needed, I think. If you insist on Clisp, I am sure I could get it on there by just running apt build-dep clisp before hand (installs the needed dependencies for clisp).

FROM ubuntu:18.04
RUN apt-get update && apt-get -y install build-essential gcc git make autoconf automake libcurl4-openssl-dev libz-dev
RUN cd /usr/local/src/ && \
    git clone https://github.com/roswell/roswell.git && \
    cd roswell && \
    git checkout v20.01.14.104 && \
    ./bootstrap && \
    ./configure && \
    make && \
    make install && \
    cd .. && \
    rm -rf roswell
RUN useradd -s /bin/bash -m roswell
USER roswell
WORKDIR /home/roswell
RUN ros init
RUN ros install sbcl/2.0.1
RUN ros install lisp-unit lisp-unit2 rove cl-ppcre

Alpine Linux is a pain in the arse to reason about with many dependencies, and at the end of the day, the more dependencies needed, the more bloated an Alpine Docker image gets. I think it would be easy to justify using Debian or Ubuntu for this.

TheLostLambda commented 4 years ago

RE Clisp: It's not too big of a bother, I usually just use SBCL with SLY / SLIME. Clisp just has a much nicer REPL (when SLIME isn't available). For the most part, I was just playing around with Roswell.

I definitely understand the preference towards stability and ease of use vs something like Alpine, but I think that Alpine might have a case to be made for it as well. I'm far from a professional when it comes to these things, but I ended up with this:

FROM alpine
# Update packages and install SBCL
RUN echo "@testing http://nl.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories
RUN apk update && apk upgrade && apk add sbcl@testing
# Download and install quicklisp
RUN wget https://beta.quicklisp.org/quicklisp.lisp
RUN sbcl --load quicklisp.lisp --eval '(quicklisp-quickstart:install)' --quit
RUN echo '(load "/root/quicklisp/setup.lisp")' > /root/.sbclrc
# Optionally install packages, to see how big it gets. This won't need to stay here
RUN sbcl --eval "(mapcar #'ql:quickload '(:lisp-unit :rove :cl-ppcre))" --quit
 # Because lisp-unit and lisp-unit2 don't like to be in the same room. Again, only for size testing.
RUN sbcl --eval "(ql:quickload :lisp-unit2)" --quit

It might not be as pretty or convenient, but the build time and size reduction is significant:

# Roswell and Ubuntu
$ time docker build . --no-cache -t ubuntu-roswell
... snip snip ...
real    3m21.719s
user    0m0.097s
sys 0m0.073s

# SBCL and Alpine
$ time docker build . --no-cache -t alpine-sbcl
... snip snip ...
real    0m26.044s
user    0m0.038s
sys 0m0.018s

# Size Comparison
$ docker images
ubuntu-roswell               latest              5f307ae61722        7 minutes ago       769MB
alpine-sbcl                  latest              7c06c705d95b        14 minutes ago      76.8MB

With a smaller build file, 1/7th of the build time, and 1/10th of the disk space, I personally would lean towards using Alpine, but I also recognize that I may lack the experience to foresee issues with this kind of solution. I just figured that I would play devil's advocate for Alpine.

timotheosh commented 4 years ago

The cut in build time and the size is definitely due to using pre-built binaries. My image installs an sbcl binary and then compiles a version of sbcl from source. We could easily cut out building sbcl from source, and then also the line that installs the needed dependencies for building sbcl. I threw it in there to show that Roswell works.

Here's why I don't trust Alpine Linux for anything other than isolated VM's (like Go, .Net, or Java): https://pythonspeed.com/articles/alpine-docker-python

The linked article addresses Python specifically, but the problem will potentially affect anything that links to C libraries.

If there is a stable sbcl package for Alpine, and forbid any Quicklisp at all, then Alpine might be fine. But we also would not be using Roswell. Roswell interprets x86_64 Alpine as a 32bit system, and tries to install the wrong version of the sbcl binary. Alpine does not allow coexistence of 32 bit and 64 bit packages without some finageling.

TheLostLambda commented 4 years ago

My previous messing about with Alpine has been mostly with code of my own, so I hadn't actually tried to compile a lot of external code. I gave it a go in Alpine and I couldn't get Woo to play nice. I agree then that we'll have to use something other than Alpine. I think Ubuntu is a plenty fine choice.

If we are using something that isn't rolling release, then I can see the boon in Roswell. I cut this down to what I think are the bare essentials:

FROM ubuntu
RUN apt-get update && apt-get -y install build-essential git automake libcurl4-openssl-dev
RUN cd /usr/local/src/ && \
    git clone https://github.com/roswell/roswell.git && \
    cd roswell && \
    ./bootstrap && \
    ./configure && \
    make && \
    make install && \
    cd .. && \
    rm -rf roswell
RUN useradd -s /bin/bash -m roswell
USER roswell
WORKDIR /home/roswell
RUN ros init
RUN ros install lisp-unit lisp-unit2 rove cl-ppcre

It's still lots slower and bigger than Alpine, even without compiling anything, coming in at 1m34s and 547MB, but it might be about the best we can do.

To get Woo to compile, I need to bring back gcc and add libev-dev, but that only adds a couple of MB, so it isn't a big deal.

Usually I'd try to use a muti-stage build to cut down on size by discarding dev dependencies, but I think that proper Quicklisp support means we will need to keep them around.

I wish we could shrink it down some more, but I agree that Ubuntu and Roswell is probably the way forward.

verdammelt commented 4 years ago

Haven't read through all the details of of the above but I wanted to chime in on a few topics:

What are people's thoughts on my proposed setup for exercises? (i.e. 2 ASDF systems (one for testing) which the student would use (asdf:test-system "exersise-slug") to run tests?) I think one problem of it is how to get ASDF to know where the systems are? Do we tell students where to put their exercises? Can we do that? Should we do that? Or can we "easily" give the students the ability to put their exercises where they want and have ASDF find them?

TheLostLambda commented 4 years ago

I'm glad I'm not the only one who has issues with Roswell. I think at the moment it's being considered as a glorified SBCL + Quicklisp installer? Here is a version without Roswell and with a working Woo:

FROM ubuntu
RUN apt-get update && apt-get -y install build-essential gcc libev-dev sbcl wget
RUN wget https://beta.quicklisp.org/quicklisp.lisp
RUN sbcl --load quicklisp.lisp --eval '(quicklisp-quickstart:install)' --quit
RUN echo '(load "/root/quicklisp/setup.lisp")' > /root/.sbclrc
RUN sbcl --eval "(mapcar #'ql:quickload '(:rove :woo))" --quit

I'll admit I'm a fan of the simpler setup with a faster build (51s) and smaller size (365MB). The only real downside I can see is older versions of SBCL sometimes (it's on 1.4.5, but it's been better tested I suppose). I do quite like this solution.

I think it's pretty safe to settle on SBCL for the test runner (and ultimately the recommendation).

I also think Portacle is a great place to send people, with some alternatives of course.

I've not looked too closely at your proposed setup, I'll have to hunt that down and give it a read. With that being said, what advantages over the (load "test-file.lisp") would a second ASDF system bring?

verdammelt commented 4 years ago

RE: ASDF setup - https://github.com/exercism/v3/blob/master/languages/common-lisp/docs/implementing-a-concept-exercise.md has my proposed exercise format.

When I wrote the original track i didn't really know much about ASDF and it seems now to be "the way" to do things. The two system setup is the ASDF suggestion for having production code and test code. Using just load I have sometimes run into issues with load vs. compile time which I still often get confused by :|

TheLostLambda commented 4 years ago

I'm still very much new to ASDF myself, but that proposal looks good and clean. Plenty of other tracks download packages that only work with a single build-tool and I think it's totally reasonable to expect the ASDF file to be in the same directory as the testing and source files. Rust puts source files in src/ and test files in tests/ with Cargo linking stuff together.

It also looks like this is a good example of defining a test-op and splitting things into two systems: https://common-lisp.net/project/asdf/asdf/Predefined-operations-of-ASDF.html I'm assuming this is the sort of thing you had in mind?

If so, I think that's a pretty clean solution overall!

verdammelt commented 4 years ago

On Tue, Feb 04 2020, Brooks Rady wrote:

It also looks like this is a good example of defining a test-op and splitting things into two systems: https://common-lisp.net/project/asdf/asdf/Predefined-operations-of-ASDF.html I'm assuming this is the sort of thing you had in mind?

That is exactly the sort of thing I was thinking of.

The one problem we will need to overcome (which might not be that big as we get better with ASDF) is how to get it to find the systems. It seems to be built about the idea of putting all systems into a central place, or configuring ASDF about other directories to look for systems in. I also believe you can use load-asd to simply load a specific ASDF system file and then it will know that system.

timotheosh commented 4 years ago

We could always read the students exercism user.json file in the expected location to see where asdf needs to load the system.

TheLostLambda commented 4 years ago

Okay, that's something I'd have to look into as well. In the meantime, here is a nice example I found of a complex ASDF file with a simple, separate testing system: https://github.com/sionescu/bordeaux-threads/blob/master/bordeaux-threads.asd. It looks like :module is used to select folders relative to the directory with ASDF?.

One question I have about your proposal, @verdammelt, is what the package.lisp file is doing. I haven't been able to find an equivalent elsewhere.

timotheosh commented 4 years ago

Okay, that's something I'd have to look into as well. In the meantime, here is a nice example I found of a complex ASDF file with a simple, separate testing system: https://github.com/sionescu/bordeaux-threads/blob/master/bordeaux-threads.asd. It looks like :module is used to select folders relative to the directory with ASDF?.

Yes, forgot about that. You can use caveman2 to create packages with similar layouts.

One question I have about your proposal, @verdammelt, is what the package.lisp file is doing. I haven't been able to find an equivalent elsewhere.

It defines the package. It doesn't "have" to be in a separate file from the main lisp program, but does keep things clean in larger projects.

TheLostLambda commented 4 years ago

So it's stuff like this:

(ql:quickload "rove")
(load "roman")

(defpackage #:roman-test
  (:use #:cl #:rove))

With this in the source files?

(in-package #:roman-test)

If so, then I think that's a nice, clean separation :)

Or is it just the defpackage?

timotheosh commented 4 years ago

Yes you have it :)

Also, this creates CL skeleton projects as great examples for ASDF, complete with unit tests. https://github.com/fukamachi/cl-project

TheLostLambda commented 4 years ago

cl-project does look nice. I wouldn't mind having the src/ and tests/ system or just the flat structure that has been proposed. I think they are both fine.

EDIT: With that being said, it looks like cl-project sets up pretty much everything, ASDF files included. That could be an easy, standard way forward. It assumes some things, like using rove but that might be acceptable.

timotheosh commented 4 years ago

Or is it just the defpackage?

Usually, you put all the package dependencies in there. Quicklisp works, but many omit it, because Quicklisp will do the right thing if your package, itself, is loaded using Quicklisp.

Technically, you can put anything in there, since the asdf file loads it.

timotheosh commented 4 years ago

EDIT: With that being said, it looks like cl-project sets up pretty much everything, ASDF files included. That could be an easy, standard way forward. It assumes some things, like using rove but that might be acceptable.

IIRC, you can pass parameters to use any test suite you want. Even if that is not the case, it would be easy to insert our choice in place of Rove.

Roswell allows you to creat your own custom templates... just sayin' Even if we do not use Roswell in the test-runner or anywhere else, it is still pretty handy for the mentor/maintainer. :)

TheLostLambda commented 4 years ago

I might have to look into some of those Roswell templates myself. While I'm still a tad hesitant about using it in production, I quite like the idea behind Roswell and could always use with picking up some new tools.

For the moment, however, I think that it would be good to check off a couple of items from the initial list.

With that being said, I'll list out some things that I think we've agreed on so far and if you still agree, you can give this message a :+1: or give it a :-1: / reply if you'd like something different.

The propositions being:

  1. SBCL will be the implementation used in the Common Lisp test runner
  2. The primary editor / implementation recommendation to students will be Portacle
  3. Consequentially, the implementation recommendation for students on all platforms will be SBCL?
verdammelt commented 4 years ago

On Tue, Feb 04 2020, Tim Hawes wrote:

We could always read the students exercism user.json file in the expected location to see where asdf needs to load the system.

No, I am not so worried about 'us' (maintainers/mentors/automated bots) loading the files but the students themselves. What instructions will they have for loading and running the tests in the stub code we provide?

verdammelt commented 4 years ago

One question I have about your proposal, @verdammelt, is what the package.lisp file is doing. I haven't been able to find an equivalent elsewhere.

It defines the package. It doesn't "have" to be in a separate file from the main lisp program, but does keep things clean in larger projects.

Yeah, not necessary but it seems, given my brief 'survey' of the community (lurking in newsgroups, Reddit etc) it seems one package.lisp at the top of the project is typical these days.

TheLostLambda commented 4 years ago

Found a fantastic document here: https://github.com/fare/asdf/blob/master/doc/best_practices.md#testing_system

It looks like, if we define the test-op, then a (asdf:test-system :foobar) from a CL REPL / SLIME in the code directory should suffice, or a sbcl --eval "(asdf:test-system :foobar)" --quit from the terminal.

I think it's fine to expect that code, tests, and system-definition files are in the same directory (or at least always have the same relative paths). It's possible I may be misunderstanding though.

verdammelt commented 4 years ago

@TheLostLambda Great doc. I'll test it out before saying it won't be that easy (I'm not known for my optimism) :)

verdammelt commented 4 years ago

This looks good to me. I think I'd lean on fiveam rather than rove as I just saw that rove's documentation has a warning: "This software is still BETA quality. The APIs will be likely to change." Not sure how true that is, or how stable fiveam is but at least it doesn't have that warning :).

TheLostLambda commented 4 years ago

Yeah, upon looking into things a bit more, I think that FiveAM looks like the standard for serious CL projects. I think it would be safe to use that for now!

TheLostLambda commented 4 years ago

Small update here, as the test-runners won't have access to the internet, I've come up with this. It's only ~70MB and is pretty bare-bones.

I've preselected some Quicklisp libraries that we may want available for some practice exercises or just for students to play with in a sandbox. Let me know what you think of those choices and if you think I should include any more!

FROM daewok/sbcl:alpine
RUN apk update && apk upgrade
ADD https://beta.quicklisp.org/quicklisp.lisp /
RUN sbcl --load quicklisp.lisp --eval '(quicklisp-quickstart:install)' --quit
RUN echo '(load "/root/quicklisp/setup.lisp")' > /root/.sbclrc
RUN sbcl --eval "(mapcar #'ql:quickload '(:alexandria :serapeum :bordeaux-threads :cl-ppcre :iterate :local-time :lparallel :named-readtables :series :cl-json :fiveam :trivial-benchmark :closer-mop :trivia))" --quit
verdammelt commented 4 years ago

This looks nice. That dockerfile is or will be over the test runners repo?

I'm still in the camp of not having the students use external libraries myself - fiveam is the only one there I know will be required.

TheLostLambda commented 4 years ago

It's not up just yet, but I'll look into getting it there :)

And I think I agree for the actual exercise solutions. I suppose I just haven't decided if I want to enforce that yet. It sounds like these test runners might only be used to test code (no sandbox coding or anything) so we can probably get away with just FiveAM (and maybe CL-JSON because the test runner needs to handle JSON).

I'll commit it to the testing repo like this, then cut out everything but FiveAM and cl-json (just so it's in git somewhere).