Request: state of web development

[vindarel]

For this I'd personally use https://stackoverflow.com/a/42838145/1506338 (and I'm very eager in writing it).

There would be examples, what's lacking, how to quickstart everything, references to libraries found in the Awesome list.

[PuercoPop]

That SO post have several omissions. For example just form generators there is:

• formlets
• cl-forms

Mariano also has clavier for validation. The ORMs list fails to mention CL-SQL. Fukamachi has written [mito] recently

I also don't understand why one would mention wu-wei but not hunchentoot.

Maybe it would be a good idea to tackle this by sections first?

[vindarel]

Thanks for the links ! They were hard to find. Just found also 1forms

Django-like web (html) forms for Common Lisp, It can be used with various Common Lisp web servers and web frameworks such as caveman2. 1forms uses CLOS and is extensible by design.

but not touched for 2 years.

Sure this post has omissions but

• it is not meant to be exhaustive, this task is left to the Awesome CL list and the State of CL ecosystem 2015 (both links given in the intro)
• in my biased opinion (I'm the author there) it's the only place I know one can have an overview. Now we could do this here.

(it does mention Hunchentoot but puts emphasize on Clack)

Maybe it would be a good idea to tackle this by sections first?

Ok. And shall we make up a list/our TOC first ?

[can3p]

I think Hunchentoot is in much better place in terms of documentation. If clack is put in the cookbook it needs a really extensive tutorial there

[PuercoPop]

The subsections/divisions could be

• Frameworks and Web Application Servers (Including websockets)
• Serialization/data storage (SQL and NoSQL?)
• Validation and form generation
• Template libraries
• Misc (Job queues, oauth, jwt, etc)

Do y'all think its ok? Have I missing an area?

[vindarel]

+1 for the division. You didn't note javascript integration, but ok to do this last. Also we can suppose we'll have a real databases page, so we would speak here only about integrating a DB with a web framework.

Those are some points I'd like addressed (what I've been researching on this topic and what blocked me while trying Caveman and Lucerne).

First of all, what application shall we write as an example throughout the page ? (as in the web scraping page)

• use and extend the Lucerne twitter clone example ?
• use an existing application (Quickdocs, Turtl, Potato,…) ?
• build the Real World medium clone or Todo Backend missing examples ?

If we present the application servers, what level of documentation do we put ? I'd say just enough to get things done, and redirect to the project's documentation.

• Web Frameworks

  • application models, database tables, web forms generation ("do we have to write 3 times our models ?") (this question the twitter clone does not answer) Looks like Caveman has models with the Datafly lib
  • how to access request parameters ? (ok)
  • and url parameters ? (not obvious nor in Caveman (very unclear) nor in Lucern, had to ask and have rely on Clack). (edit) clear in Snooze.
  • session & cookies support
  • templates engines
  • all the different kinds…
  • spinneret VS cl-who,…

• Data storage (according we have a DB page), and here say how frameworks integrate the DBs. We would test everything, show bits of code and warn about limitations.

  • migrations: does Caveman have DB migrations ? The Crane ORM advertises to have, but they don't work with SQLite (issue), and they have other issues.
  • SQL, NoSQL… and bknr.datastore ? I'd like to try that :D

Important stuff:

• deployment

  • creating an executable (an image) for a web app (doable but with a trick, see SO)
  • regular technics, examples (launching a swank server at the app startup ?)
  • Ceramic for a portable multiplatform app (Electron)
  • Caveman advertises hot reloading (with a Perl module). How well does it work ? Examples. Built in Hunchentoot ?

• testing

  • how is it ? Lucerne advertises clack-errors (edit: snooze can either print stacktraces in the browser, let the debugger handle them or display a customed 404 page).
  • How can we test code with a database, does the framework create a test database ? (as in django)

Then, again later:

• JavaScript integration
  • basics of Parenscript
  • interactive with trident-mode (Skewer) ?
  • ajax POCs (even if not production ready)
  • Panic (React) POC ?

shall we only point to the interesting projects for this JS part, with no code sample ? (like the Awesome CL list) Because all of this is unstable and changes frequently.

At the same time:

• see also…
  • we would advertise real world examples: Quickdocs, Turtl, Potato,… not really the goal of a cookbook, but not either the place of the Awesome CL. Also, no need to be exhaustive.

[vindarel]

linking to the ongoing Clack tutorial: https://jasom.github.io/clack-tutorial/pages/getting-started-with-clack/ (from reddit)

[Shinmera]

I'd appreciate it if you could also investigate, or in the very least mention, Radiance.

[pnathan]

The Lisp web framework world has forked and reworked itself a lot in the past half decade. I'm not wholly clear on the reasons, myself. I think we should exclude any web framework that isn't relatively stable at a 1.0+ level, for the reasons of sanity... if someone wants to do the indepth review/analysis of different web frameworks, they won't go to a Cookbook.

[Shinmera]

I'd also make sure that the framework has a couple of example applications already out there and has a sizeable amount of documentation to offer as well. I've seen all too many people frustrated in #lisp due to the lack of proper documentation or reference material.

[vindarel]

Greate example with ningle, closure-template, hot deploy with connection to a swank server, supervisor, jquery and pjax, watchdog autoreload: https://github.com/tarballs-are-good/quickutil/blob/master/quickutil-server/

[azzamsa]

I hope there will someone write tutorial about web development in CL,

I felt what Shinmera said

"I've seen all too many people frustrated in #lisp due to the lack of proper documentation or reference material."

As a beginner, I spent days and hours just to figure it out simple things, compared to other languages that even beginner can have so many tutorial and screencast.

It is too hard just to learn basic web development because the lack or resources. the things I see when I stuck

• Fullstacklisp
• Writing a Common Lisp Web App in caveman2
• kindista.org
• Quickdocs-server.

I hope there will be nice tutorial for web development in CL. as I seen many question asking about tutorials.

• https://github.com/fukamachi/caveman/issues/21
• https://groups.google.com/forum/#!topic/comp.lang.lisp/ShMA0kNOp24

many people say learning emacs is difficult, but I think it's harder to learn web development with CL, for non-web development there are a lot of books about CL (let over lambda, on lisp, PCL, Successful lisp, Gentle Introduction to Symbolic Computation). But we do not have any completed books of web development in CL.

I hope there will be the book as soon as possible.

I hope I can write tutorials about web dev in CL when I finished learning it.

Thank you so much for everyone contributing to this Cookbook, thank you.

[pnathan]

I have asked the maintainer of Quickutils if he minds us using the system. Although it is MIT license or so, I'd prefer to be polite (and he's a nice guy, I've met him).

[pnathan]

@azzamsa http://www.adamtornhill.com/articles/lispweb.htm is a book about Lisp for the web. I don't know how current it is though.

@Shinmera - how stable IS the Lisp web world these days? have we all come to a stable web world or are we still in the forking/rewriting stage? when I've farted with Lisp web stuff in the last 2 years, I've just used raw hunchentoot because, well, it wasn't fancy enough to matter.

[pnathan]

( @vindarel I think this is the area the Cookbook should focus on over the next two months or so. I have vacations coming up, so I should have time to help out here, but I don't have deep knowledge of web frameworks and what to expect - I mostly don't use them in general)

[azzamsa]

@pnathan Thank you so much for the response, I've known you long before I learned lisp, I visited your blog several times. I know you from your answers on Stackoverflow. Thank you very much.

Yes, lispweb by Adam is the first book I know of Lisp web Development, but I think it does not suit me because I take us immediately to CLOS.

I also have contacted the author of Fullstacklisp and he said that he will continue working on it soon

If I finished learning web dev in CL, I will write my story about it, challenging and giving a little pain :)

[pnathan]

The author of Quickutils says:

Great idea! We are happy about that.

So let's see what we can do to port it into a Cookbook tutorial

[Shinmera]

@pnathan Radiance has been at 1.0 for a while now. It comes with full documentation, a bunch of pre-written applications to look at, and an in-depth introductory tutorial. It is pretty stable and I haven't done anything to it aside from bugfixes since release, nor do I intend to do anything that would cause compatibility breakage. I don't know about the other frameworks available, as I don't use them.

[azzamsa]

@Shinmera nice to meet you :)

pre-written applications: https://github.com/orgs/Shirakumo/teams/radiance/repositories give me 404

[Shinmera]

Bah, I guess that page is private to team members. This should work instead: https://github.com/Shirakumo?utf8=✓&q=radiance&type=&language=

[azzamsa]

why I just found this, Web search engine does not tell me about it . I need 2 days to find out my error using external css file in restas, and it solved yesterday when I use caveman.

Radiance has complete tutorials.

[vindarel]

@vindarel I think this is the area the Cookbook should focus on over the next two months or so.

Ok ! I'm still taking notes on the matter (here, just a TOC and deployment examples). I began a todobackend in Ningle (here, very basic) but it's a difficulty after another (changing headers… now I can't even run a lisp on my VPS. SBCL complains about "ensure_space" and I can't use another lisp… (edit: fixed).

Given the two pre-requisites from above, that I agree with, it didn't seem doable just yet:

I think we should exclude any web framework that isn't relatively stable at a 1.0+ level [pnathan]

that leaves… Caveman, Radiance, Ningle ? But Clack's doc is lacking.

I'd also make sure that the framework has a couple of example applications already out there and has a sizeable amount of documentation to offer as well.

that leaves… nothing, except Radiance ? I know 1 example for Caveman or Ningle. Would appreciate interesting examples (more complete than a hello world).

I think Caveman's README is not bad. We could do something similar for the Cookbook but there is no much added value. I'd like to add information: testing, deployment, DB migrations (if possible),… or maybe just to show a complete simple project is already helpful.

My pb with Radiance is that I don't fully understand where it lies, if it is more a publication framework (CMS) or a full-featured web framework (Flask/Django-like). Does it help in defining DB models, in writing forms, in writing migrations, in giving a testing environment, etc ? It also announces limitations:

Data types are also restricted to integers, floats, and strings. Despite these constraints, the database interface is sufficiently useful for most applications.

and the terminology is different, so it's a more difficult than others to comprehend. And again, do we have to stick with cl-who ? Can we easily work with SQLite or Postgres ? etc

So let's see what we can do to port it into a Cookbook tutorial [about Quickutil]

Did you think about re-doing Quickutil in the Cookbook ? Not fully satisfactory to me because it doesn't use an ORM but raw SQL. It also uses closure-templates, that is maybe good but unusual for templates, I didn't encounter them before. If it is about to base our example on Ningle: why not. But its doc doesn't introduce templates and DBs, so we'll have to find our way. Shall we do a todobackend implementation ? (a todo-app)

Lastly, @pnathan, if you prefer to focus on other areas when you have time, that would still be great (error handling for example :D edit: PR of mine). I know you'll still help here in defining goals and proofreading :)

---

I didn't link to a new Weblocks branch (too alpha) but FYI its new Quickstart is pretty great: http://40ants.com/weblocks/quickstart.html simple and does backend and ajax-based clientside without JS.

[azzamsa]

maybe this help even little bit, https://www.slideshare.net/fukamachi/about-clack

[Shinmera]

if it is more a publication framework (CMS) or a full-featured web framework (Flask/Django-like).

It is both and neither at the same time, depending on how you use it.

Does it help in defining DB models

I don't know what you mean by this, but I assume the answer is no.

in writing forms

No, but iirc there are libraries for this.

in writing migrations

No.

in giving a testing environment

Again, not sure what you mean by this, but the answer is probably no.

It also announces limitations:

Radiance doesn't force you to do anything. You can use postmodern or whatever if the interfaces it provides are too restrictive.

And again, do we have to stick with cl-who

No, and I don't understand where you could have even got the idea from.

Can we easily work with SQLite or Postgres

Yes.

Radiance is different from most frameworks out there because it tries to do different things. I suggest you read some more about it.

[azzamsa]

@vindarel Sorry for asking, but what is the status for this (tutorial for web development).

I think we have to choose one of the web framework out there, then we make the tutorial or complete the documentation, for now we really need a good documentation for web app with CL.

After finishing with one framework, we can add other framework too.

[azzamsa]

This syllabus can give us some information, what we need to include in tutorial.

[Shinmera]

Radiance already has a lengthy tutorial out there if that's all you want. It isn't an introduction to web development, though, and thus doesn't cover JS, CSS, and HTML.

[azzamsa]

@Shinmera wow, that's cool, I think it's better if you put it in ReadtheDocs

[vindarel]

@azzamsa Given our prerequisites (stable version, real world examples), I'm not 100% keen on writing a tutorial here right now because there is no one method, I want to explore more possibilities (Radiance, Weblocks/Reblocks) and I don't want to write about Caveman or Ningle that I found awkward . So, as for web frameworks, we might need to rely on their own documentation. Thus kuddos to Radiance's documentation effort.

I carried on my efforts on the cookbook though, with two PRs, on condition handling and building executables, also for web apps, so it fills a gap for that tutorial. We also need a DB section (I tried Mito successfully and found Crane unfinished and buggy, and I see Radiance includes its own ubiquitous-based bindings).

Lately, as I wrote you I pretty liked Snooze (by Sly and Yasnippet author). Specifying and accessing arguments is straightforward, it deals nicely with errors (interactive debugger or print the stacktrace in the browser or print a custom 404), and its author responded. I take notes on lisp-journey/web-dev.

Now you have several options and you'll have to make a choice…

[vindarel]

I re-discovered the asynchronous Wookie: http://wookie.lyonbros.com/docs/ I didn't recall the defroute macro, and it even has hooks. Looks like a great framework.

(I started to write a little something, I want to do this now)

[vindarel]

Hi guys, I have an open first PR about this all. https://github.com/LispCookbook/cl-cookbook/pull/250

[vindarel]

Wow, reading this thread again, what a journey!