How to structure a big caveman2 project?

[daninus14]

Hi this more of a guidance or documentation question than an issue:

How can I go about splitting large parts of a web app into modules to make it more manageable for writing .asd files and separating functionality.

For example, say I want to have an authentication module which has its own classes, routes, and views: How can I define it in a system say in /src/auth/auth.asd and be able to use those routes in the main web app by importing it?

To use another package to define routes is itself not so clearly documented: https://stackoverflow.com/questions/76330818/how-to-define-caveman2-routes-in-multiple-files-packages/76332385#76332385

But given that defroute is expecting an *app* it looks like an app should be created as a caveman2 project, and then modules created outside of the project, and then another system should import the caveman2 project as a system, and then the other modules would depend on the caveman2 project as a base.

Is that correct?

[jackcarrozzo]

The way I have solved this is to have lots of packages (myproj.whatever) export functions etc that do things, then a myproj.http that depends on all of them and ties them into the app

[daninus14]

@jackcarrozzo thanks for the reply!

Yeah, that's what I was doing, but it started getting out of control in the .asd file importing every file. Unless you are referring to doing packaged inferred systems, the asd file bloats up.

I realized the way to do it (for whoever reads this) is that I can add a my-project.base.asd package which contains

• config.lisp
• db.lisp
• view.lisp
• web.lisp
• main.lisp

and then the original will contain the dependency on this my-project.base.asd and any other asd projects you declare. That way you can make smaller system definition files and encourage proper breaking up of the application code into more independent modules you can reuse over applications.

A key thing to think about here if you want to make your modules really independent is to not make them depend on the base system. For controllers, it get's tricky because of the macro defroute. So probably what I'll do is make a new macro that will instead make a list of routes to be defined once a *app* variable is provided, and then call the ningle function to register a route (which is what defroute does when macroexpanded) on all those routes to be defined.

So there will be in themy-project.asd file a few dependencies

• my-project.base
• my-project.auth (for example of an independent module which would declare routes etc)
• my-project.main which would then tie in auth into the project and would call some function/macro defined in the my-project.auth package and pass it the my-project.web:*web* app variable and define all relevant routes.

Note that in this example, if you are defining some user class or anything that is persisted on the db, the my-project.main will also be responsible for executing the code to persist that class exported or extend it applying specific project related data and then persist it.

Ok, I think this finishes this guide for structuring large projects.

Cheers!

[vindarel]

it get's tricky because of the macro defroute. So probably what I'll do is make a new macro that will instead make a list of routes to be defined once a app variable is provided,

On this, there might be an Hunchentoot feature that helps, but I don't know Clack. The thing is that you create an acceptor and define a route on it: so far so good. You can define a second Hunchentoot acceptor on another port (= web app in my head), define a new route… and all routes are available. The macro define-easy-handler (and other macros like easy-route) accept an acceptor-names parameter that allows to restrict the route to the given acceptors.

So, this could be a way to make re-usable components: to each its acceptor, but by default all are visible and accessible.

+1 for the .asd organisation.

ps: it could be nice to have Github discussions, to chat more freely.

pps: this feature is annoying when you want to develop more than one web app at the same time on the same image^^ I haven't gotten to an easy setup. I start a second Emacs/Slime.

[daninus14]

@vindarel thanks for the reply.

The thing is that you create an acceptor and define a route on it: so far so good. You can define a second Hunchentoot acceptor on another port (= web app in my head)

1. What is an acceptor?
2. Are you talking about developing multiple web apps simultaneously on the same emacs instance and how to define a route for the right app? I was talking more about within one app, how to break up the code into functionality modules which can be exported into libraries and be independent of the project where routes are defined. However since the defroutes take the *app* variable and then makes it dependent on the web.lisp file, it's hard to make it independent, and was talking about how to go about making it independent. Is that what you are talking about as well?

ps: it could be nice to have Github discussions, to chat more freely.

Yeah, originally that was my intent but couldn't find the discussions tab in the project

[vindarel]

Yes I think I am talking about something that would solve your issue/requirements (but not with Clack, which I don't know).

An acceptor is well… an *app*/an Hunchentoot server instance, defined on a given port.

You can create multiple acceptors/web servers, in the same lisp image, each on a different port.

The thing is that by default, when you create a route (define-easy-handler (say-yo :uri "/yo") (name) …) this creates the route globally for all Hunchentoot servers!

how to break up the code into functionality modules which can be exported into libraries

So yes I think that's a default Hunchentoot feature which allows to do this (and which as many lisp things I didn't understand the reason for at first). You can have independent lisp systems that create their own Hunchentoot server. When they are loaded in the same image, their routes are all globally accessible (if you don't take extra care to isolate them to their own acceptor).

You don't have to take care where you create *the* app, you create many, and they work together.

There's kinda an example here: https://lispcookbook.github.io/cl-cookbook/web.html#hunchentoot-2 with the note at the end of the paragraph: "Just a thought…"

[jackcarrozzo]

I agree with the points you guys raised- I have run into the same things. One thing I will add is that I found moving from the asd def to a quicklisp def for the package/s made it much easier to keep track of who is importing whom and so forth. I still end up with circular dependencies from time to time… all ears if there is a Recommended Method for this kind of layout.

[daninus14]

The thing is that by default, when you create a route (define-easy-handler (say-yo :uri "/yo") (name) …) this creates the route globally for all Hunchentoot servers!

Oh wow! That would not be what I would expect. It basically makes it very complicated to make an image or project which could handle multiple domains with differing functionality. This would require starting a different image for each project which seems like unnecessary overhead.

I was using caveman2 which allows switching the underlying http server. I think they even recommend Hunchentoot for development and Woo for production because it's fast. I think the ideal solution should be server independent because otherwise we loose all the supposed benefit of the caveman2/ningle frameworks.

You don't have to take care where you create the app, you create many, and they work together.

That does solve the issue, but I agree with you that it's not the ideal approach.

There's kinda an example here: https://lispcookbook.github.io/cl-cookbook/web.html#hunchentoot-2 with the note at the end of the paragraph: "Just a thought…"

Thanks for the examples and explanations. Had I not known this behavior I could see myself wasting time debugging the routes on multiple projects in the same image for hours.

One thing I will add is that I found moving from the asd def to a quicklisp def for the package/s made it much easier to keep track of who is importing whom and so forth.

What do you mean by a quicklisp def?

I think I have an approach, just need to test it. Once I get it working I'll probably make an example about it.

[vindarel]

Oh wow! That would not be what I would expect. It basically makes it very complicated to make an image or project which could handle multiple domains with differing functionality.

wow, exactly^^ I share the overall feeling and conclusions.

I wouldn't say "very" complicated, since the route has a key argument to set the acceptors to which define the route. But this adds an overhead.

This would require starting a different image for each project which seems like unnecessary overhead.

So it isn't mandatory but it's what I prefer to do in the end. Note that this is what we do with any other language that doesn't have an image-based approach: just start a new interpreter in the terminal. But in CL, we have the image, and as spoiled users we want to do more, everything in one image, so starting a new Lisp process seems a regression to us. It shouldn't be, IMO^^