Is it possible to use Djula templates in a (hopefully) self-contained binary?

[vindarel]

(I contacted Mariano by email, but since he already sent me a snippet to try out it's better if we continue the discussion in the open. I'll copy my question and his first answer(s))

I am getting my feet wet at running a web app from a binary, not from sources, and I am now trying to figure out how to use Djula templates from there.

Do you have any experience, any hints for this?

I use Deploy. I initialize Djula templates at run time, not build time, but that is not enough for a binary, because the templates are not in a "src/templates/" directory any more, they are compiled and somewhere else in the binary, maybe…

For example:

(defun init-error-templates ()
   "Set the default error templates to the system's absolute location.
   Called at startup."
   (format! t "--- setting default error templates to ~a~&"
            (setf *error-template-directory*
               (asdf:system-relative-pathname :cosmo
  *default-error-template-directory*))))

I used asdf:system-relative-pathname, it's necessary or even required when working on the REPL (where the current working directory can be anything).

I learnt how to declare my templates as static files in the .asd:

                (:module "src/templates"
                         :components
                         ((:static-file "admin.html")
                          (:static-file "base.html")
                          (:static-file "list.html")
                          …

but I don't know how to access them. (edit) I learned how to list them and get their names and their absolute path: https://github.com/lisp-tips/lisp-tips/issues/37

But I still don't know if I can use them with my binary.

Finally, on Discord, Shinmera shows me the use of deploy:runtime-directory: https://github.com/Shirakumo/trial/blob/master/toolkit.lisp#L46-L50 maybe it helps…

I understand that, to be present in a self-contained binary, a Lisp thing has to be compiled. So maybe I should declare all the templates I use as parameters at build time.

---

addition: I also went to the 40ants' Gitter, Alexander created a demo project to show how to embed any static resource in a binary: https://github.com/svetlyak40wt/cl-static-resources and yes, it's easy (just create a defvar and store the file content), but it doesn't use Djula templates.

[vindarel]

Mariano answers:

---

I don't have experience with this, but perhaps I can give you some hint on how I would try .

DJULA:RENDER-TEMPLATE works with pathnames, but also with compiled templates (functions). compiled-templates are funcallable-standard-class. There's that, and there's a store class (file-store) and a global variable (current-store*), where templates are stored. I think you could implement a memory-store class, that respects the store api, and after compiling the templates, simply stores them and fetches from a hash-table instance variable. Have a look at template-store.lisp

I think that should work.

---

I think you also have to have :djula-prod in features, as compiled templates look at disk if that's not the case. Have a look at compiler.lisp.

Also note that compiled-template closures are stored in a 'compiled-template' slot in 'compiled-template' class.

---

(important:)

Djula is reliant on file system for linked templates; includes and extends tags, unfortunately. So, this may not be straightforward to pull out.

---

What happens if you use this:? (untested)

(defclass cached-template-store (file-store)
  ((templates :initform (make-hash-table :test 'equalp))
   (templates-contents :initform (make-hash-table :test 'equalp)))
  (:documentation "A template store with a memory cache."))

(defmethod find-template ((store memory-template-store) name &optional (error-p t))
  (with-slots (templates templates-contents) store
    (or (gethash name templates)
    (let ((template (call-next-method)))
      (when template
        (setf (gethash name templates) template)
        (setf (gethash name templates-contents) (slurp template)))
      template))))

(defmethod fetch-template ((store memory-template-store) name)
  (with-slots (templates-contents) store
    (or (gethash name templates-contents)
    (call-next-method))))

(setf djula::*current-store* (make-instance 'cached-template-store))

You have to make sure to pre-compile all your templates first, so that they get cached. If that works as I expect, you should be able to access all your templates from memory then.

---

I think I did something unnecessary there, and this may be enough:

(in-package :djula)

(defclass cached-template-store (file-store)
  ((templates-contents :initform (make-hash-table :test 'equalp)))
  (:documentation "A template store with a memory cache."))

(defmethod fetch-template ((store memory-template-store) name)
  (with-slots (templates-contents) store
    (or (gethash name templates-contents)
    (let ((template-content (call-next-method)))
      (setf (gethash name templates-contents) template-content)
      template-content)))) 

---

[vindarel]

@mmontone the defclass is a cached-template-store, should it be a memory-template-store? Otherwise the latter is undefined.

My adapted code: is that correct?

(in-package :djula)

(defclass memory-template-store (file-store)
  ((templates-contents :initform (make-hash-table :test 'equalp))
   (templates :initform (make-hash-table :test 'equalp)))  ;; <------------ also a templates slot is required
  (:documentation "A template store with a memory cache."))

(defmethod fetch-template ((store memory-template-store) name)
  (with-slots (templates-contents) store
    (or (gethash name templates-contents)
    (let ((template-content (call-next-method)))
      (setf (gethash name templates-contents) template-content)
      template-content))))

(defmethod find-template ((store memory-template-store) name &optional (error-p t))
  (declare (ignorable error-p))
  (with-slots (templates templates-contents) store
    (or (gethash name templates)
    (let ((template (call-next-method)))
      (when template
        (setf (gethash name templates) template)
        (setf (gethash name templates-contents) (slurp template)))
      template))))

(setf djula::*current-store* (make-instance 'memory-template-store))

[vindarel]

Now I grab all my templates declared in the .asd and try to djula:compile-template* them (is that the thing to do?):

;; Now let's grab all our templates defined in the .asd
;; and compile them with Djula.

(defun get-component-templates (sys component)
  "sys: system name (string)
   component: system-component name (string)
   Returns: a list of absolute pathnames."
  (let* ((sys (asdf:find-system sys))
         (module (find component (asdf:component-children sys) :key #'asdf:component-name :test #'equal))
         (alltemplates (remove-if-not (lambda (x) (typep x 'asdf:static-file))
                                      (asdf:module-components module))))

    (mapcar (lambda (it) (asdf:component-pathname it))
            alltemplates)))

(let* ((paths (get-component-templates "cmdcollectivites" "src/templates")))
  (loop for path in paths
     do (uiop:format! t "~&Compiling template file ~a…" path)
       (djula:compile-template* path))
  (values t :all-done))

My .asd (elluded):

               (:module "src/templates"
                        :components
                        ;; Order is important: the ones that extend admin.html
                        ;; must be declared after it, because we compile all of them
                        ;; at build time.
                        ((:static-file "base.html")
                         (:static-file "admin.html")
                         (:static-file "list.html")

and "admin.html" extends "base.html":

;; admin.html
{% extends "base.html" %}

So I try to build my app, as before, with Deploy and asdf:make, but:

.........
Compiling template file /home/vince/projets/ruche-web/commandes-collectivites/src/templates/base.html… = OK
Compiling template file /home/vince/projets/ruche-web/commandes-collectivites/src/templates/admin.html…Unhandled TEMPLATE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING
                                      {10007285B3}>:
  {# Error: There was an error processing the token (TAG EXTENDS base.html) : Template base.html not found #}

It chokes on the extend tag.

[vindarel]

We'd need a reproducible example.

[mmontone]

@mmontone the defclass is a cached-template-store, should it be a memory-template-store? Otherwise the latter is undefined.

My adapted code: is that correct?

Yes. That looks correct to me. My hope was that wrapping and caching find-template and fetch-template transparently you could have all templates in memory. I don't understand why you would get a template-not-found error, but haven't tried this on some example, yet ...

[mmontone]

Now I grab all my templates declared in the .asd and try to djula:compile-template* them (is that the thing to do?):

Yes. That was my idea.

[vindarel]

We'd need a reproducible example.

Here's an example: https://github.com/vindarel/demo-djula-in-binaries

[mmontone]

I know where the problem is. Cache is storing fullpath as key, but template is trying to be fetch by filename only.

[mmontone]

Ah. No. We forgot to add templates directory:

(djula:add-template-directory (asdf:system-relative-pathname :demo-djula-in-binaries "src/templates/"))

If you include "admin.html", then Djula needs some directories in which to search that.

[vindarel]

Indeed! I did it, but too late, in the other file.

Now the two templates compile correctly. Great! I carry on the rest of the test… (GTG soon)

(I pushed to the demo)

[vindarel]

I tried in another project: built with this technique, moved the templates directory to be in test conditions, ran the binary:

An error occured:
 {# Error: There was an error processing the token (TAG EXTENDS base.html) : Failed to find the WRITE-DATE of /home/vince/projets/myapp/src/templates/base.html:
                                                                              No such file or directory #} ==> Epilogue.

I can't tell with the reproducible example because I have difficulties building it right now (#+linux (deploy:define-library cl+ssl::libssl :dont-deploy T) issues)

[vindarel]

I confirm with the test demo (commited), I see the templating rendering error in the browser:

Failed to find the WRITE-DATE of /home/vince/bacalisp/demo-djula-in-binaries/src/templates/admin.html: No such file or directory.
In #<COMPILED-TEMPLATE /home/vince/bacalisp/demo-djula-in-binaries/src/templates/admin.html {2042C0CB}>.
Date/time: 2022-06-18-15:15!
An unhandled error condition has been signalled:
Failed to find the WRITE-DATE of /home/vince//bacalisp/demo-djula-in-binaries/src/templates/admin.html:
No such file or directory

Backtrace for: #
0: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE-TO-STREAM #)
1: (TRIVIAL-BACKTRACE:PRINT-BACKTRACE #" {100635EC03}> :OUTPUT NIL :IF-EXISTS :APPEND :VERBOSE NIL)
2: (DJULA:RENDER-TEMPLATE* # NIL)
3: ((:METHOD HUNCHENTOOT:HANDLE-REQUEST (HUNCHENTOOT:ACCEPTOR HUNCHENTOOT:REQUEST)) # #) [fast-method]

[mmontone]

Try with: (push :djula-prod *features*)

[vindarel]

Try with: (push :djula-prod *features*)

Just tried! And…

I don't want to have false hopes, but it seems to be working O_o I'll cool down before being extatic.

[vindarel]

So, yes, it works :)

What about: you push the code, I add documentation?

I also suggest a change:

(defmethod find-template ((store memory-template-store) name &optional (error-p t))
  (declare (ignorable error-p))
  (with-slots (templates templates-contents) store
    (or (unless (stringp name) ;; <------
          ;; if it is a string: that means we are looking for an "extends" name, 
          ;; otherwise the method receives pathnames.
          ;; If we have one base system whose templates get compiled,
          ;; then a second system who defines new templates with the same name in order to override them,
          ;; we want to search again the full pathname. That way we enable overrides. 
          ;; So, only if it's not a string, get the template from the hash-table cache:
          (gethash name templates))
        (let ((template (call-next-method)))
          (when template
            (setf (gethash name templates) template)
            (setf (gethash name templates-contents) (slurp template)))
          template))))

I struggled a bit to port this mechanism to my application, but finally it's done. One issue was this one described.

For my first app I used declared and compiled Djula templates as indicated in the doc: (defparameter +base.html+ (djula:compile-template* …)). Then I added a wrapper to not have to declare them and be more dynamic (very nice for development, similar to Caveman), so it searched the file system at every call. Sounded clever, but it's adding difficulties now… because I want to ship binaries so badly, I have to handle compile-time with this memory store, or the normal way… if anybody is interested, ping me, I can show code but it's currently private.

[mmontone]

Please have a look at commit https://github.com/mmontone/djula/commit/1e21020e38c17835f5b16953140b225bfd28e9f1 One of the breaking changes is that I replaced :dula-prod feature by a *recompile-templates-on-change* global variable.

I attach a patch to your djula in binaries demo: 0001-Remove-patches.patch.gz