[Q] Help setting up i18n and xgettext?

[vindarel]

Hi there, I'd be obliged if someone could help me set up i18n with Djula.

Looking at the doc, we have clean explanations on how to mark translatable strings in templates.

{% trans "hello" %}

{_ "hello _}

I am trying the gettext backend.

(gettext:setup-gettext #:abstock "abstock")

(gettext:preload-catalogs #.(asdf:system-relative-pathname :abstock "locale/"))

(setf gettext:*current-locale* "fr")

The first line creates the _ function: (_ "hello") returns "hello".

There is the update-translations.sh script to help use xgettext and friends.

=> I am trying to extract template strings and code strings with xgettext, to no avail:

mkdir locale
$ xgettext --from-code=UTF-8 -o locale/myapp.pot src/templates/search-form.html  
=> xgettext: warning: file 'src/templates/search-form.html' extension 'html' is unknown; will try C

$ xgettext --from-code=UTF-8 -o locale/abstock.pot src/myfile.lisp
=> nothing (what patterns does xgettext recognize for lisp anyways?)

in both cases locale/myapp.pot is not created.

While this isn't strictly related to Djula any help would be appreciated. Then I'll contribute doc :)

[mmontone]

Ok, give me some time to try this. I remember gettext setup being a bit "tricky".

[vindarel]

"tricky", exactly :D

I'm progressing: adding the option -a collects the strings and creates the .pot file. It collects all strings, not only the ones in between (_ ), so that is problematic. Maybe it isn't the right option.

[mmontone]

Haven't looked anything at all yet, but I have this in one of my projects:

Makefile:

gettext-extract: ## Extract gettext translations from source files
        sbcl --eval '(ql:quickload :invoice-engine-web)' --eval '(ie::xgettext-templates)' --quit
        find src -iname "*.lisp" | xargs xgettext --from-code=UTF-8 --keyword=_ --output=i18n/ie.pot --sort-output

gettext-edit: ## Edit the extracted gettext translations
        msgmerge --update i18n/nb.po i18n/ie.pot
        xdg-open i18n/nb.po

gettext-compile: ## Compile the edited gettext translations
        msgfmt --output-file=i18n/nb/LC_MESSAGES/ie.mo i18n/nb.po

gettext-init: ## Initialize gettext translations
        msginit --input=i18n/ie.pot --locale=nb --output=i18n/nb.po

i18n folder is like this:

marian@gimli ~/work/invoice-engine $ ls -R i18n
i18n:
ie.pot  nb  nb.mo  nb.po

i18n/nb:
LC_MESSAGES

i18n/nb/LC_MESSAGES:
ie.mo

Lisp:

(defun xgettext-templates ()
  (let ((messages
          (djula.locale:directory-translate-strings (asdf:system-relative-pathname :invoice-engine "web/templates/"))))
    (with-open-file (file (asdf:system-relative-pathname :invoice-engine "web/templates-messages.lisp")
                          :direction :output :if-exists :supersede
                          :if-does-not-exist :create)
      (let ((*standard-output* file))
        (write-string ";; THIS FILE IS AUTOGENERATED. DON'T CHANGE BY HAND. USE XGETTEXT-TEMPLATES FUNCTION.")
        (terpri)
        (loop for message in messages
              do
                 (prin1 `(gettext ,message))
                 (terpri))))))

(setf djula:*translation-backend* :gettext)
(setf djula::*gettext-domain* "ie")
(setf djula::*default-language* :nb)
(setf djula::*current-language* :nb)

(gettext:setup-gettext :invoice-engine "ie")
(gettext:preload-catalogs #.(asdf:system-relative-pathname :invoice-engine "../i18n/"))
(setf gettext:*current-locale* "nb")

There may be something useful there. Hope we can write some recipes if that's the case.

[vindarel]

Thank you. Reading the manual and the script better I found out:

• I must use -k_ with the lisp files using (_ ), or as you do --keyword=_
  • also -c adds comments surrounding the translation string, if any.
  • -j joins the results to the pot (not overwriting it)
• I don't know yet what to use for the .html templates (-a extracts all strings)

will update this comment if I progress…

(ps: translate is easy to use. Might be added as an Djula i18n backend?)

[mmontone]

(ps: translate is easy to use. Might be added as an Djula i18n backend?)

Sure. We can add to TODO.

[mmontone]

• I don't know yet what to use for the .html templates (-a extracts all strings)

Note that I'm using a trick for that; perhaps there's a better way, but this is what I'm doing: Use xgettext-templates function above to generate a lisp file with gettext calls. The file ends up looking like this:

;; THIS FILE IS AUTOGENERATED. DON'T CHANGE BY HAND. USE XGETTEXT-TEMPLATES FUNCTION.
(GETTEXT "Activity")
(GETTEXT "Address 1")
(GETTEXT "Address 2")
(GETTEXT "Amount")
(GETTEXT "API")
(GETTEXT "Balance")
...

After that, we have everything in lisp files, both the translations from the templates and the lisp source files.

Then the xgettext tool does the extraction from every lisp file, including the generated from templates lisp file above:

find src -iname "*.lisp" | xargs xgettext --from-code=UTF-8 --keyword=_ --output=i18n/ie.pot --sort-output

Note that it is the GNU xgettext doing the extraction and detecting those (gettext "message") lisp expressions.

https://www.gnu.org/software/gettext/manual/html_node/xgettext-Invocation.html

[mmontone]

Also, there's an example with gettext in demo directory.

[vindarel]

I have now successfully merged gettext support in my project, thanks to @fstamour. We can see the PR here: https://gitlab.com/myopenbookstore/openbookstore/-/merge_requests/19

Of particular interest, including maybe for Djula: https://gitlab.com/myopenbookstore/openbookstore/-/blob/master/src/i18n.lisp

There should be instructions on the my project's readme.

Nonetheless, useful snippets:

;; introspection:
(gettext:preload-catalogs
   ;; Tell gettext where to find the .mo files
   #.(asdf:system-relative-pathname :bookshops "locale/"))
;; =>
#S(GETTEXT::CATALOG
   :KEY ("fr_fr" :LC_MESSAGES "bookshops")
   :HEADERS ((:PROJECT-ID-VERSION . "PACKAGE VERSION")
             (:REPORT-MSGID-BUGS-TO . "")
             (:PO-REVISION-DATE . "YEAR-MO-DA HO:MI +ZONE")
             (:LAST-TRANSLATOR . "FULL NAME <EMAIL@ADDRESS>")
             (:LANGUAGE-TEAM . "LANGUAGE <LL@li.org>") (:LANGUAGE . "")
             (:MIME-VERSION . "1.0")
             (:CONTENT-TYPE . "text/plain; charset=CHARSET")
             (:CONTENT-TRANSFER-ENCODING . "8bit"))
   :NPLURALS 2
   :PLURALS-FUNCTION #<FUNCTION (LAMBDA (GETTEXT::N)) {542C149B}>
   :MESSAGES (SERAPEUM:DICT
               "Login" '("Se connecter")
               "Password" '("Mot de passe")
               "Please login to continue" '("Veuillez vous identifier pour continuer")
               "Results: ~a. Page: ~a/~a~&" '("Resultats: ~a. Page: ~a/~a~&")
               "Welcome to OpenBookStore" '("Bienvenue dans OpenBookStore")
              ) )

;; to reload the translations while developing:

;; Run this when developping to reload the translations
#+ (or)
(progn
  ;; Clear gettext's cache
  (clrhash gettext::*catalog-cache*)
  (gettext:preload-catalogs
   ;; Tell gettext where to find the .mo files
   #.(asdf:system-relative-pathname :bookshops "locale/")))

;; Set the locale:
(set-locale "fr_fr")

In Djula templates, we use the {_ "hello" _} notation. We have a make tr Makefile target to generate the .po and .mo files.

Translations are also available when we compile templates in memory and build a standalone binary.

---

We could close this issue, unless we want to improve the out-of-the-box experience and documentation and/or the current integration example.

[mmontone]

I'll add to the docs.

[mmontone]

Added some to the docs: http://mmontone.github.io/djula/djula/Internationalization.html#Gettext

A bit sloppy but I hope it's better than nothing.

[fstamour]

Sloppy doc is infinitely better than no doc. Thanks for taking the time to do that!

[fstamour]

Maybe add a note that asdf (e.g. asdf:system-relative-pathname) shouldn't be called from an image/executable because the system file probably doesn't even exists in a released application.

If djula::xgettext-templates is meant to be called by the user of djula, shouldn't be exported?