search

d12frosted / vulpea

A collection of functions for note taking based on `org` and `org-roam`.
GNU General Public License v3.0
236 stars 12 forks source link
emacs-lisp org-mode org-roam vulpea

readme

+OPTIONS: toc:nil

+begin_html

Banner

MELPA MELPA Stable

+end_html

A collection of functions for note taking based on =org= and =org-roam=. This repository primary goal is to be a tested library for other applications and utilities around note taking.

Users of this library:

** Reasons to use =vulpea= :PROPERTIES: :ID: b3a6ccac-d32f-4e27-a50a-012063fbc08e :END:

  1. If you are automating things around Org mode files and you want to have a tested library to build upon.
  2. While =org-roam= was greatly simplified and has adopted some ideas from =vulpea= starting from =v2=, it still lacks certain pieces for programmatic usage. Just to name few of them:
    1. =vulpea-db-query= is a great interface for quick database search without knowing its scheme. In many cases you just want to have a fully materialized note instead of a structure that lacks, say tags or aliases. So you don't need to write any extra SQL. See [[id:5b44d873-179a-4fcb-88df-ff8a8d328bd0][Performance]] for more information.
    2. =vulpea-select= is a configurable interface for selecting a note with an optional filter. See [[id:485b4e0f-22a1-4ab5-88bd-98d002b6d69c][vulpea-select]] for more information.
    3. =vulpea-create= is a wrapper around =org-roam-capture-= allowing to easily provide extra properties, tags and context when creating a new note. And most importantly - it returns a newly created note. See [[id:b75b02e2-b574-4783-81d6-03ab4ed07c10][vulpea-create]] for more information.
  3. Several interactive functions:
    1. =vulpea-find= that acts like =org-roam-node-find=, but (a) uses =vulpea-select= for consistent experience and (b) allows to configure default and on-use filtering and candidates source. See [[id:29b53275-ec0c-4ab5-a86a-b42f4dae6c84][vulpea-find]] for more information.
    2. =vulpea-find-backlink= is just a convenient function to find a note linking currently opened one. For those who don't want to use =org-roam= buffer.
    3. =vulpea-insert= that acts like =org-roam-node-insert=, but (a) uses =vulpea-select= for consistent experience, (b) allows to set =vulpea-insert-default-filter= (see =vulpea-find= for examples) and (c) allows to provide an insertion handler via =vulpea-insert-handle-functions=, which is called with inserted note. See [[id:210de8cb-b340-4245-8d45-013129ce0a82][vulpea-insert]] for more information.
  4. =vulpea= is more than just a wrapper around =org-roam=, it contains more stuff:
    1. Utilities for dealing with buffer properties (e.g. =#+KEY: VALUE=).
    2. Utilities for dealing with [[id:e0f6439c-8818-471d-ac25-c9dda830df3a][metadata]] (e.g. first description list in the buffer).

** Install :PROPERTIES: :ID: b946c716-e3b3-4c84-8229-dde59ddd55ae :END:

=vulpea= is available via [[https://melpa.org/#/vulpea][MELPA]], but you can still install it manually, using [[https://github.com/raxod502/straight][straight]], [[https://github.com/quelpa/quelpa][quelpa]], or any package management tool of alike.

In short, installation process is simple:

  1. Download =vulpea= package via any preferred way.
  2. Call =vulpea-db-autosync-enable= (either via adding a hook to =org-roam-db-autosync-mode= or directly).
  3. Before first usage you need to re-sync =org-roam-db= from scratch, e.g. =(org-roam-db-sync 'force)=. This is because =vulpea= has several custom tables in addition to what =org-roam= provides.

*** =use-package= :PROPERTIES: :ID: 21ef9eb7-9613-4246-a603-8ecffba19246 :END:

+begin_src emacs-lisp

(use-package vulpea :ensure t ;; hook into org-roam-db-autosync-mode you wish to enable ;; persistence of meta values (see respective section in README to ;; find out what meta means) :hook ((org-roam-db-autosync-mode . vulpea-db-autosync-enable)))

+end_src

*** =straight.el= :PROPERTIES: :ID: 501a4489-83cc-4541-8edc-89b04ac866b5 :END:

+begin_src emacs-lisp

(straight-use-package '(vulpea :type git :host github :repo "d12frosted/vulpea"))

;; hook into org-roam-db-autosync-mode you wish to enable persistence ;; of meta values (see respective section in README to find out what ;; meta means) (add-hook 'org-roam-db-autosync-mode-hook #'vulpea-db-autosync-enable)

+end_src

In case you have [[https://github.com/raxod502/straight.el/#integration-with-use-package][integration]] with [[https://github.com/jwiegley/use-package][use-package]]:

+begin_src emacs-lisp

(use-package vulpea :straight (vulpea :type git :host github :repo "d12frosted/vulpea") ;; hook into org-roam-db-autosync-mode you wish to enable ;; persistence of meta values (see respective section in README to ;; find out what meta means) :hook ((org-roam-db-autosync-mode . vulpea-db-autosync-enable)))

+end_src

*** Doom Emacs :PROPERTIES: :ID: 9ebbac14-4032-42e0-bbb6-d38342a8bf04 :END:

+begin_src emacs-lisp

(use-package! vulpea :hook ((org-roam-db-autosync-mode . vulpea-db-autosync-enable)))

+end_src

** =vulpea-note= :PROPERTIES: :ID: 22aa7af5-fc57-4813-9e96-afdfce663e00 :END:

A note is represented as a =vulpea-note= structure with the following slots/fields:

If =ID= is not present in the note structure, this note is treated as non-existent. For example, =vulpea-select= returns such a note, when =require-match= is =nil= and the user selects non-existent note.

Example of a note:

+begin_src emacs-lisp

(vulpea-db-get-by-id "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c")

s(vulpea-note :id "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c"

:path "/Users/d12frosted/Dropbox/vulpea/20200407160812-kitsune_book.org" :level 0 :title "Kitsune Book" :primary-title nil :aliases ("vulpea" "Kitsune no Hon") :tags ("personal") :links (("https" . "https://github.com/d12frosted/vulpea") ("https" . "https://github.com/d12frosted/environment") ("https" . "https://github.com/d12frosted/vino")) :properties (("CATEGORY" . "20200407160812-kitsune_book") ("ROAM_ALIASES" . "vulpea \"Kitsune no Hon\"") ("ID" . "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c") ("BLOCKED" . "") ("FILE" . "/Users/d12frosted/Dropbox/vulpea/20200407160812-kitsune_book.org") ("PRIORITY" . "B")) :meta (("link" "[[https://github.com/d12frosted/vulpea][vulpea]]") ("users" "[[https://github.com/d12frosted/environment][environment]]" "[[https://github.com/d12frosted/vino][vino]]") ("status" "stable")))

+end_src

** Metadata :PROPERTIES: :ID: 6b5ef4a4-4cf9-49fb-9141-8858fef3a189 :END:

In general, metadata is a list of key value pairs that is represented by the first description list in the note, e.g. list like:

+begin_src org-mode

It can be manipulated programatically by using functions from either =vulpea-meta= module or from =vulpea-buffer= module (those prefixed by =vulpea-buffer-meta=). This data is also persisted in Org roam database for your convenience and is part of =vulpea-note= returned by =vulpea-db= module. See respective module documentation to find out all available functions.

Currently metadata is limited to file-level only, e.g. description lists in outlines are not handled by =vulpea=. Vote for [[https://github.com/d12frosted/vulpea/issues/75][vulpea#75]] to bring it faster.

*** Why not properties drawer :PROPERTIES: :ID: 1914d0a4-6e68-47b7-8d54-fc49cef24bf3 :END:

In many cases, properties are far better choice for storing technical 'metadata', like =ID=, =DATE=, =TAGS=, etc. - something that is not really part of note content. After all, properties drawer is a drawer:

+begin_quote

Sometimes you want to keep information associated with an entry, but you normally do not want to see it. For this, Org mode has drawers.

[[https://orgmode.org/manual/Drawers.html#Drawers][orgmode.org]]

+end_quote

Of course you can use [[https://orgmode.org/manual/Properties-and-Columns.html#Properties-and-Columns][properties drawer]] to implement simple database capabilities, but it has one important limitation - values are mere text, so you can't have real Org mode links there, meaning that [[https://orgmode.org/worg/dev/org-element-api.html][Element API]], Org roam and some other tools do not recognise them as links.

Metadata provided by library is just a part of your note content, meaning that it incorporates well into existing tools. Sure enough it's not as rich as properties and is not as battle tested as properties, but you can give them a try.

** Modules :PROPERTIES: :ID: c192e78f-08e0-4894-9fa9-a694f9e923f8 :END:

*** =vulpea= :PROPERTIES: :ID: b042b560-a2e4-451d-b44a-a290d1b0604d :END:

This one-stop module contains some generic functions that didn't find their place in separate modules. It also imports every other module.

**** =vulpea-find= :PROPERTIES: :ID: 29b53275-ec0c-4ab5-a86a-b42f4dae6c84 :END:

A one stop function to select and find (visit) a note that can be used both interactively (e.g. =M-x vulpea-find=) and programatically. In the later case it provides multiple configuration bits.

When =OTHER-WINDOW= argument is nil (default), the note is visited in the current window. In order to use the /other/ window, you may use universal argument during interactive usage (e.g. =C-u M-x vulpea-find=) or pass a non-nil value as argument:

+begin_src emacs-lisp

(vulpea-find :other-window t)

+end_src

When =REQUIRE-MATCH= argument is nil (default), user may select a non-existent note and the capture process is started. In order to disallow selection of non-existent note, pass non-nil value:

+begin_src emacs-lisp

(vulpea-find :require-match t)

+end_src

=vulpea-find= allows to configure candidates for selection in two ways - by controlling source of candidates and by controlling filtering function.

***** Filter function :PROPERTIES: :ID: 4278719d-56ce-4724-aaf3-b8323bdcc930 :END:

Filtering is easy. It's just a function that takes one argument - =vulpea-note= that is being filtered. You can configure default filtering function called =vulpea-find-default-filter= (so it is applied to interactive usage) or pass an override for the default filtering function.

For example, you wish to list only file-level notes during interactive usage of =vulpea-find= (to mimic how =org-roam-find= was behaving in v1). For that you just need to configure the value of =vulpea-find-default-filter= variable:

+begin_src emacs-lisp

(setq vulpea-find-default-filter (lambda (note) (= (vulpea-note-level note) 0)))

+end_src

But of course, it's possible to override this behaviour when =vulpea-find= is used programatically, just by passing filtering function as =FILTER-FN= argument:

+begin_src emacs-lisp

;; by default `vulpea-find' lists aliases, imagine that we want to ;; list only primary titles (vulpea-find :filter-fn (lambda (note) ;; primary-title is set only when title is one of the ;; aliases (null (vulpea-note-primary-title note))))

+end_src

***** Candidates function :PROPERTIES: :ID: 3b13b477-896e-4117-b580-8ba60066cc35 :END:

As it was already mentioned, =vulpea-find= allows to configure the source of candidates. This may be needed for performance considerations (e.g. to avoid filtering EVERY existing note in your database) or for some 'esoteric' features (like ordering).

By default =vulpea-db-query= is used as a source of candidates. Default source is controlled by =vulpea-find-default-candidates-source= variable. You should change it only when your intention is to configure behaviour of =vulpea-find= interactive usage. For example (an 'esoteric' one):

+begin_src emacs-lisp

(setq vulpea-find-default-candidates-source (lambda (filter) ;; sort notes by title, but keep in mind that your completion ;; framework might override this sorting, it's just an example (seq-sort-by

'vulpea-note-title

       #'string<
       (vulpea-db-query filter))))

+end_src

But in most cases you should not touch the configuration variable and instead apply an override via =CANDIDATES-FN= argument. For example, if you wish to 'find' a note linking to some specific note. Of course this can be achieved with a filtering function, but in this particular case performance can be drastically improved by overriding candidates source. You can achieve this by something along the lines:

+begin_src emacs-lisp

;; Let's say we have a note in the context. First, we use a ;; specialized query to find what links to a given note. (let ((backlinks (vulpea-db-query-by-links-some (list (cons "id" (vulpea-note-id note)))))) ;; Secondly, we override default CANDIDATES-FN, so it simply ;; presents us a list of backlinks. We deliberately ignore filtering ;; function. (vulpea-find :candidates-fn (lambda (_) backlinks) :require-match t))

+end_src

Don't rush into saving this function into your collection. It's already provided by =vulpea= as =vulpea-find-backlink=. Keep reading!

**** =vulpea-find-backlink= :PROPERTIES: :ID: a8152294-d4e0-41bf-8e11-e58c6d6f7adf :END:

An interactive function to select and find (visit) a note linking to the currently visited note. Keep in mind that outlines with assigned =ID= property are also treated as notes so you might want to go to beginning of buffer if you wish to select backlinks to current file.

**** =vulpea-insert= :PROPERTIES: :ID: 210de8cb-b340-4245-8d45-013129ce0a82 :END:

An interactive function to select a note and insert a link to it. When user selects non-existent note, it is captured via =org-roam-capture= process (see =org-roam-capture-templates=). Once the link is inserted, =vulpea-insert-handle-functions= is called with inserted note as an argument, so you can easily perform any necessary post-insertion actions. Selection is controlled in a similar way to =vulpea-find= - via global =vulpea-insert-default-filter= or local filter.

***** Filter function :PROPERTIES: :ID: b1162e36-b632-4b4a-a420-17e232364fd0 :END:

This argument is just a function that takes one argument - =vulpea-note= that is being filtered. You can configure default filtering function called =vulpea-insert-default-filter= (so it is applied to interactive usage) or pass an override for the default filtering function.

For example, you wish to list only file-level notes during interactive usage of =vulpea-insert= (to mimic how =org-roam-find= was behaving in v1). For that you just need to configure the value of =vulpea-insert-default-filter= variable:

+begin_src emacs-lisp

(setq vulpea-insert-default-filter (lambda (note) (= (vulpea-note-level note) 0)))

+end_src

But of course, it's possible to override this behaviour when =vulpea-insert= is used programatically, just by passing filtering function as =FILTER-FN= argument:

+begin_src emacs-lisp

;; by default `vulpea-insert' lists aliases, imagine that we want to ;; list only primary titles (vulpea-insert (lambda (note) ;; primary-title is set only when title is one of the ;; aliases (null (vulpea-note-primary-title note))))

+end_src

***** Candidates function :PROPERTIES: :ID: 3b13b477-896e-4117-b580-8ba60066cc35 :END:

Just like =vulpea-find=, =vulpea-insert= allows to configure the source of candidates. This may be needed for performance considerations (e.g. to avoid filtering EVERY existing note in your database) or for some 'esoteric' features (like ordering).

By default =vulpea-db-query= is used as a source of candidates. Default source is controlled by =vulpea-insert-default-candidates-source= variable. You should change it only when your intention is to configure behaviour of =vulpea-insert= interactive usage. For example (an 'esoteric' one):

+begin_src emacs-lisp

(setq vulpea-insert-default-candidates-source (lambda (filter) ;; sort notes by title, but keep in mind that your completion ;; framework might override this sorting, it's just an example (seq-sort-by

'vulpea-note-title

       #'string<
       (vulpea-db-query filter))))

+end_src

***** Insertion handler :PROPERTIES: :ID: 558e6704-76d0-4b6c-bd6c-28d91a5e0d89 :END:

There are cases when you want to react somehow to link insertion. For this =vulpea= provides a configuration variable =vulpea-insert-handle-functions=, which is kind of a hook with argument - =vulpea-note= that is linked.

For example, you want to tag an outline whenever a link to person is inserted (see some explanation of this use case in a dedicated [[https://d12frosted.io/posts/2020-07-07-task-management-with-roam-vol4.html][blog post]]). For that you need to define a handler function first:

+begin_src emacs-lisp

(defun my-vulpea-insert-handle (note) "Hook to be called on NOTE after `vulpea-insert'." (when-let* ((title (vulpea-note-title note)) (tags (vulpea-note-tags note))) (when (seq-contains-p tags "people") (save-excursion (ignore-errors (org-back-to-heading) (when (eq 'todo (org-element-property :todo-type (org-element-at-point))) (org-set-tags (seq-uniq (cons (vulpea--title-to-tag title) (org-get-tags nil t))))))))))

+end_src

And then you just need to add it as a hook:

+begin_src emacs-lisp

(add-hook 'vulpea-insert-handle-functions

'my-vulpea-insert-handle)

+end_src

**** =vulpea-create= :PROPERTIES: :ID: b75b02e2-b574-4783-81d6-03ab4ed07c10 :END:

This function enables programmatic creation of new notes without the need to configure =org-roam-capture-templaces=, but instead providing various bits to be inserted into new note. And yes, it returns you the created note. This function is heavily used in [[https://github.com/d12frosted/vino][vino]] and you can find several real world usage examples there.

The minimal usage example:

+begin_src emacs-lisp

(vulpea-create "Title of new note" "relative/path/to/%<%Y%m%d%H%M%S>-${slug}.org")

+end_src

This will create a note file =relative/path/to/20211119082840-title-of-new-note.org= with the following content:

+begin_src org

:PROPERTIES: :ID: 3dfd828f-fb73-41a6-9801-54bc17d41b57 :END: ,#+title: Title of new note

+end_src

As you can see, thanks to =org-roam-capture= and =org-capture= system, this allows expansion of formatted text as long as expansion of variables from capture context. Read further to learn more.

***** Synchronous vs asynchronous :PROPERTIES: :ID: 6cbb1043-18b8-47f1-a33b-9e0cea976188 :END:

By default capture process is 'asynchronous', meaning that it waits for user input and confirmation. In some cases, 'synchronous' creation is desired, so that note is created immediately and the created note is returned as result, so we can use it further. Example:

+begin_src emacs-lisp

(vulpea-create "immediate note" "%<%Y%m%d%H%M%S>-${slug}.org" :immediate-finish t)

s(vulpea-note

:id "5733ca9e-5b42-4b6b-ace9-2fef1091d421" :path "/Users/d12frosted/Dropbox/vulpea/20211119095443-immediate_note.org" :level 0 :title "immediate note" :primary-title nil :aliases nil :tags nil :links nil :properties (("CATEGORY" . "20211119095443-immediate_note") ("ID" . "5733ca9e-5b42-4b6b-ace9-2fef1091d421") ("BLOCKED" . "") ("FILE" . "/Users/d12frosted/Dropbox/vulpea/20211119095443-immediate_note.org") ("PRIORITY" . "B")) :meta nil)

+end_src

And the content of created file is:

+begin_src org

:PROPERTIES: :ID: 5733ca9e-5b42-4b6b-ace9-2fef1091d421 :END: ,#+title: immediate note

+end_src

How cool is that? Pretty cool, I'd say.

***** Extra content :PROPERTIES: :ID: 3872d1e9-e5c8-4944-814f-ad03c1fb0967 :END:

Of course, in many cases we want to add much more than that into note file. In general, the file has the following format:

+begin_src org

:PROPERTIES: :ID: ID PROPERTIES if present :END: ,#+title: TITLE ,#+filetags: TAGS if present HEAD if present

BODY if present

+end_src

So you can provide the following arguments controlling content:

Simple example to illustrate:

+begin_src emacs-lisp

(vulpea-create "Rich note" "%<%Y%m%d%H%M%S>-${slug}.org" :properties '(("COUNTER" . "1") ("STATUS" . "ignore") ("ROAM_ALIASES" . "\"Very rich note with an alias\"")) :tags '("documentation" "showcase") :head "#+author: unknown\n#+date: today" :body "It was a very nice day.\n\nBut I didn't feel that." :immediate-finish t)

s(vulpea-note

:id "568d4e29-76dd-4630-82f9-e1e2006bebdc" :path "/Users/d12frosted/Dropbox/vulpea/20211119095644-rich_note.org" :level 0 :title "Rich note" :primary-title nil :aliases ("Very rich note with an alias") :tags ("documentation" "showcase") :links nil :properties (("CATEGORY" . "20211119095644-rich_note") ("ROAM_ALIASES" . "Very rich note with an alias") ("STATUS" . "ignore") ("COUNTER" . "1") ("ID" . "568d4e29-76dd-4630-82f9-e1e2006bebdc") ("BLOCKED" . "") ("FILE" . "/Users/d12frosted/Dropbox/vulpea/20211119095644-rich_note.org") ("PRIORITY" . "B")) :meta nil)

+end_src

This creates the following note:

+begin_src org

:PROPERTIES: :ID: 568d4e29-76dd-4630-82f9-e1e2006bebdc :COUNTER: 1 :STATUS: ignore :ROAM_ALIASES: "Very rich note with an alias" :END: ,#+title: Rich note ,#+filetags: :documentation:showcase: ,#+author: unknown ,#+date: today

It was a very nice day.

But I didn't feel that.

+end_src

***** Context variables :PROPERTIES: :ID: 3cbca770-aa36-4f99-8dec-14d8552d0001 :END:

Any content piece (except for title) may have arbitrary amount of context variables in form =${VAR}= that are expanded during note creation. By default there are 3 context variables - =slug=, =title= and =id=. But you may add extra variables to the context by passing =context= variable:

+begin_src emacs-lisp

(vulpea-create "A Book" "${slug}.org" :context (list :name "Frodo") :immediate-finish t :properties '(("AUTHOR" . "${name}")) :tags '("@${name}") :head "#+author: ${name}" :body "This note was create by ${name}")

s(vulpea-note

:id "1fecedf8-ccda-4d68-875e-111b8cc5992e" :path "/home/borysb/Dropbox/vulpea/a_book.org" :level 0 :title "A Book" :primary-title nil :aliases nil :tags ("@Frodo") :links nil :properties (("CATEGORY" . "a_book") ("AUTHOR" . "Frodo") ("ID" . "1fecedf8-ccda-4d68-875e-111b8cc5992e") ("BLOCKED" . "") ("FILE" . "/home/borysb/Dropbox/vulpea/a_book.org") ("PRIORITY" . "B")) :meta nil)

+end_src

This creates the following note:

+begin_src org

:PROPERTIES: :ID: 1fecedf8-ccda-4d68-875e-111b8cc5992e :AUTHOR: Frodo :END: ,#+title: A Book ,#+filetags: :@Frodo: ,#+author: Frodo

This note was create by Frodo

+end_src

Please keep in mind that you cannot override the default context via =context= variable.

***** Mandatory ID :PROPERTIES: :ID: 7b3880e4-4aff-4b4f-8574-78886e4c03a4 :END:

By default =id= is being generated for you and you can not avoid it. This is what allows =vulpea-create= to return created note for you. In some cases you might want to provide =id= upfront instead of relying on generation. And =vulpea-create= has an argument for that.

+begin_src emacs-lisp

(vulpea-create "Custom id" "${slug}.org" :id "xyz" :immediate-finish t)

s(vulpea-note

:id "xyz" :path "/home/borysb/Dropbox/vulpea/custom_id.org" :level 0 :title "Custom id" :primary-title nil :aliases nil :tags nil :links nil :properties (("CATEGORY" . "custom_id") ("ID" . "xyz") ("BLOCKED" . "") ("FILE" . "/home/borysb/Dropbox/vulpea/custom_id.org") ("PRIORITY" . "B")) :meta nil)

+end_src

This creates the following note:

+begin_src org

:PROPERTIES: :ID: xyz :END: ,#+title: Custom id

+end_src

*** =vulpea-select= :PROPERTIES: :ID: 485b4e0f-22a1-4ab5-88bd-98d002b6d69c :END:

Common interface to select (e.g. =completing-read=) a note from the set of notes. Used in functions like =vulpea-find=, =vulpea-find-backlink=, =vulpea-insert=, etc.

+begin_html

Narrowing by aliases and tags

+end_html

There are two variants of selection: =vulpea-select-from= and =vulpea-select=. The difference between them is that the former accepts a list of notes to select from and the latter accepts a filter function which is applied to all notes in the database. Here are two examples to illustrate that:

+begin_src emacs-lisp

;; Select a note from the list of passed notes (vulpea-select-from "Grape" ;; this function returns only notes that are tagged as 'wine' and ;; 'grape' at the same time (see `vulpea-db 'documentation for more ;; information on this function). (vulpea-db-query-by-tags-every '("wine" "grape")) :require-match t)

;; Select a note from all notes filtered by some predicate. (vulpea-select "Grape" :filter-fn ;; We just manually check that the note is tagged as 'wine' and ;; 'grape' at the same time. (lambda (note) (let ((tags (vulpea-note-tags note))) (and (seq-contains-p tags "wine") (seq-contains-p tags "grape")))))

+end_src

Both of these examples achieve the same goal. The only practical difference here is performance. Sometimes you either already have a list of notes that you want to select from (so there is no need to filter all the database just to select those notes you already have) or you have a way to fetch a list of notes in a much faster way than by filtering whole database. See =vulpea-db= for more information on performance.

**** Visual configuration :PROPERTIES: :ID: 555650a9-fbaf-4841-a93e-c8fc81d06047 :END:

Each note is formatted using two functions - =vulpea-select-describe-fn= and =vulpea-select-annotate-fn=. Both of them are called by =vulpea-select= interface with a note as argument and their result is concatenated. The only difference between them is purely aesthetical - description has normal face and annotation has =completions-annotations= face.

By default =vulpea-select-describe-fn= is defined as =vulpea-note-title=; and =vulpea-select-annotate-fn= returns aliases and tags if present. To illustrate how it works, let's use some fake notes.

+begin_src emacs-lisp

(make-vulpea-note :id (org-id-new) :path (expand-file-name "note1.org" org-roam-directory) :title "Note without aliases and without tags")

(make-vulpea-note :id (org-id-new) :path (expand-file-name "note2.org" org-roam-directory) :title "Note with single tag" :tags '("tag1"))

(make-vulpea-note :id (org-id-new) :path (expand-file-name "note3.org" org-roam-directory) :title "Note with multiple tags" :tags '("tag1" "tag2"))

(make-vulpea-note :id (org-id-new) :path (expand-file-name "subdir/aliases.org" org-roam-directory) :title "Main title" :aliases '("Alias 1" "Alias 2"))

(make-vulpea-note :id (org-id-new) :path (expand-file-name "subdir/aliases.org" org-roam-directory) :title "Alias 1" :primary-title "Main title" :aliases '("Alias 1" "Alias 2"))

(make-vulpea-note :id (org-id-new) :path (expand-file-name "subdir/aliases.org" org-roam-directory) :title "Alias 1" :primary-title "Main title" :aliases '("Alias 1" "Alias 2") :tags '("tag1" "tag2"))

+end_src

These notes are converted into the following lines:

+begin_example

"Note without aliases and without tags" "Note with single tag #tag1" "Note with multiple tags #tag1 #tag2" "Main title" "Alias 1 (Main title)" "Alias 1 (Main title) #tag1 #tag2"

+end_example

+begin_html

Default describe behaviour

+end_html

Of course, you can configure this behaviour. For example:

+begin_src emacs-lisp

;; relative path // title (setq vulpea-select-describe-fn (lambda (note) (concat (string-remove-prefix org-roam-directory (vulpea-note-path note)) " // " (vulpea-note-title note))))

;; display tags and ignore aliases (setq vulpea-select-annotate-fn (lambda (note) (let* ((tags-str (mapconcat (lambda (x) (concat "#" x)) (vulpea-note-tags note) " "))) (if (string-empty-p tags-str) "" (concat " " tags-str)))))

+end_src

This results in the following lines:

+begin_example

"note1.org // Note without aliases and without tags" "note2.org // Note with single tag #tag1" "note3.org // Note with multiple tags #tag1 #tag2" "subdir/aliases.org // Main title" "subdir/aliases.org // Alias 1" "subdir/aliases.org // Alias 1 #tag1 #tag2"

+end_example

+begin_html

Custom describe behaviour

+end_html

*** =vulpea-note= :PROPERTIES: :ID: c8f81d7b-84fc-4e06-a17f-03dce4bf8dcc :END:

This module contains =vulpea-note= definition, which is represented as a structure with the following slots/fields:

If =ID= is not present in the note structure, this note is treated as non-existent. For example, =vulpea-select= returns such a note, when =require-match= is =nil= and the user selects non-existent note.

Example of a note:

+begin_src emacs-lisp

(vulpea-db-get-by-id "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c")

s(vulpea-note :id "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c"

:path "/Users/d12frosted/Dropbox/vulpea/20200407160812-kitsune_book.org" :level 0 :title "Kitsune Book" :primary-title nil :aliases ("vulpea" "Kitsune no Hon") :tags ("personal") :links (("https" . "https://github.com/d12frosted/vulpea") ("https" . "https://github.com/d12frosted/environment") ("https" . "https://github.com/d12frosted/vino")) :properties (("CATEGORY" . "20200407160812-kitsune_book") ("ROAM_ALIASES" . "vulpea \"Kitsune no Hon\"") ("ID" . "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c") ("BLOCKED" . "") ("FILE" . "/Users/d12frosted/Dropbox/vulpea/20200407160812-kitsune_book.org") ("PRIORITY" . "B")) :meta (("link" "[[https://github.com/d12frosted/vulpea][vulpea]]") ("users" "[[https://github.com/d12frosted/environment][environment]]" "[[https://github.com/d12frosted/vino][vino]]") ("status" "stable")))

+end_src

**** Tags predicate :PROPERTIES: :ID: fad159dd-aff9-4ae5-8050-2b5a03f4d001 :END:

In some cases you want to check if a note is tagged somehow. Vulpea provides two shortcuts for this:

+begin_src emacs-lisp

(setq note (make-vulpea-note :tags '("tag-1" "tag-2" "tag-3")))

s(vulpea-note nil nil nil nil nil nil ("tag-1" "tag-2" "tag-3") nil nil nil)

(vulpea-note-tagged-all-p note "tag-2" "tag-3") t

(vulpea-note-tagged-all-p note "tag-2" "tag-3" "tag-4") nil

(vulpea-note-tagged-any-p note "tag-2" "tag-3") t

(vulpea-note-tagged-all-p note "tag-2" "tag-3" "tag-4") nil

(vulpea-note-tagged-all-p note "tag-4") nil

+end_src

**** Accessing meta :PROPERTIES: :ID: ebf96ea5-50f0-473d-be21-77526ee601b9 :END:

In most cases you should not directly access =vulpea-note-meta=, but instead you should use one of the helpers - =vulpea-note-meta-get= and =vulpea-note-meta-get-list=. The only difference between these two functions is how they treat repeating keys. The former returns only the first occurrence of the key, while the latter returns a list.

Let's take the following note as example:

+begin_src emacs-lisp

(vulpea-db-get-by-id "05907606-f836-45bf-bd36-a8444308eddd")

s(vulpea-note :id "05907606-f836-45bf-bd36-a8444308eddd"

:path "..." ... :meta (("name" "some name") ("tags" "tag 1") ("tags" "tag 2") ("tags" "tag 3") ("numbers" "12") ("numbers" "18") ("numbers" "24") ("singleton" "only value") ("symbol" "red") ("url" "[[https://en.wikipedia.org/wiki/Frappato][wikipedia.org]]") ("link" "[[id:444f94d7-61e0-4b7c-bb7e-100814c6b4bb][Note without META]]") ("references" "[[id:444f94d7-61e0-4b7c-bb7e-100814c6b4bb][Note without META]]") ("references" "[[id:5093fc4e-8c63-4e60-a1da-83fc7ecd5db7][Reference]]") ("answer" "42")))

+end_src

As you can see, keys and values are strings. But that's not always useful, that's why =vulpea-note-meta-get= and =vulpea-note-meta-get-list= support string parsing of some common 'types': string (default), number, link (path of the link - either ID of the linked note or raw link), note (queries note by id from db) and symbol.

+begin_src emacs-lisp

(vulpea-note-meta-get note "name") "some name"

(vulpea-note-meta-get note "name" 'string) "some name"

(vulpea-note-meta-get-list note "name") ("some name")

(vulpea-note-meta-get note "tags") "tag 1"

(vulpea-note-meta-get-list note "tags") ("tag 1" "tag 2" "tag 3")

(vulpea-note-meta-get note "numbers" 'number) 12

(vulpea-note-meta-get-list note "numbers" 'number) (12 18 24)

(vulpea-note-meta-get note "symbol") "red"

(vulpea-note-meta-get note "symbol" 'symbol) red

(vulpea-note-meta-get note "url" 'link) "https://en.wikipedia.org/wiki/Frappato"

(vulpea-note-meta-get note "link" 'link) "444f94d7-61e0-4b7c-bb7e-100814c6b4bb"

(vulpea-note-meta-get-list note "references" 'note) (#s(vulpea-note :id "444f94d7-61e0-4b7c-bb7e-100814c6b4bb" :path "..." :title "Note without META" ...)

s(vulpea-note :id "5093fc4e-8c63-4e60-a1da-83fc7ecd5db7"

:path "..." :title "Reference" ...))

+end_src

*** =vulpea-db= :PROPERTIES: :ID: fe123255-686a-4c71-91cc-30e2e68387b4 :END:

This module contains functions to query notes from data base. In order for most of these functions to operate, one needs to enable =vulpea-db-autosync-mode= (see [[id:b946c716-e3b3-4c84-8229-dde59ddd55ae][Install]] section), for example, by using =vulpea-db-autosync-enable=. This hooks into =org-roam.db= by adding two extra tables:

Important! You might need to perform a full re-sync of =org-roam.db=.

**** =vulpea-db-get-by-id= :PROPERTIES: :ID: bc276c0e-1128-40c5-ad0a-4d2558d2ed20 :END:

The simplest function to get a note with some =ID=. Supports both file-level notes and outlines/headings. Returns =vulpea-note= if note with =ID= exists and nil otherwise.

+begin_src emacs-lisp

(vulpea-db-get-by-id "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c")

s(vulpea-note :id "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c"

:path "/Users/d12frosted/Dropbox/vulpea/20200407160812-kitsune_book.org" :level 0 :title "Kitsune Book" :primary-title nil :aliases ("vulpea" "Kitsune no Hon") :tags ("personal") :links (("https" . "https://github.com/d12frosted/vulpea") ("https" . "https://github.com/d12frosted/environment") ("https" . "https://github.com/d12frosted/vino")) :properties (("CATEGORY" . "20200407160812-kitsune_book") ("ROAM_ALIASES" . "vulpea \"Kitsune no Hon\"") ("ID" . "7705e5e4-bcd4-4e16-9ba7-fda8acdefe8c") ("BLOCKED" . "") ("FILE" . "/Users/d12frosted/Dropbox/vulpea/20200407160812-kitsune_book.org") ("PRIORITY" . "B")) :meta (("link" "[[https://github.com/d12frosted/vulpea][vulpea]]") ("users" "[[https://github.com/d12frosted/environment][environment]]" "[[https://github.com/d12frosted/vino][vino]]") ("status" "stable")))

(vulpea-db-get-by-id "xyz") nil

+end_src

**** =vulpea-db-query= :PROPERTIES: :ID: 2188a950-26ef-4f04-9e1b-e1dcd0de9ebb :END:

Function to query notes from database with optional predicate. This function is very powerful as it allows to apply Emacs Lisp predicate on /every/ =vulpea-note= in your database. This might be not very efficient on big set of notes, in such cases use specialized query functions.

When predicate is not passed, =vulpea-db-query= returns ALL notes from your database.

+begin_src emacs-lisp

(seq-length (vulpea-db-query)) 9554

+end_src

Since =vulpea-note= contains so much information, you can do many complex things, with =vulpea-db-query=.

+begin_src emacs-lisp

(vulpea-db-query (lambda (note) (and (seq-contains-p (vulpea-note-links note) (cons "id" "8f62b3bd-2a36-4227-a0d3-4107cd8dac19")) (or (seq-contains-p (vulpea-note-tags note) "grape") (seq-contains-p (vulpea-note-tags note) "cellar")))))

15 notes

+end_src

***** Custom SQL :PROPERTIES: :ID: 60480c99-5d52-4aee-95ff-e625b98b1a77 :END:

As you can see, =vulpea-db-query= doesn't allow to pass any custom SQL for filtering or whatnot. For future-proof code you should avoid querying stuff manually from database, but in case you need to, just use =org-roam-db-query=:

+begin_src emacs-lisp

(org-roam-db-query [:select title :from notes :limit 1]) (("Arianna Occhipinti"))

+end_src

**** Specialized queries :PROPERTIES: :ID: 8ef13d04-9f66-4cda-a03f-92cc44557ccc :END:

**** Other functions :PROPERTIES: :ID: 6ccde3bb-010a-439c-a7b3-c5188f1f0d91 :END:

**** Extending database :PROPERTIES: :ID: 8d0b72a8-b5ee-4be8-a17f-84b151ad85fc :END:

You may extend database by adding custom tables using =vulpea-db-define-table=:

+begin_src emacs-lisp

(vulpea-db-define-table ;; name 'my-custom-table ;; version 1 ;; schema '([(note-id :unique :primary-key) (some-column :not-null) (some-other-column)] ;; useful to automatically cleanup your table whenever a note/node/file is removed (:foreign-key [note-id] :references nodes [id] :on-delete :cascade)) ;; index '((custom-node-id-index [note-id])))

+end_src

Consult with [[https://github.com/magit/emacsql/][magit/emacsql]] documentation to learn more about schema and indices.

In order to populate your table with data, you should add a hook to =vulpea-db-insert-note-functions=. It is called with a single argument of type =vulpea-note= (keep in mind that =vulpea-note-links= slot is empty, open a ticket if you need it).

*** =vulpea-meta= :PROPERTIES: :ID: 9bb0311f-c257-46f1-8e1f-68c735a1a07c :END:

This module contains functions for manipulating note [[id:e0f6439c-8818-471d-ac25-c9dda830df3a][metadata]] represented by the first description list in the note, e.g. list like:

+begin_src org-mode

Functions of interest:

*** =vulpea-buffer= :PROPERTIES: :ID: 6f01bc38-414d-455f-99ad-c8ae73476a49 :END:

This module contains functions for prop and meta manipulations in current buffer.

**** Buffer properties :PROPERTIES: :ID: 0af66e12-5653-4e1e-8cef-e583db6c0f1c :END:

Buffer properties are key-values defined as =#+KEY: VALUE= in the header of buffer.

**** Metadata :PROPERTIES: :ID: e0f6439c-8818-471d-ac25-c9dda830df3a :END:

Metadata is defined as the first description list in the buffer, e.g. list like:

+begin_src org-mode

***** Example 1 - getting values :PROPERTIES: :ID: ae533583-76fc-4c83-bcdd-9636fabef530 :END:

Consider the following Org Mode file.

+begin_src org

:PROPERTIES: :ID: 05907606-f836-45bf-bd36-a8444308eddd :END: ,#+title: Note with META

In order to get anything from meta, first you need to parse it:

+begin_src emacs-lisp

(vulpea-buffer-meta) (:file "/path-to/with-meta.org" :buffer (org-data ...))

+end_src

And then you can retrieve values from parse meta:

+begin_src emacs-lisp

(setq test-meta (vulpea-buffer-meta))

(vulpea-buffer-meta-get! test-meta "name") "some name"

(vulpea-buffer-meta-get! test-meta "tags") "tag 1"

(vulpea-buffer-meta-get-list! test-meta "tags") ("tag 1" "tag 2" "tag 3")

(vulpea-buffer-meta-get-list! test-meta "numbers" 'number) (12 18 24)

(vulpea-buffer-meta-get! test-meta "symbol" 'symbol) red

(vulpea-buffer-meta-get! test-meta "url" 'link) "https://en.wikipedia.org/wiki/Frappato"

(vulpea-buffer-meta-get! test-meta "link" 'link) "444f94d7-61e0-4b7c-bb7e-100814c6b4bb"

(vulpea-buffer-meta-get-list! test-meta "references" 'note) (#s(vulpea-note :id "444f94d7-61e0-4b7c-bb7e-100814c6b4bb" ...)

s(vulpea-note :id "5093fc4e-8c63-4e60-a1da-83fc7ecd5db7"

...))

+end_src

***** Example 2 - setting values :PROPERTIES: :ID: f0677558-4c51-4874-b13d-1685da09d06b :END:

Consider the following Org Mode file.

+begin_src org

:PROPERTIES: :ID: 05907606-f836-45bf-bd36-a8444308eddd :END: ,#+title: Note with META

Imagine that we evaluated the following code in this buffer.

+begin_src emacs-lisp

;; put a value in the beginning of the list (vulpea-buffer-meta-set "date" "[2021-12-05]")

;; replace existing name value (vulpea-buffer-meta-set "name" "new name")

;;replace list of references with new one (vulpea-buffer-meta-set "references" (list (vulpea-db-get-by-id "8f62b3bd-2a36-4227-a0d3-4107cd8dac19")))

;; append to the end of list (vulpea-buffer-meta-set "years" '(1993 1994) 'append)

;; remove numbers key (vulpea-buffer-meta-remove "numbers")

+end_src

The resulting buffer will look like this:

+begin_src org

:PROPERTIES: :ID: 05907606-f836-45bf-bd36-a8444308eddd :END: ,#+title: Note with META

*** =vulpea-utils= :PROPERTIES: :ID: b904b2fd-3ae2-4cad-9ed5-d0c196d9cffa :END:

This module contains various utilities used by other modules. Functions of interest:

** Performance :PROPERTIES: :ID: 5b44d873-179a-4fcb-88df-ff8a8d328bd0 :END:

*** Query from database :PROPERTIES: :ID: b5069fa7-28ea-4bc1-bfce-32710d4cabc9 :END:

This library provides multiple functions to query notes from the database. Basically, there is one powerful =vulpea-db-query= allowing to filter notes by any =vulpea-note= based predicate. The only downside of this power is performance and memory penalty as all notes are loaded into memory. In cases when performance is critical and the set of notes can be narrowed down, one can use specialized queries:

The following table displays time required to query notes by using =vulpea-db-query= vs specialized query on the database of 9554 [[https://github.com/d12frosted/vulpea-test-notes/][generated notes]]. The difference between various test cases is partially explained by the fact that filtering functions result in different amount of notes. Since we need to retrieve full note structure, the more notes we have, the more time it takes.

| test | result size | generic | specialized | ratio | |---------------+-------------+--------------------+--------------------+-----------| | =tags-some= | 30 notes | 1.0112478712 | 0.0066033426 | 153.14182 | | =tags-every= | 3168 notes | 1.0059819176 | 0.5709392964999999 | 1.7619770 | | =links-some= | 1657 notes | 1.0462236128999999 | 0.4248580532 | 2.4625251 | | =links-every= | 92 notes | 1.0204833089 | 0.0545313596 | 18.713696 |

+TBLFM: $5=$3/$4

See [[https://github.com/d12frosted/vulpea/discussions/106#discussioncomment-1601429][this comment]] for more background on why these functions where created.

In order to make these functions as fast as possible, =vulpea-db= module builds and maintains a view table called =notes=. While it does drastically improve query performance (see the table below), it adds a small footprint on synchronisation time. See [[https://github.com/d12frosted/vulpea/pull/116][vulpea#116]] for more information on this feature and measurements.

| test | result size | [[https://github.com/d12frosted/vulpea/blob/551495a59fb8c3bcd49a091b233e24e4cb8b584c/vulpea-db.el#L76-L187][regular]] | view table | ratio | |---------------+-------------+--------------------+--------------------+-----------| | =tags-some= | 30 notes | 4.6693460650999995 | 1.0112478712 | 4.6174100 | | =tags-every= | 3168 notes | 4.7333844436999996 | 1.0059819176 | 4.7052381 | | =links-some= | 1657 notes | 4.8095771283 | 1.0462236128999999 | 4.5970833 | | =links-every= | 92 notes | 4.5517473337999995 | 1.0204833089 | 4.4603839 |

+TBLFM: $5=$3/$4

** Coding :PROPERTIES: :ID: 74fe6b58-e289-4c8d-ad0b-49203227c905 :END:

Vulpea is developed using [[https://github.com/doublep/eldev/][eldev]]. If you are using =flycheck=, it is advised to also use [[https://github.com/flycheck/flycheck-eldev][flycheck-eldev]], as it makes dependencies and project files available thus mitigating false negative results from default Emacs Lisp checker.

** Building and testing :PROPERTIES: :ID: 7a68aea9-315a-4415-a619-0c088772b3f4 :END:

Vulpea tests are written using [[https://github.com/jorgenschaefer/emacs-buttercup/][buttercup]] testing framework. And [[https://github.com/doublep/eldev/][eldev]] is used to run them both locally and on CI. In order to run the tests locally, first [[https://github.com/doublep/eldev/id:b946c716-e3b3-4c84-8229-dde59ddd55aeation][install]] =eldev= and then run:

+begin_src bash

$ make test

+end_src

Please note, that the linter is used in this project, so you might want to run it as well:

+begin_src bash

$ make lint

+end_src

** Acknowledgements :PROPERTIES: :ID: 4470139c-2d98-41c1-9240-91bb62870d33 :END:

[[https://barberry.io/images/vulpea-logo.png][Logo]] was created by [[https://www.behance.net/irynarutylo][Iryna Rutylo]].