+TITLE: Org Element Cache

• Introduction The [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] allows parsing & manipulating org-mode files in the form of lisp objects.

A cache for these objects is already included in org-mode and disabled by default.

However, when working with large numbers (> 1000) of files, populating and processing this cache for each file takes a long time.

This package tries to alleviate this problem by:

1. Allowing users to register hooks that compute some data form an org element
2. Persisting the results on disk
3. Taking care of keeping the cached data in sync with the actual buffer contents

In the current version, only hooks on a file-level are implemented. Future versions might include a way to register hooks for element types.

• Structure of a Cache A cache has four components:

1. A hash table with a plist for each file managed by the cache
2. A list of folders used to find files managed by the cache
3. A list of hooks to generate cached data
4. A file to persist the cache in

** Populating the Cache A cache is populated with entries by opening each file in a temporary buffer, parsing it to an ~org-element~ and passing this element to each of the hooks.

Doing this for the first time can take a few seconds, depending on the number of files in the cache.

Once the cache has been saved to disk for the first time, a combination of the ~find~ and ~sha1sum~ shell commands is used to determine which entries need to be updated.

Assuming you're only using one emacs instance at a time, updating the cache on startup should take only a few milliseconds.

• Usage ** Defining Caches The ~def-org-el-cache~ macro can be used to define a new cache.

+begin_src emacs-lisp

(def-org-el-cache my-cache ;; name of the cache / variable to store it in (list "~/org") ;; directories managed by this cache "~/org/.cache.el" ;; file to persist the cache in )

+end_src

** Adding Hooks Hooks can be added to a cache with the ~org-el-cache-add-hook~ function.

+begin_src emacs-lisp

(org-el-cache-add-hook my-cache ;; Cache to add the hook to :my-property ;; Property name to use for this hook (lambda (filename element) ;; Do something with the element and return some value ))

+end_src

** Updating Caches

• ~(org-el-cache-update cache)~ updates all outdated entries in the cache
• ~(org-el-cache-force-update cache)~ re-initializes the cache

If you changed the definition of a hook or added a new one, use ~org-el-cache-force-update~ to re-initialize the cache. ** Accessing Cached Data

• ~(org-el-cache-get cache filename)~ returns the cache entry for a file
• ~(org-el-cache-file-property cache filename prop)~ returns the value of a files property

The following functions work on all entries of a cache. For more information on them, refer to the functions documentation (e.g. ~C-h f org-el-cache-map~).

• ~org-el-cache-reduce~
• ~org-el-cache-map~
• ~org-el-cache-select~
• ~org-el-cache-each~ ** Persisting Caches All caches are saved to disk at regular intervals (configurable via ~org-el-cache-presist-interval~, defaults to 10 minutes).

Usually, there is no need to manually persist or load a cache. If you want to do so anyway, you can use the following functions:

• ~(org-el-cache-persist-all)~ persists all caches
• ~(org-el-cache-persist cache)~ persists a single cache
• ~(org-el-cache-load cache)~ loads a cache from disk ** Example [[file:examples/file-selector.el]] contains the code for a file-selector (similar to Emacs' find-file) that uses the titles of files instead of their paths.

For more information, you can check out the integration tests in [[file:org-el-cache-test.el]].

• Prior / Similar Art
• [[https://github.com/ndwarshuis/org-sql][ndwarshuis/org-sql]]
• [[https://github.com/sigma/pcache][sigma/pcache]]
• Limitations Cached data is limited to objects that can be printed / read (serialized / deserialized).

This means that there is no (elegant) way to cache markers in files, e.g. when using cached headline data for org-agenda views.

A possible workaround would be attaching IDs to every headline, then using this ID (instead of a marker) to jump to a headline e.g. to change its TODO state.

• Dependencies org-el-cache uses a number of shell commands to find files that need to be updated.

1. ~find~
2. ~xargs~
3. ~sha1sum~

These should be installed by default on most Linux / Unix distros.

• Hashing The hash of a file or buffer is used to determine if it has changed since it was last processed by the cache.

When updating a single file, ~(buffer-hash)~ is fast enough.

To speed up recursively searching directories for =.org= files and calculating their hashes, the ~find~ and ~sha1sum~ shell commands are used instead of ~directory-files-recursively~ and ~buffer-hash~.

• Benchmarks Working on a collection of ~1500 files with ~200k lines in total, initializing the cache takes around ~36s on my machine (Thinkpad L470, SSD).

Updating the cache once it has been initialized / loaded from disk takes around 200ms.

• Relation to ~org-element-cache~ Org-mode already includes a cache for parsed elements. This is disabled by default since there seem to be problems with the implementation that cause Emacs to hang after making changes to org files.

In the long term, it would be nice to reuse as much of the existing code as possible and figure out where the bugs are.

• Changelog ** [2020-02-08 Sat 12:50] *** Internal
• Use ~secure-hash~ instead of ~buffer-hash~
• Use ~defun~ instead of ~defmethod~, the function help looks nicer this way and we don't need overloaded methods
• License Copyright © Leon Rische and contributors. Distributed under the GNU General Public License, Version 3