Alternative identifier formats? - Note signatures

[sthesing]

First off, thanks for denote! I love it!

Is there a way for the user to define an alternative format for the identifier instead of a date? My skill in reading emacs-lisp are limited, but if I see it correctly, it's more complicated than setting denote-id-format and denote-id-regexp, right?

Concretely, I'd like to be able to use a Luhmann-style identifier format:

1--first-note__foo_bar.org
1a--a-branch__foo.org
1b--more-notes.org
1b1--a-sub-branch__baz.org
1c--yet-more-notes.org
2--second-note__bar_baz.org

Is that possible?

[protesilaos]

Hello @sthesing!

It would require changes to denote.el for this to happen. It is possible and I have been thinking about making this an option. The common use-case is to have dates as 2022-12-09 and then the time component as 16:30:00.

With regard to the Luhmann-style format, I am okay with it, though I wonder how we will derive it. Say I have a note that is 1a--title__keyword.org. How do I add the 1b or the 2?

EDIT: add missing underscore.

[MirkoHernandez]

Hello @protesilaos

Just like @sthesing I'm trying to create a proper zettelkasten workflow on top of denote.

Currently I just prepend the zettelkasten Id to the title in the file name and not the front-matter. This works as expected (the links display only the title) but I'm not sure if this is the best approach to integrate the zettel id with denote.

The missing piece for me is how to sort the files in dired so that it reflects the zettelkasten structure. A simple regex should do but I did not find a straightforward way to do this in dired. With that in place I think it should be trivial to create useful zettelkasten commands (go to parent, go to next, create zettel in same level, create subentry, etc).

Thank you for your work, I have been using denote for a few months and I think it is great.

[sthesing]

Hi @protesilaos ,

thanks for the quick reply.

With regard to the Luhmann-style format, I am okay with it, though I wonder how we will derive it. Say I have a note that is 1a--title__keyword.org.

I'm not quite sure I understand what you mean be "how we will derive it" (English isn't my native language). Do you mean how the system works in general?

I assume so...

How do I add the 1b or the 2?

At the end of the day, it's arbitrary, but it designates a relationship. Luhmann used paper (in the format of index cards), so when he wrote on topic Foo, he'd maybe fill three pieces of paper. Then he'd continue with topic Bar. But when later he'd continue (or branch of) topic Foo, he'd insert new pieces of paper by switching to alphabetical notation and so on...

1 Topic Foo 2 Further Foo 2a Something interesting about Foo 3 Yet more Foo 4 Topic Bar

So you see two notes with very similar ids like 3b2a7a and 3b2a7c are related in many ways, but whether they are continuations of each other is contingent.

(Just to be precise, actually Luhmann prepended each id with a number that designated a broad topic category and a comma. So the actual zettel signatures looked like "1,3b2a7a".)

Was that your question? If not, I'm sorry for writing a lot of unnecessary stuff ;-)

[nobiot]

Hi all,

Denote is flexible and it's very easy to adapt it to your own needs. Just for your information, I have done a quick proof of concept for the "Luhmann-style identifier format" example that @sthesing mentions above.

Figure 1. The buffers are, from the top

• note ID 2 linking to note ID 1b
• note ID 1b
• backlink buffer for note ID 1b, which correctly shows note ID 2; note that the denote's fontification to color different parts of the file name is not adjusted to work

---

Figure 2. The buffers are, from the top:

• Dired showing denote-directory
• Minibuffer when you call denote-open-or-create; note the last file starting with "test--test" is not shown because it does not follow the ID scheme

One thing you should notice is that the sorting in Dired is not what you expect. I use Ubuntu for this. I think you would need to play with the naming scheme to achieve the sorting you want.

---

You would need to override a variable (constant) and a function. There are some other things that do not work; e.g. the "identifier" property in the note still uses the standard date format. These need to be adjusted if you wish to go down this path. But the code change is rather easy and small.

The minimal change I did for this proof of concept is as follows:

(setq denote-id-regexp "\\([[:digit:]]+?[[:alpha:]]*[[:digit:]]*\\)")
;; Note this regexp is the key. It will only work for
;; <number><alphabet><number>. I'm sure you can go on further, or
;; different variants; e.g. I have seen something like "57/12a1".  You
;; would need to adjust the regexp according to your ID scheme.

(defun denote-format-file-name (path id keywords title-slug extension)
  "Format file name.
PATH, ID, KEYWORDS, TITLE-SLUG are expected to be supplied by
`denote' or equivalent: they will all be converted into a single
string.  EXTENSION is the file type extension, as a string."
  (let ((kws (denote--keywords-combine keywords))
        (file-name (concat path (read-string "ID:")))) ;; << Just change this line.
        ; This will ignore the ID denote automatically generates based
        ; on the date and time and will let Denote prompt for
        ; you to type the ID
    (when (and title-slug (not (string-empty-p title-slug)))
      (setq file-name (concat file-name "--" title-slug)))
    (when (and keywords (not (string-blank-p kws)))
      (setq file-name (concat file-name "__" kws)))
    (concat file-name extension)))

[nobiot]

You will need to change more variables and (potentially) functions if you go down the path. For what it's worth, here is my file to personalize denote for my ID scheme.

This is my file naming scheme. I use markdown-yaml.

  ;; yyyy-mm-ddThhmmss_C_title.ext. "C" is a single ascii character that
  ;; represents the keyword. I will not have multiple keywords and it
  ;; represents one of the few categories. The categories I use are:
  ;; (C)reate, (R)eference, and (I)ndex. 

[protesilaos]

@MirkoHernandez

Currently I just prepend the zettelkasten Id to the title in the file name and not the front-matter. This works as expected (the links display only the title) but I'm not sure if this is the best approach to integrate the zettel id with denote.

My wish is to make this formal in a way that users of all skill levels can accomplish. What you do sounds fine in terms of how it works for you, though we can probably make refinements to it once/if we refactor Denote to have a more flexible identifier scheme.

The missing piece for me is how to sort the files in dired so that it reflects the zettelkasten structure. A simple regex should do but I did not find a way a straightforward way to do this in dired. With that in place I think it should be trivial to create useful zettelkasten commands (go to parent, go to next, create zettel in same level, create subentry, etc).

Have you tried to edit the ls flags that Dired uses? Check this and the link it has to dired-virtual-mode: https://protesilaos.com/emacs/denote#h:a7fd5e0a-78f7-434e-aa2e-e150479c16e2. Maybe this is helpful.

Thank you for your work by the way, I have been using denote for a few months and I think it is great.

You are welcome!

---

@sthesing

I'm not quite sure I understand what you mean be "how we will derive it" (English isn't my native language). Do you mean how the system works in general

Sorry for the misunderstanding! I was thinking in terms of how Denote should generate the file name. The Luhmann scheme is sequential, so the first note gets the number 1, the second is 2, and so on. I guess Denote would then have to keep a record of which number it last used and increment it each time. But what happens if I create a third note, which gets the number 3 and then decide to delete that file. Should the fourth note be numbered 4 or 3? These are the sort of technicalities that we avoid with Denote's date+time identifier.

Then we have the relational identifiers of the 1a sort. I guess we would have to have some way of letting the user tell Denote when to use those. For example, a new command that picks an existing file and derives a relational identifier, so if the file is 1a then it would come up with 1b. Though as you showed with the "1,3b2a7a" example, this is not straightforward.

While this is not what you are asking for and I am still keen on making Denote more flexible in this regard, how about using keywords and keeping the Denote identifier as-is? So the file name would be like:

ID--TITLE__KEYWORDS.EXTENSION

The user can define whatever keyword they want and then they can easily search for, say, all notes in the 1a branch by targeting _1a. I know this is not Luhmann-style, though maybe it is worth experimenting with.

---

@nobiot

Denote is flexible and it's very easy to adapt it to your own needs. Just for your information, I have done a quick proof of concept for the "Luhmann-style identifier format" example [...]

I like this idea and, as I noted, I am happy to have a user option for this. Put differently, the "Denote way" IS NOT the "one true way" of naming notes.

The obvious use-case is to add hyphens and colons to the date and time components of the identifier. Though we can build on that.

This is not easy though, as there are lots of tricky cases we will probably have to consider. We must also be mindful of potential extensions to Denote, whose developers will probably be expecting things to be a certain way.

As an aside, I will revisit denote.el one of these days in order to prepare the release of the new version. There is a lot of valuable work there and we need to formalise it.

[nobiot]

This is not easy though, as there are lots of tricky cases we will probably have to consider. We must also be mindful of potential extensions to Denote, whose developers will probably be expecting things to be a certain way.

I agree. As this point is not directly relevant for the issue here, perhaps we could take it to the mailing list.

[sthesing]

@nobiot

I have done a quick proof of concept for the "Luhmann-style identifier format" example that @sthesing mentions above. Cool, thanks for that! I tried it out and at first it worked, but now, none of the methods of note creating prompt me for an ID anymore. Maybe I'm doing something wrong or I broke something without realising it. I'll experiment further and try to reproduce it.

---

@protesilaos

Sorry for the misunderstanding! I was thinking in terms of how Denote should generate the file name.

Ah, I see. Sorry. As far as my question was going, I didn't expect Denote to generate the identifier automatically, but prompt me for it, similar to denote-create-using-date.

While this is not what you are asking for and I am still keen on making Denote more flexible in this regard, how about using keywords and keeping the Denote identifier as-is? [...] The user can define whatever keyword they want and then they can easily search for, say, all notes in the 1a branch by targeting _1a. I know this is not Luhmann-style, though maybe it is worth experimenting with.

Oh my goodness, I didn't think of that. That's brilliant! I'm really flabbergasted of how much functionality can be achieved by cleverly naming the files... The only concern I would have is that these keywords (let's call'em: "signature keywords") like 1a4b etc. would clatter the list of candidates in the keywords prompt, when denote-infer-keywords is set to non-nil. But that could be solved by e.g. a user option to filter out keywords with a regular expression?

Thanks!

[MirkoHernandez]

Hi guys.

I think there is value in preserving the default denote identifier and find a way to implement the zettelkasten id on top of it. This would allow the following workflows.

• Easily rename previously created notes so that they form part of a zettelkasten (without breaking any previously created links).
• Create a bunch of notes using denote in the default way and then add them to the zettelkasten in a two step process (this is how Luhmanns's "reading notes" worked I believe)

Besides I don't know how links would work without a proper identifier, how would you link notes with the same zettelkasten id?.

@protesilaos

But what happens if I create a third note, which gets the number 3 and then decide to delete that file.

This is not how the zettelkasten is supposed to work, notes are always added and not deleted.

The Luhmann scheme is sequential, so the first note gets the number 1, the second is 2, and so on. I guess Denote would then have to keep a record of which number it last used and increment it each time.

This how I'm currently doing this (It does not cover edge cases like an id that end in z or an id involving commas).

(defvar my-zettel-last-id "1"
  "Variable used to automate the creation of zettel notes")

(defun my/zettel-increment-string (s)
  "Increments the ascii value of the last character of S.
 Used for creating a zettelkasten id."
  (let ((string-without-last-char  (substring s 0 -1 ))
    (last-char  (substring s  -1 nil)))
    ;; Increment string that ends in 9.
    (if  (equal last-char "9")
    (cond ((equal string-without-last-char "")
           "10")
          ((string-match "[0-9]" (substring string-without-last-char -1 nil))
           (concat (my/zettel-increment-string string-without-last-char) "0" ))
         (t (concat  string-without-last-char  "10")))
    (concat string-without-last-char (char-to-string (1+ (string-to-char last-char)))))))

Have you tried to edit the ls flags that Dired uses? Check this and the link it has to dired-virtual-mode

Thanks! this seems promising I will try it out later today.

[protesilaos]

Changing the format of the identifier is an idea worth considering, though it is not something to be decided easily. It is an integral part of the system and we need to think about it carefully.

For less drastic measures, I gather from @sthesing the idea of prompting for a Luhmann ID (let's call it LUHMANN for now) and from @MirkoHernandez I learn more about the manual intervention of this workflow. These together tell us that Denote does not really need to have any complex logic in place and that it could simply add an extra field to the file name.

We currently have the user option denote-prompts, as well as convenience commands such as denote-create-note-using-date. We can build on this model to add an extra field to the file name.

The default file name is:

DATE--TITLE__KEYWORDS.EXTENSION

Where do we add a potential new LUHMANN field? And what delimiter do we use? Influenced by the way @nobiot arranges file names, I am thinking of this:

DATE==LUHMANN--TITLE__KEYWORDS.EXTENSION

Notice the = as a field delimiter. From what I can tell, it is a legal character for Unix and Windows.

This is just me thinking out loud. Ideas are welcome!

EDIT: fix symbol of denote-create-note-using-date.

[sthesing]

Where do we add a potential new LUHMANN field? And what delimiter do we use? Influenced by the way @nobiot arranges file names, I am thinking of this:

DATE==LUHMANN--TITLE__KEYWORDS.EXTENSION

This would be perfect for my use case! And if it's a user-defined signature string, then many use cases could be covered, without a need for Denote to compute complex logic under the hood.

Just for the record, I experimented with @protesilaos' idea above, to input the signatures as keywords. I tinkered a bit with the denote-keywords function. The regexp still needs work and it breaks if denote-filtered-keywords-regexp is nil. But I could already work with this quick 'n dirty hack.

(setq denote-filtered-keywords-regexp "^[0-9]\\{1\\}$\\|[0-9][a-z]")

(defun denote-keywords ()
    "Return appropriate list of keyword candidates.
  If `denote-infer-keywords' is non-nil, infer keywords from
  existing notes and combine them into a list with
  `denote-known-keywords'.  Else use only the latter."
    (delete-dups
     (if denote-infer-keywords
         (seq-filter (lambda (x) (not (string-match-p denote-filtered-keywords-regexp x )))
         (append (denote--inferred-keywords) denote-known-keywords))
       denote-known-keywords))) 

I'd much prefer the DATE==LUHMANN--TITLE__KEYWORDS.EXTENSION idea, though!

[protesilaos]

I forgot about a keywords' filter @sthesing! We can have that as well, the same way we do with denote-excluded-directories-regexp (part of current main but not officially released).

I think those two ideas can co-exist.

[protesilaos]

Maybe this diff works for the keywords' filter @sthesing.

 denote.el | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/denote.el b/denote.el
index ffe9ace..9558ab9 100644
--- a/denote.el
+++ b/denote.el
@@ -395,6 +395,15 @@ (defcustom denote-excluded-directories-regexp nil
   :package-version '(denote . "1.2.0")
   :type 'string)

+(defcustom denote-excluded-keywords-regexp nil
+  "Regular expression of keywors to not infer.
+Keywords are inferred from file names and provided at relevant
+prompts as completion candidates when the user option
+`denote-infer-keywords' is non-nil."
+  :group 'denote
+  :package-version '(denote . "1.2.0")
+  :type 'string)
+
 ;;;; Main variables

 ;; For character classes, evaluate: (info "(elisp) Char Classes")
@@ -765,8 +774,10 @@ (defun denote--inferred-keywords ()
   "Extract keywords from `denote-directory-files'.
 This function returns duplicates.  The `denote-keywords' is the
 one that doesn't."
-  (mapcan #'denote-extract-keywords-from-path
-          (denote-directory-files)))
+  (let ((kw (mapcan #'denote-extract-keywords-from-path (denote-directory-files))))
+    (if-let ((regexp denote-excluded-keywords-regexp))
+        (seq-filter (lambda (k) (not (string-match-p regexp k))) kw)
+      kw)))

 (defun denote-keywords ()
   "Return appropriate list of keyword candidates.

The code, if you prefer it intact:

(defcustom denote-excluded-keywords-regexp nil
  "Regular expression of keywors to not infer.
Keywords are inferred from file names and provided at relevant
prompts as completion candidates when the user option
`denote-infer-keywords' is non-nil."
  :group 'denote
  :package-version '(denote . "1.2.0")
  :type 'string)

(defun denote--inferred-keywords ()
  "Extract keywords from `denote-directory-files'.
This function returns duplicates.  The `denote-keywords' is the
one that doesn't."
  (let ((kw (mapcan #'denote-extract-keywords-from-path (denote-directory-files))))
    (if-let ((regexp denote-excluded-keywords-regexp))
        (seq-filter (lambda (k) (not (string-match-p regexp k))) kw)
      kw)))

EDIT: Just to clarify that I did not test this.

[sthesing]

EDIT: Just to clarify that I did not test this.

I did. It works fine with the development version of denote. Thanks a bunch, @protesilaos!

If you find the time to implement your other idea (DATE==LUHMANN--TITLE__KEYWORDS.EXTENSION), I'd be thrilled and happy to test that, too.

[protesilaos]

I did. It works fine with the development version of denote. Thanks a bunch, @protesilaos!

Very well! I made some minor tweaks and added it.

If you find the time to implement your other idea (DATE==LUHMANN--TITLE__KEYWORDS.EXTENSION), I'd be thrilled and happy to test that, too.

This can be done as well and I think it is a good addition for those who need it. Two points:

1. Let's also read what the others may have to write about it.
2. I prefer to delay this a few days until I prepare version 1.2.0. We have a lot of useful additions and I want to formalise them before adding more major features. Let's say that we do this next weekend.

[sthesing]

Hi, I've been manually using the DATE==SIGNATURE--TITLE__KEYWORDS.EXTENSIONS scheme for a few days now and played around with the dired fontification. I got it to work, but since I'm not very skilled with emacs lisp, I'm not sure I did something against conventions or the logic of your code.

(defvar denote-faces--file-name-regexp
  (concat "\\(?1:[0-9]\\{8\\}\\)\\(?2:T[0-9]\\{6\\}\\)"
          "\\(?:\\(?3:==\\)\\(?4:[[:alnum:][:nonascii:]=]*\\)\\)?"
          "\\(?:\\(?5:--\\)\\(?6:[[:alnum:][:nonascii:]-]*\\)\\)?"
          "\\(?:\\(?7:__\\)\\(?8:[[:alnum:][:nonascii:]_-]*\\)\\)?"
          "\\(?9:\\..*\\)?$")
  "Regexp of file names for fontification.")

(defconst denote-faces-file-name-keywords
  `((,(concat " " denote-faces--file-name-regexp)
     (1 'denote-faces-date)
     (2 'denote-faces-time)
     (3 'denote-faces-delimiter nil t)
     (4 'denote-faces-signature nil t)
     (5 'denote-faces-delimiter nil t)
     (6 'denote-faces-title nil t)
     (7 'denote-faces-delimiter nil t)
     (8 'denote-faces-keywords nil t)
     (9 'denote-faces-extension nil t )))
  "Keywords for fontification of file names.")

And of course I did the same for denote-faces-file-name-keywords-for-backlinks

The face itself I set to inherit tooltip, but I also thought about font-lock-warning-face. What do you think?

(defface denote-faces-signature '((t :inherit tooltip))
  "Face for file name signature in Dired buffers."
  :group 'denote-faces
  :package-version '(denote . "1.3.0"))

Is that the way to adapt the dired fontification? If so, I could prep a pull request, if you like.

[protesilaos]

@sthesing This is fine. About the face, I think it is better to pick something from the 'font-lock' family or, generally, a face that normally does not have a background. It will look consistent with the other faces.

Regarding the prospect of a PR, we need to consider it further. To support this change, we must make changes to several functions. It still is not clear that there is demand for this addition. Even though I personally am fine with it in principle, I prefer that we wait before introducing such a major feature. Doing it prematurely will make it difficult for us to change course afterwards.

What do you think?

[sthesing]

Hi @protesilaos ,

regarding the face: you're right. I already switched to font-lock-warning-face. A face with a background doesn't stand out well when searching for signature strings in dired.

regarding the waiting: no need to hurry, as far as I'm concerned. With the fontification, this is already a setup I can work with. The only downside is that when I use denote-rename-file-using-front-matter, I have to manually re-insert my signature string into the file name. But I seldomly adjust the keywords after the creation of a note, so that's not much of an inconvenience. (Who knows, maybe I take this as a opportunity to learn some emacs lisp and try to figure out how denote will leave the signature strings alone and even prompt me for them during note creation. Might be a fun project to study a new language.)

So long story short: No need to hurry. I'm all for taking the time to think things through.

[MirkoHernandez]

To support this change, we must make changes to several functions. It still is not clear that there is demand for this addition. Even though I personally am fine with it in principle, I prefer that we wait before introducing such a major feature.

Hi @protesilaos, when I was trying to implement a zettelkasten in denote I considered 3 approaches, using the filename id, using keywords and using the front-matter. The keywords approach didn't seem right semantically, and I didn't wanted to change the default id, using the front matter would have prevented me the use of dired (and it seem like it would take me some time to find a vertico related solution) so I ended up just annotating the title.

I think that using the filename for this use case is fine but a more universal and useful approach (I think there would be more demand for this kind of stuff) would be to facilitate the use of the front-matter (make it easier to create a custom note) . Each user could add metadata to the front-matter and then grep the first 10 lines or so of each denote file and use this information in find-dired, vertico, marginalia, etc for whatever purpose needed. Integrating the front-matter with other emacs tools seems like it could solve these kinds of issues (Luhmann id) and other related ones.

Also thank you for your dired-virtual-mode suggestion, it works!.

[sthesing]

@MirkoHernandez

Just out of curiosity:

I was trying to implement a zettelkasten in denote

What exactly are you adding on top of what denote already does? I ask because a lot of people say that all you need to have a Zettelkasten is a hypertext system with granular/atomic notes and keywords, preferrably with a way to inspect backlinks. Denote does all that out of the box.

Am I right when I assume you want to establish some sort of sequentiality of notes? What Luhmann called "Folgezettel" – followup notes? (To clarify what I mean, I turned an old zettel of mine into a blogpost: Zettelkasten: Hypertext, Linearity, Sequentiality).

If my assumption is wrong, ignore the rest, but - as I said - I'm interested in what you're trying to achieve beyond denote's current features. If my assumption is right, then...

I considered 3 approaches, using the filename id, using keywords and using the front-matter.

• I agree with you that keywords aren't the right tool to establish sequentiality.
• changing the default ID is possible (as @nobiot demonstrated above), but as @protesilaos pointed out, a rather drastic measure.

Using the front matter is another way (in fact I used it in my command line tool "Zettels"), but it has a few drawbacks. One is that you (or grep) have to open the files to inspect the front matter. Depending on how many notes you have, that might become a scaling issue. Though to be fair, modern grep implementations like ripgrep are very fast. The other question is how to designate note sequences with the front matter. In Zettels, I had a followups field in my YAML-front matter. But I wasn't really satisfied with that. How would you approach it?

I think that using the filename for this use case is fine but a more universal and useful approach (I think there would be more demand for this kind of stuff) would be to facilitate the use of the front-matter

Using the filename gives you an edge in scalability, because you don't have to open the files. All the needed information is in the filename already. Also, denote accomplishes much of its functionality just by leveraging the mind-boggingly clever naming scheme. To use the filename fits that approach.

As I mentioned above, I tried Prot's idea of adapting the naming scheme with a "==SIGNATURE" component for some time:

As you can see, once I have identified the start of a sequence (e.g. by searching for keywords), I can easily identify the rest of the sequence by searching and/or narrowing in dired. And because it's already sorted by date (due to denote's identifier), the sequentiality of these related notes is directly apparent. To me, that's pretty universal and very useful.

[MirkoHernandez]

What exactly are you adding on top of what denote already does? I ask because a lot of people say that all you need to have a Zettelkasten is a hypertext system with granular/atomic notes and keywords, preferrably with a way to inspect backlinks. Denote does all that out of the box.

Hi @sthesing , yes this to me is completely wrong, that is why I did not even considered the other zettelkasten packages. I think the Luhmann id is invaluable . What I would like denote to do - just to enable the possibility to set the luhman id somewhere, in the filename or the front-matter. I think this is the original request.

With the id in place it could be possible to add the following functionality using elisp (just extending denote). I'm not asking @protesilaos to add this or denote to have this functionality, just stating what could be possible by making use of the Id.

• create a note with a Luhmann id created manually
• create a note with a Luhmann id created automatically (same level, sublevel, etc)
• display (in a beautiful vertico buffer) relevant notes (same level of the current note).
• display where to "insert" new note.
• display all the notes 1,2 levels up
• display all the top level notes
• display all the notes 1,2 levels down.
• go to next note this level.
• go to parent note
• etc...

Using the filename gives you an edge in scalability, because you don't have to open the files. All the needed information is in the filename already. Also, denote accomplishes much of its functionality just by leveraging the mind-boggingly clever naming scheme. To use the filename fits that approach. As I mentioned above, I tried Prot's idea of adapting the naming scheme with a "==SIGNATURE" component for some time:

I have been using the ==SIGNATURE in the same way and it is a good solution. virtual-dired-mode can be used for numerical sorting of the ID (a bit janky but it gets the job done)

The filename vs front-matter is an important consideration, I use ripgrep for searching backlinks and for me it annoyingly takes about 6 seconds, luckily the results are cached an this only happens the first time. But limiting the search to the first 6-10 lines or so of each file (the size of the front-matter) should be considerable faster. For my use case using the filename would be fine (you could probably add the listed functionality to Zettels, or complement it using bash or python scripting and fzf). But what if someone else has the need to add 10 items of metadata or 50 keywords, or something like that?. I have nothing in mind right now but someone else might have creative uses of the front-matter that we are not thinking about, that is why I was trying to highlight this issue.

[protesilaos]

Good day folks and thank you for this fruitful discussion!

We can have both the ==SIGNATURE and the inclusion of additional front matter. There is no problem in that regard. For me, the only consideration is to think about the design and to distill what we want. Then we can implement it.

I am not familiar with the technicalities of Luhnmann's method and am eager to learn from you. Understanding the workflows helps greatly. For example, it helps determine whether ==SIGNATURE is a single string of characters or whether it can be something like ==this=is=a=sig.

[sthesing]

Hi @protesilaos,

thanks for spending time with this!

I am not familiar with the technicalities of Luhnmann's method and am eager to learn from you. Understanding the workflows helps greatly. For example, it helps determine whether ==SIGNATURE is a single string of characters or whether it can be something like ==this=is=a=sig.

Hm... I don't have a definitive answer to this. My best attempt is to answer on three levels:

1. In principle, Luhmann's signature systematic has no use for delimiter characters (as in this=is=a=sig), since he designates a semantic change (a new branch of the sequence) with a change from enumeration to alphabetical order and so on: 1a3b etc. While it probably won't ever happen in practice, a signatures like 1a1000db2 is possible, if on the third branching level, a thousand notes follow each other.
2. In practice, Luhmann (and others) use(d) some special characters in their signatures. Luhmann e.g. used a broad thematic category at the beginning of his signatures, followed by a slash or comma and started his alternating system from there: 1/1b4d would be a different note than 2/1b4d.
3. In general, I believe that signature systems might differ from user to user.

Is it possible to just let a user put some arbitrary string between == and --? Or do you need a more restricted set of rules for the regexp to make it work?

[protesilaos]

Thank you @sthesing for the explanation!

Is it possible to just let a user put some arbitrary string between == and --? Or do you need a more restricted set of rules for the regexp to make it work?

Everything is possible. What matters is the workflow we have in mind. What I have understood thus far is that the ==SIGNATURE is best left as a free-form string that the user will provide. Each person will design their own workflow around it. In this case, Denote will not do anything smart, such as process the input to add delimiters between words (the way it does for the --TITLE and __KEYWORDS).

I think we can have it be free-form in this way because it is an opt-in feature. The default file-naming scheme will still be what we have. The user who wants the ==SIGNATURE will add signature to the denote-prompts and/or use a convenience command such as denote-signature alias denote-create-note-using-signature (e.g. look at the denote-subdirectory command).

Using the ==SIGNATURE will add a relevant entry to the front matter, unless we think it shouldn't (to be discussed).

Does this make sense?

The alternative is to treat ==SIGNATURE the same way we do with __KEYWORDS:

• add the = delimiter between words;
• infer existing signature entries from file names;
• use completing-read-multiple to insert one or more signatures at the relevant prompt.

That granted, I think making ==SIGNATURE yet another __KEYWORDS effectively restricts its potential and duplicates what we are already doing.

[sthesing]

What I have understood thus far is that the ==SIGNATURE is best left as a free-form string that the user will provide. Each person will design their own workflow around it. In this case, Denote will not do anything smart

Yes, I think that is the most universal and useful use case for those who opt in.

Using the ==SIGNATURE will add a relevant entry to the front matter, unless we think it shouldn't (to be discussed).

Yes, having the signature present in the buffer (e.g. as front matter) would be useful. For example, I imagine that a user who wants to create a followup note to the note in their current buffer would copy the current note's signature and paste it into the prompt for the new note and adapt it as needed.

The alternative is to treat ==SIGNATURE the same way we do with __KEYWORDS: -- snip -- That granted, I think making ==SIGNATURE yet another __KEYWORDS effectively restricts its potential and duplicates what we are already doing.

I agree. That would only serve as two separate sets of keywords.

[MirkoHernandez]

Everything is possible. What matters is the workflow we have in mind. What I have understood thus far is that the ==SIGNATURE is best left as a free-form string that the user will provide. Each person will design their own workflow around it. In this case,

Thanks @protesilaos this would be perfect, at least for my use case. A variable similar to denote--id-regexp is all that I had in mind but commands like denote-create-note-using-signature seem really convenient.

Yes, having the signature present in the buffer (e.g. as front matter) would be useful. For example, I imagine that a user who wants to create a followup note to the note in their current buffer would copy the current note's signature and paste it into the prompt for the new note and adapt it as needed.

@sthesing Yes! this is why I think a zettelkasten workflow is so good compared with a link based workflow. Navigating and creating the notes by using commands that manipulate the Id and not inserting and following links manually.

[protesilaos]

Very well! I will implement this in the coming days. Maybe over this weekend. Will keep you posted.

[protesilaos]

I found some time and started working on it, but I must now pause it and resume at another time. The diff below.

What remains to be done:

• Update all renaming commands.
• Make it handle front matter.
• Document it in the manual.

 denote.el | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++-------------
 1 file changed, 67 insertions(+), 17 deletions(-)

diff --git a/denote.el b/denote.el
index 8632998..c400f3b 100644
--- a/denote.el
+++ b/denote.el
@@ -210,6 +210,8 @@ (defcustom denote-prompts '(title keywords)
   value of that KEY is used to populate the new note with
   content, which is added after the front matter.

+- TODO 2022-12-23: document `signature'.
+
 The prompts occur in the given order.

 If the value of this user option is nil, no prompts are used.
@@ -431,6 +433,9 @@ (define-obsolete-variable-alias
 (defconst denote-id-regexp "\\([0-9]\\{8\\}\\)\\(T[0-9]\\{6\\}\\)"
   "Regular expression to match `denote-id-format'.")

+(defconst denote-signature-regexp "==\\([[:alnum:][:nonascii:]]*\\)"
+  "Regular expression to match the SIGNATURE field in a file name.")
+
 (define-obsolete-variable-alias
   'denote--title-regexp
   'denote-title-regexp
@@ -580,6 +585,12 @@ (define-obsolete-function-alias
   'denote-file-has-identifier-p
   "1.0.0")

+(defun denote-file-has-signature-p (file)
+  "Return non-nil if FILE has a Denote identifier."
+  (when file
+    (string-match-p denote-signature-regexp
+                    (file-name-nondirectory file))))
+
 (defun denote-file-directory-p (file)
   "Return non-nil if FILE is a directory.
 Omit FILE if it matches the value of user option
@@ -1167,6 +1178,14 @@ (defun denote-retrieve-filename-identifier (file)
         (match-string 0 file))
     (error "Cannot find `%s' as a file with a Denote identifier" file)))

+(defun denote-retrieve-filename-signature (file)
+  "Extract signature from FILE name."
+  (if (denote-file-has-signature-p file)
+      (progn
+        (string-match denote-signature-regexp file)
+        (match-string 1 file))
+    (error "Cannot find `%s' as a file with a Denote signature" file)))
+
 (define-obsolete-function-alias
   'denote--retrieve-filename-identifier
   'denote-retrieve-filename-identifier
@@ -1293,13 +1312,14 @@ ;;;; New note

 ;;;;; Common helpers for new notes

-(defun denote-format-file-name (path id keywords title-slug extension)
+(defun denote-format-file-name (path id keywords title-slug extension &optional signature)
   "Format file name.
-PATH, ID, KEYWORDS, TITLE-SLUG are expected to be supplied by
-`denote' or equivalent: they will all be converted into a single
-string.  EXTENSION is the file type extension, as a string."
+PATH, ID, KEYWORDS, TITLE-SLUG, EXTENSION and optional SIGNAUTRE
+are expected to be supplied by `denote' or equivalent command."
   (let ((kws (denote--keywords-combine keywords))
         (file-name (concat path id)))
+    (when (and signature (not (string-empty-p signature)))
+      (setq file-name (concat file-name "==" signature)))
     (when (and title-slug (not (string-empty-p title-slug)))
       (setq file-name (concat file-name "--" title-slug)))
     (when (and keywords (not (string-blank-p kws)))
@@ -1334,13 +1354,16 @@ (defun denote--format-front-matter (title date keywords id filetype)
          (kws (denote--format-front-matter-keywords keywords filetype)))
     (if fm (format fm title date kws id) "")))

-(defun denote--path (title keywords dir id file-type)
-  "Return path to new file with ID, TITLE, KEYWORDS and FILE-TYPE in DIR."
+(defun denote--path (title keywords dir id file-type &optional signature)
+  "Return path to new file.
+Use ID, TITLE, KEYWORDS, FILE-TYPE and optional SIGNATURE to
+construct path to DIR."
   (denote-format-file-name
    dir id
    (denote-sluggify-keywords keywords)
    (denote-sluggify title)
-   (denote--file-extension file-type)))
+   (denote--file-extension file-type)
+   (denote--slug-no-punct signature)))

 ;; Adapted from `org-hugo--org-date-time-to-rfc3339' in the `ox-hugo'
 ;; package: <https://github.com/kaushalmodi/ox-hugo>.
@@ -1384,12 +1407,13 @@ (defun denote--date (date file-type)
      (t
       (denote-date-org-timestamp date)))))

-(defun denote--prepare-note (title keywords date id directory file-type template)
+(defun denote--prepare-note (title keywords date id directory file-type template &optional signature)
   "Prepare a new note file.

 Arguments TITLE, KEYWORDS, DATE, ID, DIRECTORY, FILE-TYPE,
-and TEMPLATE should be valid for note creation."
-  (let* ((path (denote--path title keywords directory id file-type))
+TEMPLATE, and optional SIGNATURE should be valid for note
+creation."
+  (let* ((path (denote--path title keywords directory id file-type signature))
          (buffer (find-file path))
          (header (denote--format-front-matter
                   title (denote--date date file-type) keywords
@@ -1471,7 +1495,7 @@ (define-obsolete-function-alias
 ;;;;; The `denote' command and its prompts

 ;;;###autoload
-(defun denote (&optional title keywords file-type subdirectory date template)
+(defun denote (&optional title keywords file-type subdirectory date template signature)
   "Create a new note with the appropriate metadata and file name.

 When called interactively, the metadata and file name are prompted
@@ -1498,9 +1522,11 @@ (defun denote (&optional title keywords file-type subdirectory date template)

 - TEMPLATE is a symbol which represents the key of a cons cell in
   the user option `denote-templates'.  The value of that key is
-  inserted to the newly created buffer after the front matter."
+  inserted to the newly created buffer after the front matter.
+
+- SIGNATURE is a string or a function returning a string."
   (interactive
-   (let ((args (make-vector 6 nil)))
+   (let ((args (make-vector 7 nil)))
      (dolist (prompt denote-prompts)
        (pcase prompt
          ('title (aset args 0 (denote-title-prompt
@@ -1512,7 +1538,8 @@ (defun denote (&optional title keywords file-type subdirectory date template)
          ('file-type (aset args 2 (denote-file-type-prompt)))
          ('subdirectory (aset args 3 (denote-subdirectory-prompt)))
          ('date (aset args 4 (denote-date-prompt)))
-         ('template (aset args 5 (denote-template-prompt)))))
+         ('template (aset args 5 (denote-template-prompt)))
+         ('signature (aset args 6 (denote-signature-prompt)))))
      (append args nil)))
   (let* ((title (or title ""))
          (file-type (denote--valid-file-type (or file-type denote-file-type)))
@@ -1528,9 +1555,10 @@ (defun denote (&optional title keywords file-type subdirectory date template)
                       (denote-directory)))
          (template (if (stringp template)
                        template
-                     (or (alist-get template denote-templates) ""))))
+                     (or (alist-get template denote-templates) "")))
+         (signature (or signature "")))
     (denote-barf-duplicate-id id)
-    (denote--prepare-note title kws date id directory file-type template)
+    (denote--prepare-note title kws date id directory file-type template signature)
     (denote--keywords-add-to-history keywords)))

 (defvar denote--title-history nil
@@ -1647,6 +1675,13 @@ (define-obsolete-function-alias
   'denote-template-prompt
   "1.0.0")

+(defvar denote--signature-history nil
+  "Minibuffer history of `denote-signature-prompt'.")
+
+(defun denote-signature-prompt ()
+  "Prompt for signature string."
+  (read-string "Provide signature: " nil 'denote--signature-history))
+
 ;;;;; Convenience commands as `denote' variants

 (defalias 'denote-create-note 'denote
@@ -1720,6 +1755,20 @@ (defun denote-template ()
 (defalias 'denote-create-note-with-template 'denote-template
   "Alias of `denote-template' command.")

+;;;###autoload
+(defun denote-signature ()
+  "Create note while prompting for a file signature.
+
+This is the equivalent to calling `denote' when `denote-prompts'
+is set to \\='(signature title keywords)."
+  (declare (interactive-only t))
+  (interactive)
+  (let ((denote-prompts '(signature title keywords)))
+    (call-interactively #'denote)))
+
+(defalias 'denote-create-note-using-signature 'denote-signature
+  "Alias of `denote-signature' command.")
+
 ;;;;; Other convenience commands

 (defun denote--extract-title-from-file-history ()
@@ -2069,10 +2118,11 @@ (defun denote-rename-file (file title keywords &optional date)
       current-prefix-arg)))
   (let* ((dir (file-name-directory file))
          (id (denote-retrieve-or-create-file-identifier file date))
+         (signature (denote-retrieve-filename-signature file))
          (extension (file-name-extension file t))
          (file-type (denote-filetype-heuristics file))
          (new-name (denote-format-file-name
-                    dir id keywords (denote-sluggify title) extension))
+                    dir id keywords (denote-sluggify title) extension signature))
          (max-mini-window-height 0.33)) ; allow minibuffer to be resized
     (when (denote-rename-file-prompt file new-name)
       (denote-rename-file-and-buffer file new-name)

EDIT: Added missing task for front matter.

[sthesing]

I found some time and started working on it, but I must now pause it and resume at another time. The diff below.

What remains to be done:

* Update all renaming commands.

* Make it handle front matter.

* Document it in the manual.

This is exciting! Thank you so much, @protesilaos . Will test it. If there's anything else you want me to do, just say the word.!

[protesilaos]

@sthesing The code is in the signature branch. The tricky part is to get it into the front matter without interfering with existing setups and/or with the workflow of people who will not use the signature.

[nobiot]

@protesilaos, all

The tricky part is to get it into the front matter without interfering with existing setups and/or with the workflow of people who will not use the signature.

This might be a good prompt for me to raise a related question and mention the experiment I have been doing.

The question is: How far would we, as a project, like to push Denote's flexibility with naming scheme?

I am referring to this from @protesilaos above:

the "Denote way" IS NOT the "one true way" of naming notes.

Here is my experimentation: https://git.sr.ht/~nobiot/denote-custom-file-name-scheme

It lets us quickly experiment with the following features:

1. Adding N number of note's properties ("fields")
2. Defining the file name scheme with all or subset of the properties you define in (1) and delineators of choice ("==", "--", "__" and so on)
3. Changing the sequence of properties in the file name
4. Defining non-date property as the note's ID -- separating the ID and date
5. Adding any properties to the note's front matter upon creation of notes (for renaming/updating existing notes, only a fix set of properties can be altered by Denote at the moment)

The files relevant for the present discussion are:

• denote-custom-file-name.el
• denote-custom-file-rename.el
• denote-custom-zettel-id.el (zettel-ID is the "signature ID")

This is not a productive code and by no means means meant to replace the current Denote's code. My intention is that it lets us quickly experiment with different naming schemes and systematically see where things fail or hold.

As Prot notes, front matter presents challenges, especially in file rename (see denote-custom-file-rename.el).

At a minimum, it seems that we would need to review the signature (API) of the following functions:

• denote--rewrite-front-matter
• denote--add-front-matter
• denote--format-front-matter

There are more observations I have from the experiment, but I think I will stop here for now.

Back to the question: How far would we, as a project, like to push Denote's flexibility with naming scheme? Would we allow N number of properties and any sequence of them in the file name? Any properties in the front matter?

I think this should also be asked in the wider forum in the mailing list.

nobiot

[protesilaos]

@nobiot

The question is: How far would we, as a project, like to push Denote's flexibility with naming scheme?

It is a good question. What we are doing now with the ==SIGNATURE already gives us an idea of what to expect. It is still not done, though it shows how we need to revise a few things before we proceed, specifically with how we handle the front matter.

In general, the tricky part with all this flexibility is that it gets complex internally. My concern is that many permutations of the file name will also be hard to document. If, however, we can do this in a reasonable way, then I see no problem with being flexible.

---

I think the way the front matter is defined right now as one large string is not good. Instead of this:

(defvar denote-org-front-matter
  "#+title:      %s
#+date:       %s
#+filetags:   %s
#+identifier: %s
\n"
  "Org front matter.
It is passed to `format' with arguments TITLE, DATE, KEYWORDS,
ID.  Advanced users are advised to consult Info node `(denote)
Change the front matter format'.")

We can have this (documentation to-be-updated):

(defvar denote-org-front-matter
  '("#+title:      %s"
    "#+date:       %s"
    "#+filetags:   %s"
    "#+identifier: %s"
    "\n")
  "Org front matter.
It is passed to `format' with arguments TITLE, DATE, KEYWORDS,
ID.  Advanced users are advised to consult Info node `(denote)
Change the front matter format'.")

This gives us the power to insert another string in this list. We can also keep backward-compatibility by checking if the front matter value is a list or string.

We can format the list of strings as we like, such as:

(mapconcat #'identity FRONT-MATTER "\n")

---

@nobiot

At a minimum, it seems that we would need to review the signature (API) of the following functions:

• denote--rewrite-front-matter
• denote--add-front-matter
• denote--format-front-matter

I agree. Those were kept private because they can still be improved. A simple rewrite of denote--format-front-matter that accepts a list of strings for the front matter is this:

(defun denote--format-front-matter (title date keywords id filetype)
  "Front matter for new notes.

TITLE, DATE, KEYWORDS, FILENAME, ID are all strings which are
provided by `denote'.  FILETYPE is one of the values of
`denote-file-type'."
  (let* ((front-matter (denote--front-matter filetype))
         (fm (if (listp front-matter) (mapconcat #'identity front-matter "\n") front-matter))
         (title (denote--format-front-matter-title title filetype))
         (kws (denote--format-front-matter-keywords keywords filetype)))
    (if fm (format fm title date kws id) "")))

---

@nobiot

I think this should also be asked in the wider forum in the mailing list.

Yes, I agree. My general idea is that Denote is already flexible so we can make it even more so. If we are smart about it, we will handle complexity just fine. Though we should be careful not to introduce too many features too quickly, as it is then hard to withdraw them (we will be breaking user's workflows).

EDIT: fix format of blockquote.

[sthesing]

@protesilaos

I've been using the signature branch for the last few days and it works fine for my workflow. Creating notes with signatures works as expected and updating keywords (i.e. renaming using the front matter) respects the existing signature. In my everyday usage and I've run into no bugs or such and it's already incredibly helpful. So, thanks!

The tricky part is to get it into the front matter without interfering with existing setups and/or with the workflow of people who will not use the signature.

Yeah, I imagine it can quickly become quite complex with every new "field" in the filename/front matter.

Speaking of people who will not use the signature: One thing I did notice is that notes without a signature can't be renamed using the front-matter (as of commit 1272a12): Cannot find ‘/home/user/foo/20220101T133700--bar__baz.org’ as a file with a Denote signature

[MirkoHernandez]

@protesilaos

I fund a minor issue with denote--path , the optional signature argument is being used as a required argument. My workaround, just a when statement, I'm not too familiar with the code base but I guess that nothing else should be required.

(defun denote--path (title keywords dir id file-type &optional signature)
  "Return path to new file.
Use ID, TITLE, KEYWORDS, FILE-TYPE and optional SIGNATURE to
construct path to DIR."
  (denote-format-file-name
   dir id
   (denote-sluggify-keywords keywords)
   (denote-sluggify title)
   (denote--file-extension file-type)
   (when signature
     (denote--slug-no-punct signature))))

[protesilaos]

@MirkoHernandez Yes, the when is correct in this case. Do you want to send a PR or patch?

@sthesing The front matter is the blocking issue right now. It is not flexible enough and we cannot just augment the file-naming scheme without it. Turning the front matter variables from a single string to a list of strings that get concatenated can help us a bit, but is still not a robust solution. Basically, we need to be able to insert front matter conditionally but also to be smart about all related renaming operations.

[MirkoHernandez]

Unfortunately I won't be able to until a few weeks. I hope that posting the issues is helpful for now. I have found denote incredibly useful, so I will contribute some code as soon as I can. Skimming some of the comments I see that there is interest in improving denote's flexibility in some of its core parts, I don't know how useful this suggestion would be but I have found the book Software Design for Flexibility extremely useful for this kind of programming, some of the techniques are very Lisp centric and hard to implement in other languages but it should be straight forward in Emacs Lisp.

[MirkoHernandez]

Just a suggestion for functions that are intended for users but can also be called from code. Use the approach used for dired, an optional parameter no-error-if- for those kinds of functions (no-error-if-not-found-p, no-error-if-not-dir-p). Right now I have a custom version of some denote functions that are basically the same but ignore errors. (my/denote-retrieve-filename-signature).

Right now the mentioned example causes an issue in denote-rename-file for notes without a signature, denote-retrieve-filename-signature triggers an error.

[xiaoxinghu]

This is a long thread full of inspirational discussions. At the risk of being off-topic, it seems to me that the kind of information that's useful for filtering/sorting notes in file names can be roughly summarised as:

• date/time
• title
• many-to-many categorisation (for lack of a better word), i.e. tags in org-mode lingo
• one-to-many categorisation, i.e. category in org-mode lingo

My understanding of the Luhmann signature is very limited, but given the discussion above, it seems to me that it falls into the one-to-many kind of categorisation. Isn't that easily solvable with folders (if that's the case)?

The only difference I can think of is the way it renders in dired, instead of looking like this

DATE==SIGNATURE--TITLE__KEYWORDS.EXTENSIONS

It'd look more like this

SIGNATURE/DATE--TITLE__KEYWORDS.EXTENSIONS

(again, this is my own abstraction of the issue when I think about it, I could be wrong)

[protesilaos]

@MirkoHernandez:

Right now I have a custom version of some denote functions that are basically the same but ignore errors.

Sounds promising! Can you show me a sample?

Hello @xiaoxinghu:

(again, this is my own abstraction of the issue when I think about it, I could be wrong)

I am not an expert on this either. I think subdirectories can perform that role. The difference, I guess, is that they introduce a structure that can be broken more easily. For example, you accidentally move the file from the subdirectory to the main directory. As the file name does not carry its own information, it is no longer possible to know what its "role" is. Whereas embedding that information in the file name makes it self-explanatory.

The "signature" can be arranged on the user's side by tweaking their workflow accordingly. A tag could work for this as well, but it does not stand out as much so it is not exactly the same.

[MirkoHernandez]

Sounds promising! Can you show me a sample?

Yes, I just copy the functions and return nil instead of the error. Maybe I should have posted this as an example.

(defun my/denote-retrieve-filename-signature (file)
  "Modified version of denote-retrieve-filename-signature without error message."
  (when (denote-file-has-signature-p file)
      (progn
        (string-match denote-signature-regexp file)
        (match-string 1 file))))

I would like to reuse the original denote function instead of creating this kind of functions. The idea would be to do something like some of the dired functions (dired-insert-subdir). For instance denote-retrieve-filename-signature would be changed to something like this:

(defun denote-retrieve-filename-signature (file &optional no-error-if-no-signature)
  "Extract signature from FILE name."
  (if (denote-file-has-signature-p file)
      (progn
        (string-match denote-signature-regexp file)
        (match-string 1 file))
    (if no-error-if-no-signature 
           nil
         (error "Cannot find `%s' as a file with a Denote signature" file)
    ))

[nbehrnd]

@xiaoxinghu

many-to-many categorisation (for lack of a better word), i.e. tags in org-mode lingo

Let me speculate that you refer to controlled vocabulary, i.e. a constrained set of keywords/tags used to categorize something. I didn't get to know about controlled vocabulary as a name until I stumbled over Karl Voit's filetags (a Python-based implementation to organize files), and his talk at GLT18 The Advantages of File Name Conventions and Tagging. He addresses this about 16 min into the video.

And indeed, usually, the less tags at disposition, the better -- easier to recall what you have at disposition, easier to maintain. This approach really shines if you use multiple tags per item instead of a single one, and altogether with an orthogonal property not related to a keyword (time stamp). To have a large set of keywords at disposition only makes sense if you are so deep into the field, that by training one, or an other keyword became somewhat natural to think about it. An example may be the keyword index by Wiley-VCH specific for publications in chemistry.* Though policies vary by journal, about five keywords per publication is the limit here.

What I wonder -- perhaps these features already are implemented, or in development -- are points like

• does denote offer storage of a controlled vocabulary in an external file (in case of filetags, it is ~/.filetags)?
• does denote recognize some tags are already used previously, and to suggest their use for new items (sections, or files) by shortcut of a number (as later in the video)?
• does denote offer to define and use a controlled vocabulary of mutually exclusive tags (documents either in state of a draft, or eventually are final; either are for internal use only, or may be shared with the public)?

* It is quite useful if a journal/database has a public list of a constrained vocabulary curated by the editors, rather than the pattern of «it is up to the authors to define the keywords, their publication in front of them» (contrasting to select/choose from the cv).

[sthesing]

* one-to-many categorisation, i.e. `category` in org-mode lingo

My understanding of the Luhmann signature is very limited, but given the discussion above, it seems to me that it falls into the one-to-many kind of categorisation. Isn't that easily solvable with folders (if that's the case)?

That's not really the case, no. But I understand that it looks like a categorisation, because it groups similar thoughts and notions together. However, it doesn't group in a hierarchical logic (as categories do), nor a undirected logic (as keywords do) but in a sequential logic. So it's somehing inbetween "many-to-many" and "one-to-many". To illustrate that point, consider the following nested list, which might have started out like this:

- Apples
- Bananas
- Pineapples

but later evolved into this, with "Oranges" being the last addition:

- Apples
  - General thoughts about round things
    - Spheres
      - Oranges 
    - Circles   
- Bananas
- Pineapples

It looks like a hierarchy, but it is not. It's a (clumsy) representation of a semantic sequence. In fact, it's an excursus on geometry while thinking about fruits. When (at a later time) I search for oranges (or maybe keyword fruit), I'll find the notes on oranges, which might rely on some thought I had on round things. At the time I wrote the notes on Oranges, I probably wouldn't have felt the need to manually designate that relationship (like a denote link), because it was so present in my mind at the time I wrote it, one thought just naturally led to another. So I'd wish for a way to see the sequential context the note was written in.

To be clear: I know you didn't suggest that signatures are hierarchies. But folders and subfolders are designed to represent hierarchies. And yes, because such sequences can be clumsily represented like hierarchies, one could use a folder and subfolder structure to represent it. But the "==SIGNATURE" approach does it much better:

20210110T133700==1a--apples__fruit.org
20210110T133701==1b--bananas__fruit.org
20210110T133702==1c--pineapples__fruit.org
20220110T133700==1a1--round-things__geometry.org
20220110T133701==1a1a--spheres.org
20220110T133702==1a1b--circles.org
20230110T133700==1a1a1--oranges__fruit.org

It preserves a chronological order and I can inspect the semantic sequences (e.g. by filtering for =1a).

The only difference I can think of is the way it renders in dired, instead of looking like this

DATE==SIGNATURE--TITLE__KEYWORDS.EXTENSIONS

It'd look more like this

SIGNATURE/DATE--TITLE__KEYWORDS.EXTENSIONS

That is a significant difference. Both in workflow (having to create and navigate folders) and in presentation to the user (signature first instead of chronologically). A lot of the workflow problems that see could be probably be mitigated by helper functions and tweaks of the representation in dired, I imagine. But it wouldn't be in line with the wonderful approach of denote to be able to infer a lot of semantic information just by opening it in standard file management tools like dired, by searching and filtering without ever opening a file or entering a directory.

[protesilaos]

@nbehrnd

What I wonder -- perhaps these features already are implemented, or in development -- are points like

* does `denote` offer storage of a controlled vocabulary in an external file (in case of `filetags`, it is `~/.filetags`)?

No. The central idea of Denote is to not maintain any kind of database or external resource. The file name is designed in such a way as to provide as much valuable information as possible. It should be self-explanatory or, at least, easy to reason about and infer knowledge from.

This is why I like the =SIGNATURE: it is consistent with what we have been doing thus far and extends the principles in an organic way.

* does `denote` recognize some tags are already used previously, and to _suggest_ their use for new items (sections, or files) by shortcut of a number (as later in the video)?

Yes. This is done by reading the existing notes and inferring from them their keywords. Those inferred keywords are dynamically combined with a predefined list of keywords and are presented at the selection prompt when the user creates a new file (or performs a relevant operation). This way, the user can re-use an existing tag.

This functionality is subject to a user option for those who do not want to make use of it (because of the point below). The variable is called denote-infer-keywords.

* does `denote` offer to define and use a controlled vocabulary of mutually exclusive tags (documents either in state of a `draft`, or eventually are `final`; either are for `internal` use only, or may be shared with the `public`)?

No. What Denote offers is a way to define a list of keywords in advance. The user option is denote-known-keywords. Those "known keywords" are presented at the selection prompt as I noted above. I find that this gives us the [self-imposed] constrained set we need, especially if we disable the keyword inference.

I did not experiment with mutual exclusivity as I felt it would make the code complex. The cost seemed considerably higher that the benefit/convenience it would provide. I am not against the idea per se: I just think that it is not essential.

[MirkoHernandez]

@sthesing

I was wondering, how are you able to sort the entries according to the signature?. My current approach works but It is a mess, I need to use sed to modify the title '--' prefix to '=@' this way the sort command works correctly, afterwards I change it back. I'm not too familiar with the sort command, sort -Vk 3 -t = almost worked correctly but failed in a few cases. Maybe the signature would need to be enclosed using the same kind of separator.

I would be interested in a better solution to this problem, maybe It could be generalized to a sorting approach for any arbitrary signature. Here is my current sorting approach.

(defun my/zettel-execute-find-command (regex)
  (shell-command-to-string
   (concat "find * -regex " "'\..*==" regex ".*' | sed  's/--/=@/' | sort -t '=' -Vk 3,3 | sed 's/=@/--/' ")))

[sthesing]

@sthesing

I was wondering, how are you able to sort the entries according to the signature?

I don't. If they are sorted chronologically, they are already sorted by signature. It's just that the other notes from other note sequences are in between. So I filter. In dired, I type %, m to mark files by regex, enter my search string (e.g. =1a) to mark all files belonging to that sequence, followed by t to invert (toggle) and k to remove (kill) the other notes from the current view. Doing that with the fruit example above, would yield this listing in dired:

20210110T133700==1a--apples__fruit.org
20220110T133700==1a1--round-things__geometry.org
20220110T133701==1a1a--spheres.org
20220110T133702==1a1b--circles.org
20230110T133700==1a1a1--oranges__fruit.org

Of course, this only establishes a sorted order for signatures that designate a sequence - as is the case with Luhmann's style. Does this help you, @MirkoHernandez ?

[protesilaos]

@MirkoHernandez

I would be interested in a better solution to this problem, maybe It could be generalized to a sorting approach for any arbitrary signature.

Depending on your workflow, you can use "virtual dired": https://protesilaos.com/emacs/denote#h:d35d8d41-f51b-4139-af8f-9c8cc508e35b

[MirkoHernandez]

@sthesing Thanks for the clarification. I'm using a similar approach but with vertico instead of dired.

@protesilaos I have found virtual dired to be very limiting compared to regular dired. I'm currently using vertico for finding notes and maybe I will use vertico + embark in the future for particular use cases.

My understanding is that the sorting that dired does depends on ls, sort, or other command line tools. I think It would be helpful to establish the signature using separators that enclose it. So that the sort command can work very easily with the -k option.

20210110T133700==1a=--apples__fruit.org
20220110T133700==1a1=--round-things__geometry.org
or 
20210110T133700__1a_--apples__fruit.org
20220110T133700__1a1_--round-things__geometry.org

just a suggestion as the signature idea is in early development.

[protesilaos]

@MirkoHernandez

I have found virtual dired to be very limiting compared to regular dired. I'm currently using vertico for finding notes and maybe I will use vertico + embark in the future for particular use cases.

I agree. The other option is to try the new denote-menu package: https://github.com/namilus/denote-menu. I still need to mention it in the manual.

just a suggestion as the signature idea is in early development.

The application you have in mind makes the suggestion appropriate, though I do not like this direction for two reasons:

1. It does not look nice and is not consistent with the other delimited fields.

2. We are thinking in terms of sort or ls whereas we should remain generic. Some other tool, like what you mention with vertico and embark, may be able to do a better job. The pattern we are discussing now makes the end point of the signature predictable as it will be among those:

   DATE==SIGNATURE--TITLE__KEYWORDS.EXTENSION
   DATE==SIGNATURE--TITLE.EXTENSION
   DATE==SIGNATURE__KEYWORDS.EXTENSION
   DATE==SIGNATURE.EXTENSION
   DATE==SIGNATURE

I have not experimented much with vertico's sorting, though I would expect it, together with something like orderless, to give us the pattern-matching flexibility we need. Capturing/exporting those with embark is the final step.

Furthermore, note that we need to economise on our use of the = as we might need it as a separator within the SIGNATURE, just how we do it with - and _. I am not saying we "should" have a separator here, but it is better to have that option if needed. Otherwise we will have to break stuff to make the requisite changes.

[xiaoxinghu]

@sthesing Fascinating! Thanks for the detailed explanation. My understanding of the signature is that it visualises the thought process in list view, just like the graph view in other roam-like systems, but without reading the content of the files, which is the ingenious part of denote's design around file names. Am I right?

So you probably can archive a similar effect by reading all the files and parsing all the links to a graph, but that would be super inefficient, or you can mitigate that with an external DB, which we don't want.

[sthesing]

@sthesing Fascinating! Thanks for the detailed explanation. My understanding of the signature is that it visualises the thought process in list view, just like the graph view in other roam-like systems

I'm not too familiar with roam, but I believe that graph would show manual links between notes. That's a different sort of relation. In denote, it's what the denote:-links do. And I believe someone is working on a visualization tool, too. I also think that roam doesn't have the functionality to designate note sequences, at the moment. They lack a concept of what Luhmann called a "Folgezettel". If you're interested, I blogged about it, the other day.

And yes, the cool thing about denote's ingenious file naming scheme is that so much semantic relationships between notes can be encoded in the filename, without the need to open the files or maintain a db.