Creation of new notes in subdirectories.

[jeanphilippegg]

As @protesilaos suggested in this closed pull request, I am opening this issue to start a discussion on the creation of notes in subdirectories.

As of now, new notes are created at the root of the denote-directory. @protesilaos suggested a few alternatives:

• We already have in the manual samples of let binding the denote-directory: https://protesilaos.com/emacs/denote#h:f34b172b-3440-446c-aec1-bf818d0aabfe. We can update/repurpose/copy those for a new section that talks about subdirectory support.

• Have a new user option, which will be off by default, that makes all our file-creating commands use the current directory if it is a subdir of denote-directory.

I am adding another possibility to the list:

• Prompt the user for a directory where to create the note. The default would be the current directory if it is inside denote-directory.

[jeanphilippegg]

Here is the rationale behind my suggestion: I see subdirs just like additional tags (hierarchical and in front of the title). Tags being prompted, it seems consistent that the directory would be prompted too. For users with a flat directory structure, this additional prompting could be annoying, so I propose this option:

(defcustom denote-prompt-for-directory t
   "If t (the default), prompt for a directory in which to create the note.
    If nil, create the note at the root of the `denote-directory`.")

[protesilaos]

In the spirit of your suggestion, we can have a new command denote-subdir which will always prompt for a directory. Kind of how we do it with denote-type. We just need helper functions to determine the subdir candidates.

Whatever we do, I think we need to pick one to keep things simple for us and the user.

[jeanphilippegg]

We commented within seconds of each other, but yes, that's another good idea!

[protesilaos]

Indeed! I say we keep this issue open for a little while to see if others have any feedback to share.

[shrysr]

In the spirit of your suggestion, we can have a new command denote-subdir which will always prompt for a directory. Kind of how we do it with denote-type. We just need helper functions to determine the subdir candidates.

This makes sense to me, and has been how I've been mixing text and org files for example. For example - most of your notes may go into the main directory, but you can pick and choose when you specifically want a subdirectory prompt. The latter could also be augmented with org-capture for those who desire it.

[protesilaos]

[about denote-subdir] This makes sense to me [...]

I think a command gives us maximum flexibility. It is consistent with how we provide M-x denote-type.

@ggjp Are you okay if we implement denote-subdirectory? We can also have a defalias for it, such as denoete-new-note-in-subdirectory.

The latter could also be augmented with org-capture for those who desire it.

Of course! I forgot about it. We must take care to make this work.

[jeanphilippegg]

@ggjp Are you okay if we implement denote-subdirectory? We can also have a defalias for it, such as denoete-new-note-in-subdirectory.

Yes! I think it is a good idea!

[protesilaos]

Thanks! I will work on it later. For now let's priority the id: part.

[protesilaos]

I found some time to write this:

;;;;; The `denote-subdirectory' command

(defvar denote--subdir-history nil
  "Minibuffer history of `denote-subdirectory'.")

(defun denote--subdirs ()
  "Return list of subdirectories in variable `denote-directory'."
  (seq-remove
   (lambda (filename)
     ;; TODO 2022-07-03: Generalise for all VC backends.  Which ones?
     ;;
     ;; TODO 2022-07-03: Maybe it makes sense to also allow the user to
     ;; specify a blocklist of directories that should always be
     ;; excluded?
     (or (string-match-p "\\.git" filename)
         (not (file-directory-p filename))))
   (directory-files-recursively (denote-directory) ".*" t t)))

(defun denote--subdirs-completion-table (dirs)
  "Match DIRS as a completion table."
  (let* ((def (car denote--subdir-history))
         (table (denote--completion-table 'file dirs))
         (prompt (if def
                     (format "Select subdirectory [%s]: " def)
                   "Select subdirectory: ")))
    (completing-read prompt table nil t nil 'denote--subdir-history def)))

(defun denote--subdirs-prompt ()
  "Handle user input on choice of subdirectory."
  (let ((subdirs (if (null (denote--subdirs))
                     (user-error "No subdirs in `%s'; create them manually"
                                 (denote-directory))
                   (denote--subdirs))))
    (denote--subdirs-completion-table subdirs)))

;;;###autoload
(defun denote-subdirectory (directory title keywords)
  "Like `denote' but ask for DIRECTORY to put the note in.
The DIRECTORY is a subdirectory of the variable `denote-directory'.

The TITLE and KEYWORDS are the same as the `denote' command."
  (interactive
   (list
    (denote--subdirs-prompt)
    (denote--title-prompt)
    (denote--keywords-prompt)))
  (let ((denote-directory directory))
    (denote--prepare-note title keywords)
    (denote--keywords-add-to-history keywords)))

(defalias 'denote-create-note-in-subdirectory (symbol-function 'denote-subdirectory))

The only problem I see right now is with the inferred keywords. They only read the root directory, not all subdirs:

• Do we want it to read everything from all directories?
• Or do we let each subdir be its own silo? In this case, we need to revise the denote--inferred-keywords and related functions.

EDIT: fix formatting.

EDIT2: Rewrote the denote-directory command.

[jeanphilippegg]

The only problem I see right now is with the inferred keywords. They only read the root directory, not all subdirs:

* Do we want it to read everything from all directories?

* Or do we let each subdir be its own silo?  In this case, we need to
  revise the `denote--inferred-keywords` and related functions.

I think what the user is suggested in the keywords prompt should include all keywords from denote-directory and its subdirs. I thought it was already the case.

EDIT: I have tested and denote--inferred-keywords returns all keywords in all subdirs for me. Is it not what you get?

[jeanphilippegg]

Thanks for the code! I think let binding the denote-directory will restrict inferred-keywords to the selected subdir. Is that what your meant?

[shrysr]

I think what the user is suggested in the keywords prompt should include all keywords from denote-directory and its subdirs.

My thoughts as well. If the same keywords are meant to have different contexts, the user would know based on note location anyway. Also, perhaps I am mistaken, but maybe it is possible to enable silo based tag restriction while renaming the note, perhaps through .dir-locals.el (as was explained in the mailing list for restricted linking) ?

;; Maybe it makes sense to also allow the user to ;; specify a blocklist of directories that should always be ;; excluded?

I think yes, but I'm not too sure. Maybe it would be more flexible to consider a regex based exclusion instead of subdir blocklist?

[protesilaos]

Sorry folks! Apparently denote-keywords already reads the subdirs. I missed it...

I updated the denote-directory and think it works properly now:

(defun denote-subdirectory (directory title keywords)
  "Like `denote' but ask for DIRECTORY to put the note in.
The DIRECTORY is a subdirectory of the variable `denote-directory'.

The TITLE and KEYWORDS are the same as the `denote' command."
  (interactive
   (list
    (denote--subdirs-prompt)
    (denote--title-prompt)
    (denote--keywords-prompt)))
  (let ((denote-directory directory))
    (denote--prepare-note title keywords)
    (denote--keywords-add-to-history keywords)))

Also, perhaps I am mistaken, but maybe it is possible to enable silo based tag restriction while renaming the note, perhaps through .dir-locals.el (as was explained in the mailing list for restricted linking) ?

I have not tested it, but in principle we can make that happen. It might need changes in the code base, however, as denote-infer-keywords is a boolean whereas we might need to provide different options.

In general, the user options were designed with a flat directory in mind, so we might need to repurpose them now that subdir support is in main.

;; Maybe it makes sense to also allow the user to > ;; specify a blocklist of directories that should always be > ;; excluded?

I think yes, but I'm not too sure. Maybe it would be more flexible to consider a regex based exclusion instead of subdir blocklist?

Whatever makes more sense. We can, at an initial stage, implement it as a defvar (not defcustom), to get a feel for it. I think it is better not to overwhelm users with premature options. There are some other cases in the code where this approach is used.

[protesilaos]

About directory-local "silo" tags, I forgot to mention the user option denote-known-keywords. Again, I have not tested this, but together with a nil value for denote-infer-keywords it can create a restricted vocabulary for select dirs. This issue needs to be explored further and be mentioned in the docs.

[jeanphilippegg]

I think the root directory should be selectable when prompted for a subdirectory. I have not tested, but maybe we can just add denote-directory to denote--subdirs.

[protesilaos]

I think the root directory should be selectable when prompted for a subdirectory.

Oh right. I just thought the user would invoke denote for that. But I don't mind including it.

[protesilaos]

Just pushed the change. Please give it a try.

[jeanphilippegg]

I gave it a try. It seems to work great!

[protesilaos]

I am closing this now. If there are bugs about it, feel welcome to start a new thread.