Nested silo interferes with `denote-directory-files` (was: "dblock enhancement add a silo target")

[aadi58002]

Denote config

(use-package denote
  :config
  ;; Load silo extras as well
  (require 'denote-silo-extras)
  (setq denote-excluded-directories-regexp ".*[archived|addons].*"
        denote-known-keywords ()
        denote-save-buffers t
        denote-date-prompt-use-org-read-date t
        denote-directory "~/Documents/denote"
        denote-notes-directory "~/Documents/denote/notes"
        denote-tasks-directory "~/Documents/denote/tasks"
        denote-pastime-directory "~/Documents/denote/pastime"
        denote-silo-extras-directories (list denote-notes-directory denote-pastime-directory denote-tasks-directory)))

Feature Request

I wanted to create a dblock that lists all file links with the tag _music in them. Since all my music-related notes are in the pastime silo, the dblock kept being empty because it might have been looking in the top-level directory of the denote-directory.

Proposed Solutions

• Update the buffer local value of denote-directory based on current buffer path and checking if it is inside any silo.
• Add a :recursive keyword that would allow the dblock to recursively search through all nested folders, such as :recursive t or :search recursive/nested.
• Add a :silo option to narrow down the denote-directory path to a specific silo directory. This option can be selected during dblock creation and can serve as an extra command for the silo extras package.

Current work around

Just put the below code in the .dir-locals.el file to set denote-directory to the correct silo path when inside that silo

;;; Directory Local Variables            -*- no-byte-compile: t -*-
;;; For more information see (info "(emacs) Directory Variables")

((nil . ((eval . (setq denote-directory denote-pastime-directory)))))

PS: Thank you for the amazing package! It has really helped me become a more organized person and motivated me to keep learning about software in general. Every past interaction I've had with you has been the best experience I've ever had in any community, and I'm very, very thankful that you continue to maintain these useful and beautiful packages.

[protesilaos]

From: Aditya Yadav @.***> Date: Sun, 4 Aug 2024 10:08:17 -0700

[... 20 lines elided]

I wanted to create a dblock that lists all file links with the tag _music in them. Since all my music-related notes are in the pastime silo, the dblock kept being empty because it might have been looking in the top-level directory of the denote-directory.

Interesting! Normally it is recursive (see 'denote-directory-files', which ultimately calls 'denote--directory-all-files-recursively'). Though the nested silo is probably interfering with this. I need to test it.

Proposed Solutions

• Add a :recursive keyword that would allow the dblock to recursively search through all nested folders, such as :recursive t or :search recursive/nested.

This should be the current behaviour. If it is not because of the silos, then we need to fix it.

• Add a :silo option to narrow down the denote-directory path to a specific silo directory. This option can be selected during dblock creation and can serve as an extra command for the silo extras package.

Given that the notion of silos is that they are strictly separate from each other, what is the general use-case here? In your specific scenario, and assuming we fix the aforementioned bug, you can specify the directory you want to target as part of the :regexp (since it is matched against full file system paths).

PS: Thank you for the amazing package! It has really helped me become a more organized person and motivated me to keep learning about software in general. Every past interaction I've had with you has been the best experience I've ever had in any community, and I'm very, very thankful that you continue to maintain these useful and beautiful packages.

You are welcome! I am happy to read this and help however I can. Denote benefits from the contributions of many people. We are lucky in this regard.

-- Protesilaos Stavrou https://protesilaos.com

[aadi58002]

I created a minimal example and its seems like I might have found a bug. Denote-silo-dblock-bug.webm

I first create the task1 file in the silo and then the metaTask file to group all the tasks with tag1.

On update I get 0 results. Then copying the files to denote-directory top level. Then updating the dblock in the top-level dir. I get the expected output.

~But then going back to the silo. On update it then seems to start picking up files with tag1 including it self~. Correction: It was picking up files in the denote-directory top level dir not in the silo dir.

PS: If the recording is not clear or any other sort of testing is required. Please let me know.

Config in use to switch to denote-test directory for testing

(setq denote-directory "~/denote-test"
      denote-notes-directory "~/denote-test/notes"
      denote-silo-extras-directories (list denote-notes-directory))

[aadi58002]

denote-test.zip

[protesilaos]

I cannot reproduce this on my end. Here is a screenshot, showing the entire setup:

Does the following not include the file inside the silo?

(denote--directory-all-files-recursively)

If not, then I suspect what may be happening is that it evaluates the (denote-directory) each time it switches to a folder, hence the bug.

(defun denote--directory-all-files-recursively ()
  "Return list of all files in variable `denote-directory'.
Avoids traversing dotfiles (unconditionally) and whatever matches
`denote-excluded-directories-regexp'."
  (directory-files-recursively
   (denote-directory)  ; THE PROBLEM MAY BE HERE
   directory-files-no-dot-files-regexp
   :include-directories
   #'denote--directory-files-recursively-predicate
   :follow-symlinks))

Please modify and then evaluate the following. Then call it to see how many files it returns:

(defun denote--directory-all-files-recursively ()
  "Return list of all files in variable `denote-directory'.
Avoids traversing dotfiles (unconditionally) and whatever matches
`denote-excluded-directories-regexp'."
  (directory-files-recursively
   "/path/to/your/denote-directory/not/the/silo"
   directory-files-no-dot-files-regexp
   :include-directories
   #'denote--directory-files-recursively-predicate
   :follow-symlinks))

[aadi58002]

All the above results are before creating a .dir-locals.el

On creating the .dir-locals.el file to change the denote-directory variable to the current silo path makes it work inside the silo.

[aadi58002]

Really sorry to take up your time with this issue. It seems like I wrote one of the exclude expression wrong in my config which was cause of the bug.

 (setq denote-excluded-directories-regexp ".*[archived|addons].*")

[protesilaos]

From: Aditya Yadav @.***> Date: Sun, 18 Aug 2024 07:28:25 -0700

Closed #407 as completed.

Hello again!

I did not have time to review this. Sorry! Did you close it by accident or did something change?

-- Protesilaos Stavrou https://protesilaos.com

[aadi58002]

Closing wasn't accident. During config changes i wrote a wrong regex by mistake that was the reason for all entire bug. (setq denote-excluded-directories-regexp ".*[archived|addons].*")

While it should have been (setq denote-excluded-directories-regexp ".*(archived|addons).*")

I messed up the brackets.

[aadi58002]

Thanks again for taking the time to create the example.

PS: Sorry if it took too much of your time.

[protesilaos]

Oh, I see. Good to know!