Suggest existing titles as completion candidates (was: When creating a new note with Denote, sometimes the candidate list disappeared.)

[yibie]

M-x denote

M-x denote-link-after-creating

M-x dentoe-create-note-with-signature

M-x dentoe-link-or-create

M-x denote-open-and-create

How to debug?

[yibie]

My Emacs version: GNU Emacs 29.4 (build 1, aarch64-apple-darwin23.6.0, NS appkit-2487.70 Version 14.6.1 (Build 23G93)) of 2024-09-16

My denot config:

(use-package! denote
  :ensure t
  :hook
  ((text-mode . denote-fontify-links-mode-maybe)
   (dired-mode . denote-dired-mode))
  :config
  (setq denote-directory (expand-file-name "~/Documents/notes/"))
  (setq denote-save-buffers nil)
  (setq denote-infer-keywords t)
  (setq denote-sort-keywords t)
  (setq denote-file-type nil) ; Org is the default, set others here
  (setq denote-prompts '(title))
  (setq denote-excluded-directories-regexp nil)
  (setq denote-excluded-keywords-regexp nil)
  (setq denote-rename-confirmations '(rewrite-front-matter modify-file-name))
  (setq denote-backlinks-show-context t)
  (setq denote-date-format nil)
  (setq denote-rename-buffer-mode t)
  (setq denote-rename-buffer-format "%s%k%t%d")
  (denote-rename-buffer-mode 1)
  (setq denote-org-front-matter
  "#+title:      %1$s
#+date:       %2$s
#+filetags:   %3$s
#+identifier: %4$s
\n")
   (with-eval-after-load 'org-capture
    (setq denote-org-capture-specifiers "%l\n%i\n%?")
    (add-to-list 'org-capture-templates
                 '("n" "New note (with denote.el)" plain
                   (file denote-last-path)
                   #'denote-org-capture
                   :no-save t
                   :immediate-finish nil
                   :kill-buffer t
                   :jump-to-captured t))

    ;; This prompts for TITLE, KEYWORDS, and SUBDIRECTORY
    (add-to-list 'org-capture-templates
                 '("N" "New note with prompts (with denote.el)" plain
                   (file denote-last-path)
                   (function
                    (lambda ()
                      (denote-org-capture-with-prompts :title :keywords :signature)))
                   :no-save t
                   :immediate-finish nil
                   :kill-buffer t
                   :jump-to-captured t)))
   )

(use-package! consult-denote
  :ensure t
  :after denote
  :config
  (consult-denote-mode 1))

(add-hook 'dired-mode-hook #'denote-dired-mode-in-directories)

(map! :leader
      :prefix ("d" . "denote")
      "n" #'denote
      "m" #'denote-image-note
      "c" #'denote-link-after-creating
      "z" #'list-denotes
      "u" #'denote-url-note
      "t" #'denote-template
      "i" #'denote-link
      "I" #'denote-add-links
      "f" #'denote-find-link
      "b" #'denote-backlinks
      "l" #'denote-find-back-links
      "r" #'denote-rename-file
      "R" #'denote-rename-file-using-front-matter
      "f" #'consult-denote-find
      "g" #'consult-denote-grep
      )

(let ((url-subdir (expand-file-name "urls" denote-directory)))
  (unless (file-exists-p url-subdir)
    (make-directory url-subdir t)))

(let ((img-subdir (expand-file-name "images" denote-directory)))
  (unless (file-exists-p img-subdir)
    (make-directory img-subdir t)))

(setq denote-templates
      '((url-note . "#+title: S-URL-%?
#+filetags: :url:

URL: %^{URL}

")))

(defun denote-url-note ()
  "Create a new URL note in the 'urls' subdirectory."
  (interactive)
  (let* ((title (read-string "Title: "))
         (url (read-string "URL: "))
         (denote-prompts nil)  
         (denote-file-type nil)  
         (denote-directory (expand-file-name "urls" denote-directory)))
    (denote
     (concat "S-URL " title)  
     '("url") 
     nil 
     "url-note")  
    (with-current-buffer (current-buffer)
      (goto-char (point-max))
      (insert url))))

(defvar my-image-source-directory "~/Documents/temp_convert/"
  "Directory containing images to be processed and added to Denote.")

(defvar my-supported-image-formats '("webp" "jpg" "jpeg" "png" "gif" "tiff" "bmp")
  "List of supported image file formats.")

(let ((img-target-dir (expand-file-name "images" denote-directory)))
  (unless (file-exists-p img-target-dir)
    (make-directory img-target-dir t)))

(setq denote-templates
      (append denote-templates
              '((image-note . "#+title: IMG-%?
#+filetags: :image:

File: [[file:%^{Image File}][%^{Image Name}]]
Description: %^{Image Description}

"))))

(defun my-filter-image-files (files)
  "Filter FILES to only include supported image formats."
  (seq-filter
   (lambda (file)
     (when-let ((ext (file-name-extension file)))
       (member (downcase ext) my-supported-image-formats)))
   files))

(defun my-denote-image-note ()
  "Create a new image note, selecting an image from a fixed source directory."
  (interactive)
  (let* ((image-files (my-filter-image-files
                       (directory-files my-image-source-directory nil nil t)))
         (image-file (when image-files
                       (expand-file-name
                        (completing-read "Select Image: " image-files nil t)
                        my-image-source-directory)))
         (image-name (when image-file (file-name-nondirectory image-file))))

    (if (not image-file)
        (message "No supported image files found in the source directory.")
      (let* ((title (read-string "Image Title: " (file-name-base image-name)))
             (description (read-string "Image Description: "))
             (denote-prompts nil)
             (denote-file-type nil)  
             (denote-directory (expand-file-name "images" denote-directory))
             (new-image-file (expand-file-name
                              (concat (format-time-string "%Y%m%d%H%M%S-") image-name)
                              denote-directory)))

        (copy-file image-file new-image-file t)

        (denote
         (concat "IMG-" title)
         '("image")
         nil
         "image-note")

        (with-current-buffer (current-buffer)
          (goto-char (point-max))
          (insert (format "File: [[file:%s][%s]]\nDescription: %s\n"
                          new-image-file image-name description)))))))

;;;denote-menu 

(use-package! denote-menu)

(define-key denote-menu-mode-map (kbd "c") #'denote-menu-clear-filters)
(define-key denote-menu-mode-map (kbd "/ r") #'denote-menu-filter)
(define-key denote-menu-mode-map (kbd "/ k") #'denote-menu-filter-by-keyword)
(define-key denote-menu-mode-map (kbd "/ o") #'denote-menu-filter-out-keyword)
(define-key denote-menu-mode-map (kbd "e") #'denote-menu-export-to-dired)

;;;denote-explore 
(use-package! denote-explore
  :custom
  ;; Location of graph files
  (denote-explore-network-directory "~/Documents/notes/graphs/")
  (denote-explore-network-filename "denote-network")
  ;; Output format
  (denote-explore-network-format 'graphviz)
  (denote-explore-network-graphviz-filetype "svg")
  ;; Exlude keywords or regex
  (denote-explore-network-keywords-ignore '("bib")))

[protesilaos]

Hello @yibie! If I understand the pictures correctly, you are missing the minibuffer history. For example, if denote-title-history is nil then you do not see any candidates in the minibuffer.

Does this happen while you are using Emacs or is it only after a restart?

Do you use the built-in savehist-mode? Its purpose is to save minibuffer histories.

[yibie]

Does this happen while you are using Emacs or is it only after a restart?

I’m not sure, at first the candidate list displayed normally, but after I restarted Emacs once or a few times, it stopped showing.

Do you use the built-in savehist-mode?

I’m not sure if savhist-mode was properly activated when I was using denote. Just now, I ran M-x savhist-mode, and then ran M-x denote, and the candidate list displayed correctly.

[protesilaos]

From: yibie @.***> Date: Wed, 18 Sep 2024 00:21:32 -0700

[... 5 lines elided]

Do you use the built-in savehist-mode?

I’m not sure if savhist-mode was properly activated when I was using denote. Just now, I ran M-x savhist-mode, and then ran M-x denote, and the candidate list displayed correctly.

Okay, keep using this setup and let me know if the issue appears again.

-- Protesilaos Stavrou https://protesilaos.com

[yibie]

My previous description was incorrect. Enabling savehist-mode was of no use. The reason why the notes list can be displayed is that I customized a function myself:

(defun my-denote-with-preview ()
  (interactive)
  (let* ((existing-notes (denote-directory-files))
         (existing-titles (mapcar (lambda (file)
                                    (or (denote-retrieve-title-value file 'org)
                                        (file-name-base file)))
                                  existing-notes))
         (chosen-title (ivy-read "Choose or enter new note title: "
                                 existing-titles
                                 :caller 'my-denote-with-preview
                                 :action (lambda (x)
                                           (setq ivy-text x))
                                 :require-match nil)))
    (if (member chosen-title existing-titles)
        (find-file (car (delq nil (mapcar (lambda (file)
                                            (when (string= (or (denote-retrieve-title-value file 'org)
                                                               (file-name-base file))
                                                           chosen-title)
                                              file))
                                          existing-notes))))
      (denote chosen-title))))

[protesilaos]

From: yibie @.***> Date: Wed, 18 Sep 2024 06:19:37 -0700

My previous description was incorrect. Enabling savehist-mode was of no use. The reason why the notes list can be displayed is that I customized a function myself:

(defun my-denote-with-preview ()
  (interactive)
  (let* ((existing-notes (denote-directory-files))
         (existing-titles (mapcar (lambda (file)
                                    (or (denote-retrieve-title-value file 'org)
                                        (file-name-base file)))
                                  existing-notes))
         (chosen-title (ivy-read "Choose or enter new note title: "
                                 existing-titles
                                 :caller 'my-denote-with-preview
                                 :action (lambda (x)
                                           (setq ivy-text x))
                                 :require-match nil)))
    (if (member chosen-title existing-titles)
        (find-file (car (delq nil (mapcar (lambda (file)
                                            (when (string= (or (denote-retrieve-title-value file 'org)
                                                               (file-name-base file))
                                                           chosen-title)
                                              file))
                                          existing-notes))))
      (denote chosen-title))))

I do not know how 'ivy-read' works. What we do internally is arrange to display the entries from the history as completion candidates. Maybe this is something you can also do here?

-- Protesilaos Stavrou https://protesilaos.com

[yibie]

I can understand that by accessing the file history, it’s also possible to filter notes. Additionally, if it’s a newly created file, the history can easily differentiate between old and new notes.

However, the problem now is that the history doesn’t include my previous notes. Perhaps, building the history could create a mechanism similar to file caching, which could serve as an interface in the future, giving denote more potential for development. But I think a function should be introduced to include old notes into the file history. This would help avoid the issue I’m facing.

Also, I’m not sure what denote’s history function is, but I can give it a try and see what I can do.

[protesilaos]

Also, I’m not sure what denote’s history function is, but I can give it a try and see what I can do.

Let me take a step back here to explain how this is supposed to work. It will help us work on this issue.

• Every Denote prompt has its own minibuffer history. This way, we know what belongs where.
• Prompts for title and signature will use the elements in their own minibuffer history as possible completion candidates. Users can thus select a previous entry or write a new one. If the history is empty, there is nothing to show and the user must write something new. Users can disable this history-based completion by setting denote-history-completion-in-prompts to nil.
• The prompt for files (e.g. what we get with M-x denote-link) lists the files in the denote-directory. This is not a history: they are the actual files available at that moment.

However, the problem now is that the history doesn’t include my previous notes.

You mean that the denote-file-prompt should use history completion for files you have already created? What would be the difference relative to what it does now, where it shows files in your denote-directory?

Note that this prompt does have a history, which you can navigate with M-p and M-n or even with the consult-history command if you prefer completion.

[yibie]

I'm understand now. It seems that my denote history missing.

My emacs directory was removed and rebuilded. So, may be this can explain why my denote can't found the history file?

[protesilaos]

So, may be this can explain why my denote can't found the history file?

Maybe yes. Though we need to figure out how your history was stored before. If you rebuilded your entire configuration the way you had it before, then you will probably have everything were it was. So it is only a matter of doing what you used to do and the histories will be updated accordingly.

[yibie]

May be there is something brocked denote history. Let me see. It's wired, because once time, I'm cleaned all my config except denote's, the issue presist.

[protesilaos]

Can you evaluate this and see if you see any completion candidates there?

(let ((denote-title-history '("one" "two" "three")))
  (denote-title-prompt))

[yibie]

The result looks like correctly.

[protesilaos]

Fine, so the reading form the history works as expected. When you restart Emacs, what is the value of denote-title-history? If it is nil, then you will not see any completion candidates. This is the expected behaviour. If the history is not empty, then you should see those in the minibuffer prompt.

[yibie]

But numbers of history notes, should be 97, not empty. Why other notes can not include in denote-title-history?

[protesilaos]

From: yibie @.***> Date: Fri, 20 Sep 2024 06:20:28 -0700

But numbers of history notes, should be 97, not empty. Why other nots can not include in denote-title-history?

Other note titles are not included here because we would need to go through all the files in the 'denote-directory' and extract each title. With many files, this is a costly operation that will slow down Emacs.

Also, it is unlikely that a user will want to use the same title many times. Whereas this makes sense for keywords.

-- Protesilaos Stavrou https://protesilaos.com

[yibie]

There's no need to traverse every time. You only need to traverse once at the beginning, and the next time you can just read the cached value in title-history. I've also noticed that notes created under the denote command are saved as well. The issue now is that notes previously created through denote are not included in the title-history cache.

[protesilaos]

There's no need to traverse every time.

Indeed. Still, this is something we need to consider. We would need extra logic to maintain a cache and to update it. Then we need to do the same for signatures. Maybe it is worth it, but it is outside the scope of what we are currently providing.