[Bug] Cannot find entry with ID after creation with org-roam template in latest org-roam

[wakatara]

I am trying to use roam-md to handle my blog entries (which are in hugo and thus markdown - I prefer not to use the ox-hugo approach).

Despite creating md files via the org-roam template approach, org-roam refuses to see them or navigate to them when clicking on links.

I am using the latest roam and have installed roam-md via the straight package manager (so pulls from master, not releases afiak).

Relevant config

(use-package md-roam
    :straight (:host github :repo "nobiot/md-roam"))
(require 'md-roam)

;; ========== Org-roam config ==========
(use-package org-roam
    :straight t (org-roam :type git :host github :repo "org-roam/org-roam")
    :init
  (setq org-roam-v2-ack t)
    (setq org-roam-mode-sections
            (list #'org-roam-backlinks-section
             ;; #'org-roam-reflinks-section
                        #'org-roam-unlinked-references-section
                        ))
    :custom
    (org-roam-directory (file-truename "~/Documents/org/"))
    (org-roam-file-extensions '("org" "md"))
    (md-roam-file-extension "md")
    (org-roam-dailies-directory "logs")
    (org-roam-dailies-capture-templates
        ...

I am triggering md-roam before the db-autosync and cano confirm that md-roam-mode is on as far as emacs is concerned.

  )
(md-roam-mode 1)
(org-roam-db-autosync-mode)

And I am using the following template to try to generate the skeleton of a blog post:

("b" "Blog" plain ""
      :target (file+head "~/Documents/org/areas/blog/%<%Y-%m-%d>-${slug}.md"
"---
id:          ${slug}
layout:      post
title:       \"${title}\"
permalink:   \"/${slug}\"
date:        %<%Y-%m-%d>-Thh:mm:ss+07:00
location:    Singapore
tags:        []
description: 'Clever description of contents.'
draft:       true
---

"):unnarrowed t)))

So far, so good. The file gets generated where I want it on closing the capture. slug in this case is id: emacs-gtd-and-crm-flow-redux.

However, when I then try to either find the file with an org-roam node search or attempt to link through from the originating file to the new markdown file, I get a "Cannot find entry with ID: ..." error and in fact, Roam seems not to know about the file at all. is there something special I need to do with the new org-roam id style or similar? Kinda stumped as it should seem like everything would be working.

Querying the SQLite dtabase directly - I do not see the file being created at all so wondering if the latest org-roam has broken roam-md or I've done something horribly wrong with my config.

[nobiot]

Md-roam seems to work on my end with your template:

Did you save the buffer? It sounds simple but a common cause of the behaviour your report.

I just skimmed your config excerpt but nothing pops up to me.

[nobiot]

My org-roam is org-roam-20221231.2122 -- latest, I believe.

[wakatara]

I changed the config just to now to have org-roam pull from the main package via straight for org-roam, nuked the SQLite DB, and rebuilt all packages from scratch. I'm still seeing the same error (and I did force save the file to make sure it created.). Still no joy.

Wrote /Users/daryl/Documents/org/areas/blog/2023-01-14-emacs-gtd-and-crm-flow-revisited.md

Emacs still gives a org-roam-id-open: Cannot find entry with ID "emacs-gtd-and-crm-flow-revisited"

And checking SQLite there's no record of the file and id in the database when I query it. Hmmmm...

md-roam-mode is on, but it does not seem to be "seeing" markdown files for whatever reason or is simply not sving the index to the SQLite database on save.

Bit stumped. Any other things I can do here to run down the problem? (verbose mode or something? Seeing if the variables are set for it properly - wondering if it's the straight config vs more classic config you have.). Pretty stumped, tbh.

[wakatara]

nb: I also changed the config to see if that was the problem to something a bit more in line with the one on the site:

(use-package md-roam
    :straight (:host github :repo "nobiot/md-roam"))
;; (require 'md-roam)

;; ========== Org-roam config ==========
(use-package org-roam
  ;; :straight t (org-roam :type git :host github :repo "org-roam/org-roam")
    :straight t
    :init
    (setq org-roam-v2-ack t)
    (setq org-roam-file-extensions '("org" "md"))
    (setq md-roam-file-extension "md")
    (setq org-roam-directory (file-truename "~/Documents/org/"))
    (setq org-roam-dailies-directory "logs")
    (setq org-roam-mode-sections
            (list #'org-roam-backlinks-section
             ;; #'org-roam-reflinks-section
                        #'org-roam-unlinked-references-section
                        ))
    :custom
    (org-roam-dailies-capture-templates
     ... 

No dice.

And also, changed the directory to the true name root directory in case the issue was I was mkaing the file in a subdirectory. That didn' t help either... 🤔

[nobiot]

I will look a bit further with emacs -Q and see if there is any part of my config doing something special later -- a bit tied now.

org-roam-id-open: Cannot find entry with ID "emacs-gtd-and-crm-flow-revisited"

This error is a bit strange. I put a edebug point to this function org-roam-id-open. But it does not get used on my end; edebugger didn't start.

[nobiot]

By the way, why do you put all the config in the init part of use-package? I don’t use use-package, but I just wonder why you don’t use config part.

Perhaps you can try vanilla way to config Md and Org-roam without use-package?

For more investigation, I would use edug-defun on a couplenof db update related functions in md and org-roam and see if they are even triggered on save. You can also check value of after-save-buffer hook locally in your markdown files.

[wakatara]

org-roam-id-open: Cannot find entry with ID "emacs-gtd-and-crm-flow-revisited"

This error is a bit strange. I put a edebug point to this function org-roam-id-open. But it does not get used on my end; edebugger didn't start.

I should have been a little clearer. The link gets created as well as the file when I C-c n i to create the link. Since I can't find it via org-roam, I go to the link I created to it in the page and hit return. This normally opens the link for me as a separate buffer. That is the error that occurs when I click through. The error is what made me crack open the SQLite database and look to see if an entry for the saved file was being created (sadly, no... ).

So, what i am trying to figure out, since md-roam-mode is active, and org-roam creates the template is if the variables for md-roam are being set since perhaps it's not picking up that md files are supposed to be saved and linked. It's a bit mysterious.

The only other thing I can think of (besides I'm using emacs 29.0.60) is that the 'true name' directory is a symlink to my iCloud, but since that does not affect org-roam itself when creating or saving files, I'm assuming it has no bearing on md-roam.

Rest well. Kinda interested to figure out what it could be now. Scratching me head here in TH. =]

[wakatara]

By the way, why do you put all the config in the init part of use-package? I don’t use use-package, but I just wonder why you don’t use config part.

I only moved it into init in order to see if perhaps it made a difference over the :custom block, since I have to admit I find some of the directives in use-package and straight somewhat similar (and thus, confusing). Only reason.

[nobiot]

Hi @wakatara ,

Tried with my config directory completely removed. It works. My Emacs version is also 29.0.60. I would have to guess the cause of your issue is somewhere in your config but I can't see where -- You might like to remove the use of use-package?

My script for this is at the bottom.

For this screen shot:

1. Created another markdown file via org-roam-node-find
2. In the second file, added a wiki link to the first one via org-roam-node-insert
3. Back to the first file, show the backlink buffer via org-roam-buffer-toggle.

The buffer above of the screen is the first file, the buffer below is the backlink buffer.

My Emacs version:

GNU Emacs 29.0.60 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.24.33, cairo version 1.16.0) of 2022-12-22

Script and configuration for this test:

(load-library "package")
(add-to-list 'package-archives (cons "melpa" "https://melpa.org/packages/"))
(package-refresh-contents)
(package-install 'org-roam)
(package-install 'markdown-mode)

(add-to-list 'load-path "~/src/md-roam")
(load-library "md-roam")

(setq org-roam-directory "/home/nobiot/org-roam/")
(setq org-roam-file-extensions '("org" "md")) ; enable Org-roam for a markdown extension

(md-roam-mode 1) ; md-roam-mode must be active before org-roam-db-sync
(setq md-roam-file-extension "md") ; default "md". Specify an extension such as "markdown"
(org-roam-db-autosync-mode 1) ; autosync-mode triggers db-sync. md-roam-mode must be already active

(define-key global-map (kbd "C-c n /") #'org-roam-node-find)
(define-key global-map (kbd "C-c n c") #'org-roam-capture)
(define-key global-map (kbd "C-c n I") #'org-roam-node-insert)
(define-key global-map (kbd "C-c n l") #'org-roam-buffer-toggle)

(add-to-list 'org-roam-capture-templates
         '("b" "Blog" plain ""
          :target (file+head "~/org-roam/%<%Y-%m-%d>-${slug}.md"
                 "---
id:          ${slug}
layout:      post
title:       \"${title}\"
permalink:   \"/${slug}\"
date:        %<%Y-%m-%d>-Thh:mm:ss+07:00
location:    Singapore
tags:        []
description: 'Clever description of contents.'
draft:       true
---

"):unnarrowed t))

(define-key minibuffer-local-completion-map (kbd "SPC") 'self-insert-command)

(markdown-toggle-wiki-links)

[nobiot]

sqlite-mode-open-file shows ~/.emacs.d/org-roam.db has stored the nodes as expected:

[wakatara]

Interestingly, I tried to emulate your init-ing of md-roam somewhat while still keeping straight for org-roam:

;; (use-package md-roam
;;      :straight (:host github :repo "nobiot/md-roam"))
(add-to-list  'load-path "/Users/daryl/.config/emacs/md-roam")
(require 'md-roam)

(setq md-roam-file-extension "md")

;; ========== Org-roam config ==========
(use-package org-roam
  ;; :straight t (org-roam :type git :host github :repo "org-roam/org-roam")
  :straight t
    :init

And received this error launching:

⛔ Warning (initialization): An error occurred while loading ‘/Users/daryl/.config/emacs/init.el’:
File is missing: Cannot open load file, No such file or directory, org-roam

Butttttt... org-roam works even when I see this error, which would seem to indicate an issue with the md-roam in terms of it not being able to "see" org-roam if it is loaded via package manager (and surprisingly, even via straight when I pull it via git which is a bit of a shocker.). This occurs using load-library or require.

I use straight pretty seamlessly with all other packages in my config (which is far from trivial) and this is the first time I've run into an issue with it.

Perhaps there is something you're doing in terms of your looking for org-roam or how you hook into it? Lemme know how I can debug further to help narrow this down.

[nobiot]

It seems to be a loading sequence issue in your config.

I think you would need to move (require 'md-roam) somewhere inside (use-package org-roam) form. Md-roam does (require 'org-roam). I think it’s too early in your config, as Emacs does not know about org-roam just yet.

[wakatara]

OK, I switched back to the straight config and am putting all md-roam directives after the org-roam blocks. I have org-roam picking up both org and md files as being "org-roam" via (org-roam-file-extensions '("org" "md")) in the :custom config block in org-mode.

;; End of org-roam config block

;; Loading md-roam after org-roam has been loaded.
(use-package md-roam
  :straight (:host github :repo "nobiot/md-roam"))
(setq md-roam-file-extension "md")
(md-roam-mode 1)

(org-roam-db-autosync-mode)

md-roam mode is on and active. I can see that. But I am still getting the same error and nothing is being added to the DB.

I've even disabled org as the file exension for org-roam and only made it md and I'm still getting the same problem - no id being saved and no id entry for the markdown file in the DB. When the md file was saving though I did get this rather strange message:

Bookmark set error: (error: "Invalid face" bookmark-menu-heading)

(which may somehow be related to using nano emacs in some way since I'd seen a similar error message with ivy) ¯\(ツ)/¯

Any ideas? md-mode is definitely on but I'm a little stumped how to get down to this further. (and btw, thanks for all your help. I really do want to use this package to have org and md live side by side, especially for blogging, but am honestly scratching my head here at what could be wrong.).

[nobiot]

I don't believe Bookmark set error is related to Org-roam or Md-roam.

I've gone back to the vanilla Emacs by deleting all the config, installed use-package, org-roam, markdown-mode; I have my md-roam in my src directory (which should be equivalent to cloning it to your local).

I have done the following after casually reading use-package README -- I don't use it, but it should be conceptually correct. See the use-package org-roam part. It works on my end.

Md-roam does not work on its own, so I think it would make sense to wrap it within org-roam config like I do below. You just need to supply load-path to md-roam, because it's not automatically added to it as it is local.

;; (load-library "package")
;; (add-to-list 'package-archives (cons "melpa" "https://melpa.org/packages/"))
;; (package-refresh-contents)
;; (package-install 'use-package)
;; (package-install 'org-roam)
;; (package-install 'markdown-mode)

(define-key minibuffer-local-completion-map (kbd "SPC") 'self-insert-command)

(use-package markdown-mode
  :config
  (markdown-toggle-wiki-links))

(use-package org-roam
  :bind (("C-c n /" . #'org-roam-node-find)
         ("C-c n c" . #'org-roam-capture)
         ("C-c n I" . #'org-roam-node-insert)
         ("C-c n l" . #'org-roam-buffer-toggle))
  :config
  (use-package md-roam
    :load-path "~/src/md-roam")
  (setq org-roam-directory "/home/nobiot/org-roam/")
  (setq org-roam-file-extensions '("org" "md")) ; enable Org-roam for a markdown extension
  (md-roam-mode 1) ; md-roam-mode must be active before org-roam-db-sync
  (setq md-roam-file-extension "md") ; default "md". Specify an extension such as "markdown"
  (org-roam-db-autosync-mode 1) ; autosync-mode triggers db-sync. md-roam-mode must be already active
  (add-to-list 'org-roam-capture-templates
           '("b" "Blog" plain ""
         :target (file+head "~/org-roam/%<%Y-%m-%d>-${slug}.md"
                    "---
id:          ${slug}
layout:      post
title:       \"${title}\"
permalink:   \"/${slug}\"
date:        %<%Y-%m-%d>-Thh:mm:ss+07:00
location:    Singapore
tags:        []
description: 'Clever description of contents.'
draft:       true
---

"):unnarrowed t)))

Use of straight complicates the things for me (I don't use it either). But if all it does it to clone the repo in a declarative language, I believe the following should work:

(use-package org-roam
  :bind (("C-c n /" . #'org-roam-node-find)
         ("C-c n c" . #'org-roam-capture)
         ("C-c n I" . #'org-roam-node-insert)
         ("C-c n l" . #'org-roam-buffer-toggle))
  :config  
  (use-package md-roam
    :straight (:host github :repo "nobiot/md-roam")
    :load-path "~/src/md-roam") ;; maybe not needed if straight does it for you    
  (setq org-roam-directory "/home/nobiot/org-roam/")
... [Other configuration as the example above follows] ...

[wakatara]

Wow... I really appreciate you trying to get it working with use-package if you don't even use it. Thank you. (I have to admit I thought everyone used use-package in some form... it's quite common across the 'macsVerse.).

use-package's (and straight's) directives are not so tricky, I believe the issue here is in the load order of what needs firing when for md-roam to work.

:init stuff to do before loading the package :config stuff to do after loading the package :custom variables to set for the package (so like setq) :bind gives you keybindings for the package

There is also an :after for loading things in sequence. I used this with md-roam and made some progress! Woo! \0/ At least, I got the ids to show up in the links when org-roam-insert link (C-c n i) and they are correct.

Howver, at the moment, for reasons I'm not quite understanding, the file links don't get written to the SQLite DB. Anyhow, much closer with the following config. (id's written properly from created markdwon files into links... ).

... org-roam config up here

(use-package md-roam
  :straight (:host github :repo "nobiot/md-roam")
  :after org-roam     ;; this appears to be the key to making things work... =] 
  :custom
  (md-roam-file-extension "md")
  )

(md-roam-mode 1)
(org-roam-db-autosync-mode)

[wakatara]

So, the good news is that when I load up the database again, because the md files now have the proper id they get added to the database. However, I am still not able to get them to save into the DB when I save them explicitly or Capture the template.

I've tried variations of adding the load path and requite etc, though unsure what is not working that it does not write to DB.

This is the config I have working at least taking the ids and making sure they are correct for the page:

;; org-roam config via straight up here

(use-package md-roam
  :straight (:host github :repo "nobiot/md-roam")
  :after org-roam
  ;; :load-path "/Users/daryl/.config/emacs/straight/repos/md-roam" made no difference
  :config
  (setq org-roam-directory (file-truename "~/Documents/org/"))
  (setq org-roam-file-extensions '("org" "md"))
  (md-roam-mode 1)
  (setq md-roam-file-extension "md")
  )

(org-roam-db-autosync-mode)

nb: One thing that I noticed is that the org-roam-directory (file-truename "~/Documents/org/")) needs to be set in org-roam or you get an error, so I set it there in custom: and and then reset it via setq in the md-roam config.

Anyhow, any other insights you have would be great. It's definitely a load order issue by the looks of things. I might try your put all of md-roam in :config approach tomorrow, but need to give this a rest for now.

thanks again for your help.

[nobiot]

so I set it there in custom: and and then reset it via setq in the md-roam config.

I don't think your resetting is necessary. md-roam-mode just adds some advice around org-roam-db to extend its support to markdown files. org-roam-directory is common.

Once you see the database is being updated for files, I think configuration for md-roam should be correct. I would look elsewhere for the reason why "the file links don't get written to the SQLite DB".

Check the value of variable after-save-hook locally in your markdown files. I suspect for some reason it is missing org-roam-db-autosync--try-update-on-save-h, which is responsible for triggering db sync when you save your node buffer (markdown and org buffer within org-roam-directory). You may be affected by the same issue as reported in Org-roam repo, if you use symlink to your blog directory.

On a sidenote, and a tongue-in-cheek comment (;-). Evidently, it's hard to defend this for our case:

use-package's (and straight's) directives are not so tricky

All you need to do to set up md-roam to work with org-roam is the following. I don't know why it seems so hard to do it with use-package and straight...

1. Set load-path for org-roam and md-roam
2. Load org-roam and md-roam -- e.g. require or load-library
3. Set some user options -- setq, custom-set-variables, etc.
4. Activate md-roam-mode first; and then org-roam-db-autosync-mode

This is straightforward.

[nobiot]

Another thing you could try is, when save-buffer does not add the file or links in it to the database, to run org-roam-db-sync. save-buffer can behave differently from org-roam-db-sync.

This could narrow down the possible causes.

[nobiot]

@wakatara I have a feeling that you have resolved the issue. Would you mind sharing your solution?

[wakatara]

Unfortunately, I did not (pursued whether it just made sense to write the blog posts in org mode after spending an evening on md-roam. =] ).

Also, I am not sure whether you changed something recently, but while I can get the id to show up in the links now when I do an org-roam-find-or-create-node so that the id gets written into the link, even on restarting emacs now, the MD files do not get added to the org-roam database. 😢 Mind you, org mode files do not immediately get added to the database if they are in a subdirectory, so I think there is a general bug in org-roam. But when I restart emacs the files are read and indexed. For some reason, md files are not now. 🤔

Here's my config. I'd love to get this working, but it's a little mystifying at this point what's wrong.

(use-package org-roam
    :straight t
    :init
    (setq org-roam-directory (file-truename "~/Documents/org/"))
    (setq org-roam-file-extensions '("org" "md"))
    (setq org-roam-dailies-directory "logs")
    (setq org-roam-mode-sections
          (list #'org-roam-backlinks-section
           ;; #'org-roam-reflinks-section
           #'org-roam-unlinked-references-section
           ))
    :bind (("C-c n l" . org-roam-buffer-toggle)
         ("C-c n f" . org-roam-node-find)
         ("C-c n i" . org-roam-node-insert))
     :config
     (org-roam-setup)
     )

(use-package md-roam
  :straight (:host github :repo "nobiot/md-roam")
  :after org-roam
  :config
  (md-roam-mode 1)
  (setq md-roam-file-extension "md")
  )
(load-file "~/.config/emacs/org-roam-capture-templates.el")  ;; loads org roam templates and the md template via setq
(org-roam-db-autosync-mode)

[nobiot]

I see. Then I can only suggest that you try without your configuration (or emacs -Q). I tried it and cannot reproduce the same situation, so I am afraid this is as far as I can come at this stage.

[wakatara]

Totally understand. And thanks for the time you have spent trying to resolve this with me. I appreciate it.

I plan to zero myconfig next weekend and introduce org, org-roam, and md-roam in as base from expand from there and see if I can make it work. I can't tell if it's something to do with subdirectories or some other package, but it is kinda weird. Right now, I'm working around it by creating the md files and (since they get picked up by Deft and Xeft) just grabbing them that way. But it is pretty weird and curious.

(honestly, I'm worried it's an ARM64 OSX or weird iCloud bug thing cuz I'm stumped what could be causing it in my config. It'll turn out to be something weird like an errant ' in a templte or something 🤣 )

[nobiot]

What was the cause of the problem? How did you fix it?

[wakatara]

I have not found a solution yet, and downgrading and rebuilding my init.el from scratch did not solve it. I have a feeling it has something to do with osx, icloud or subdirectories but is not specifically related to md-roam.