[plugin] org-roam.nvim progress & discussion

[chipsenkbeil]

About this task

This issue started discussing a potential bug, and ended up evolving into a rundown of the ongoing work on org-roam.nvim, a plugin to implement Org-roam in neovim.

Scroll further down to see details about the plugin, code pointers and highlights, etc.

The original bug report is kept below for clarity.

Describe the bug

I am writing a plugin that creates some buffers with a buftype of nofile. The purpose of these is to generate some immutable orgmode content that you can navigate. In particular, I want to take advantage of the fantastic folding that orgmode offers.

Unfortunately, when I go to collapse a section, I get an error about the current file is not found or not an org file, caused by https://github.com/nvim-orgmode/orgmode/blob/261c987345131a736066c25ea409f4d10904b0af/lua/orgmode/files/init.lua#L106

Steps to reproduce

1. Create a scratch buffer via vim.api.nvim_create_buf(false, true)
2. Set the filetype to org via vim.api.nvim_buf_set_option(bufnr, "filetype", "org")
3. Populate with some org headings via vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, {"* Heading 1-a", "** Heading 2", "* Heading 1-b"})
4. Navigate to the buffer and hit tab to try to collapse or expand a section

Expected behavior

Section is properly folded.

Emacs functionality

No response

Minimal init.lua

-- Import orgmode using the minimal init example

Screenshots and recordings

No response

OS / Distro

MacOS 14.4

Neovim version/commit

0.9.5

Additional context

Using orgmode on master branch at commit 651078a.

[kristijanhusak]

File is considered valid only if it has an org or org_archive extension. Try setting a name for a buffer that has an org extension with nvim_buf_set_name

  local bufnr = vim.api.nvim_create_buf(false, true)
  vim.api.nvim_buf_set_option(bufnr, 'filetype', 'org')
  vim.api.nvim_buf_set_name(bufnr, 'somename.org')
  vim.api.nvim_buf_set_lines(bufnr, 0, -1, true, { '* Heading 1-a', '** Heading 2', '* Heading 1-b' })
  vim.cmd('b'..bufnr)

[chipsenkbeil]

@kristijanhusak that solved the error about not being a current file! I'm hitting an issue where the plugin still doesn't detect a fold. There may be something wrong with my buffer or setup. Applies on any heading.

[kristijanhusak]

What would you expect to be folded here? Did you try doing zx to recalculate folds? I see you are trying to implement org-roam. I had similar idea but didn't catch time to start working on it.

[chipsenkbeil]

The first heading. If I create a file manually and reproduce the contents, I can fold it.

I had to make the repo private while I got permission to work on it on personal time. Now that I have it, here's the current plugin: https://github.com/chipsenkbeil/org-roam.nvim

About the plugin

Database

I wrote a simplistic implementation of a database with indexing supported that provides an easy search for links and backlinks. It isn't as full-fledged as SQL and I'm sure will struggle with larger roam collections, but for me this is enough.

Parser

Leveraging the orgmode treesitter parser for me to find the details I need to build the above database.

Completion of node under cursor

Covers both completing within a link and under a cursor. Essentially does the complete everywhere out of the box. Here's a demo:

https://github.com/nvim-orgmode/orgmode/assets/2481802/0c71540d-fdca-47e3-b0b4-74c61e56ae93

Quickfix jump to backlinks

I like the quickfix list, so while Emacs doesn't have this and uses an org-roam buffer, this was easy to whip up for neovim:

https://github.com/nvim-orgmode/orgmode/assets/2481802/a4ece5f8-c534-4a0e-9e03-f386a501e1ab

Org buffer

I was generating an org file instead of writing a custom buffer. I may switch to the custom buffer highlighting and format anyway because I need to capture link navigation to open in a different buffer versus the current one.

Id navigation

Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently. So I'll need to write an org-roam variant to support opening id links. Shouldn't be hard at all. I also built out id generation for a uuid-v4 format in pure Lua. My test nodes aren't using it, though.

[kristijanhusak]

Thanks for opening it up. I just skimmed through it, and it seems you did a lot of custom stuff. A recent refactor was also done to support org-roam with some of the internals. This is what I had in mind:

1. Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51
2. Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though
3. Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23
4. Use orgmode capture class with custom templates only for org-roam where it would append the properties with id

Doesn't seem to work for me even though I thought it was introduced into the plugin somewhat recently

In the issue description you wrote you are using commit 651078a2fe60b12c93903e3a2b655491c951bf9d. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.

[chipsenkbeil]

Thanks for the pointers! When I looked at the plugin's API and data structures, it was (and I believe still is) missing crucial information I'd need for a fully-functional org-roam implementation. Would it make sense to move this discussion to a separate issue? I know you have the plugin API pinned issue, but there's a lot of different questions and dialog I'd want to have about needs and usage that feels better for a separate issue.

Use orgmode.files to load org-roam specific directories https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/init.lua#L51

Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under cursor, etc.

Wonder how much of that has changed or was misinformed from my first look.

I also didn't check to see how you're loading files. I tried to keep most of my logic async - in the sense of leveraging neovim's scheduler and callbacks - so I can (in the future) easily report loading a large set of files, monitoring file changes and reloading, etc.

Use OrgFile:get_links() to collect all links instead of using a database. I'm not sure if this would work though

I don't know on this one. The database I wrote is a way to both track outgoing links (ones you'd find in a file) and incoming links (aka backlinks into a file). I like the design I've got, so I'll most likely keep this.

Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.

Add a custom source for completion through the orgmode internals like it is done for everything else here https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/autocompletion/init.lua#L23

Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.

Use orgmode capture class with custom templates only for org-roam where it would append the properties with id

I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.

In the issue description you wrote you are using commit https://github.com/nvim-orgmode/orgmode/commit/651078a2fe60b12c93903e3a2b655491c951bf9d. That's a fairly old one, and id was not supported there yet. To generate ids you can use this class https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

Good to know. I've got different versions on different machines. Given I started fiddling with this a couple of months back, I guess the org id was implemented after.

Not sure if this information changes anything for you, but I planned to do this. I generally wouldn't suggest using internal classes, but I planned to do that since everything would be part of the same organization.

I want to reduce the hacks and redundancy where possible. My implementation is meant to build on top of your plugin to supply the org-roam features, but when I started it looked like there was enough missing that I ended up only leveraging the treesitter language tree to get the information I needed.

Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.

[kristijanhusak]

We can either have a separate issue or rename this one.

Haven't looked at this. I wanted to be fairly flexible in the API used to load files, and I also needed some information I didn't see a couple of months back such as top-level drawer access, location information regarding where links/headings/properties are, detection of links/nodes under the cursor, etc.

You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files Files are loaded asynchronously using luv. Most methods in file and headline provide the element content and the node containing location information (range). For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.

Populating the database, on the other hand, could definitely switch from a custom parser to what you've got, but to my understanding your links and other structures do not capture their location within a file, which I need in order to show where backlinks, citations, and unlinked references come from.

This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.

I haven't looked at this yet. I know that Emacs' org-roam implementation needed to explicitly create its own templating system due to some incompatibilities with orgmode's templates; however, I don't know what those are and I would much rather use what you've already built.

I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.

Is this for omnicomplete or other aspects? I built out a selection UI that looks similar to the one I've seen commonly used in Emacs with org-roam. Plan to keep this UI for node completion at point and other selections, but if you have something built in that handles providing omni completion, I'd definitely want to supply ids from my database to it.

Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.

Would be interested in working with you on bridging the gap in functionality and getting this plugin moving forward, unless you were wanting to build your own version of org-roam.

I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.

[chipsenkbeil]

We can either have a separate issue or rename this one.

Renamed this one.

You can check this folder for all the logic around loading files and accessing different elements in the org file https://github.com/nvim-orgmode/orgmode/tree/master/lua/orgmode/files Files are loaded asynchronously using luv. Most methods in file and headline provide the element content and the node containing location information (range). For example, I recently added get_properties() and get_property(name) to be able to get top-level properties from a file. These do not contain any information about the location, but we can extend those as we go.

I'll give these a look in a week or two to see how they operate.

This method just gets all the links, and creates a Link class, from which you can get different parts of the url. It does not contain a location within the file, but we could add it if you think it will be helpful. I didn't look into the details what you have done, so I'll leave this decision to you.

I don't know if it makes sense to add it to that method, but the locations of links within a file are needed for org-roam in order to support listing multiple references to the same backlink and to be able to pull in a sample of the content that linked to a node. I think it's easier to see from this person's tutorial of the Emacs usage: https://youtu.be/AyhPmypHDEw?si=mLGsAdosnKjTZ-1f&t=1690

I think you will be able to use it. You just need to create custom Templates class that gives the list of templates, and you can also create a custom OrgRoamTemplate class while extending Template class here, for stuff like dynamic file name with title, slug and such.

I was really hoping that I could reuse it, so this makes me happy to hear. :) Will be giving that a look as soon as I reach that point.

Yes this is for omnicompletion and completion through cmp, basically any autocompletion. You can see how other builtin sources are added and you can add your own through add_source method.

Got it. Yes, I definitely want to use your code to handle omnicompletion. The selector is more like a built-in telescope interface, which I wanted to have similar to what is shown in the emacs tutorial above when it comes to selecting nodes where you can not only select between choices but also filter the choices further. So having both is my plan.

I wanted to build my own version that would be part of https://github.com/nvim-orgmode organization, but I will not have time to do that any time soon (like 6 months), so it's probably best to delegate this to you since you already did a lot of stuff for it.

I don't really mind where my plugin lives, so if you would be interested in taking this in at some point in the future once we remove as much of the custom logic as makes sense, then I'd be happy to hand it over to you. I need something like this for work, and it didn't exist, which is why I'm building it now.

[kristijanhusak]

I added range property to links in c4eeb3d9caa6403583e6d2285627126d70aef691 that holds the position of the links, including [[ and ]] markers.

We can add a few more things as we go if you find them missing, just let me know.

[chipsenkbeil]

@kristijanhusak are these indexed starting at 0 or 1?

[kristijanhusak]

It's 1.

[chipsenkbeil]

@kristijanhusak I just updated to c4eeb3d and when trying to open an id link using org_open_at_point (bound to a local leader mapping), I'm still getting the issue about "No headline found with id: ...". I'm assuming this is because having a file-level id is unique to org-roam and not orgmode, which I "think" only has the concept of ids in property drawers at the heading level. But I'm not sure if global ids follow that logic or not.

From https://orgmode.org/manual/Handling-Links.html

If the headline has a ‘CUSTOM_ID’ property, store a link to this custom ID. In addition or alternatively, depending on the value of org-id-link-to-org-use-id, create and/or use a globally unique ‘ID’ property for the link 28. So using this command in Org buffers potentially creates two links: a human-readable link from the custom ID, and one that is globally unique and works even if the entry is moved from file to file. The ‘ID’ property can be either a UUID (default) or a timestamp, depending on org-id-method. Later, when inserting the link, you need to decide which one to use.

Given this fact and your desire to maintain only core orgmode functionality within this plugin, I'm assuming that unless the global ID at a file level is part of core orgmode, I will need to write my own open logic for links that navigates to the correct id that supports file-level ids. The thought process is to have that function fall back to your implementation if the link is not an id link. Thoughts?

[chipsenkbeil]

I changed the org-roam buffer to more closely match the org-roam variation. Looks like they changed to use magit underneath, which I'm just replicating in style right now.

Also, progress on following node change under cursor. Don't know the performance considerations of this given I've got tiny tests.

https://github.com/nvim-orgmode/orgmode/assets/2481802/c454bcb6-15a7-41cf-893f-fed9a752d1b0

[kristijanhusak]

Id links work fine for me. There's also a test that confirms that it's working. Note that your files need to be part of org_agenda_files otherwise, it won't be found.

[chipsenkbeil]

@kristijanhusak your test has a property drawer with an id that is within a section with a headline. Does it work with a top-level property drawer?

The case I'm referring to is a top-level property drawer not within a section.

My dotfiles configure every org file (including those of the roam directory) to be within the agenda:

[kristijanhusak]

Ah yes, you are correct, there was no support for top-level id. I added that now on the latest master, let me know if it's not working as expected.

[chipsenkbeil]

@kristijanhusak fantastic! Quick test has it working just fine. :) One less thing I have to manage myself.

[chipsenkbeil]

@kristijanhusak I haven't been able to find it yet. Do you have a version of org-id-get-create (part of org-id.el)?

This is referenced in org-roam's manual and is used to inject an ID into a head as seen in this demo.

[kristijanhusak]

Yes, you can use this https://github.com/nvim-orgmode/orgmode/blob/master/lua/orgmode/org/id.lua

local id = require('orgmode.org.id').new()

There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already.

[chipsenkbeil]

There's also a method on headline classid_get_or_create that adds id property to a headline if there isn't one already.

This is what I'm specifically looking for, which both does the work of generating the id and placing it in the headline. Is there a version that also works with a property drawer at the file level?

When I tested org-id-get-create in Emacs, it works with a property drawer at the file level. So, ideally, this would be a command that someone could run whether they're in a heading or not.

Here's an example:

https://github.com/nvim-orgmode/orgmode/assets/2481802/bf97e15f-2a27-4ed4-bf7b-81f06a9a2366

[kristijanhusak]

We could add that, but I don't think you will need it. For org-roam you will have custom templates that will already populate this information before showing the capture window. When creating a template, just call the orgmode.org.id to generate an id for you.

function OrgRoamTemplate:compile()
   --call parent compile
  local content = OrgTemplate:compile()
  content =  vim.list_extend({
    ':PROPERTIES:',
    ':ID: '..require('orgmode.org.id').new(),
    ':END:'
  }, content)
end

[chipsenkbeil]

@kristijanhusak I'm switching back to having the org-roam buffer be an actual org file w/ syntax. I've seen examples of org-roam that use magit as the interface and others where it has a plain org file.

From the discussions I've seen regarding how the buffer is used, people will either use it to open a backlink/citation/unlinked reference directly in the frame (via RET) or they can force it to open in a different frame (via C-u RET).

I know OrgMappings:open_at_point() exists to open the link or date at the point given. Is there any function that lets you open at point while specifying a different window? Was looking at that function and considering if I need to build a wrapper for that function to be able to point to a different window.

Org-roam manual reference

Link: https://www.orgroam.com/manual.html#Configuring-the-Org_002droam-buffer-display

Crucially, the window is a regular window (not a side-window), and this allows for predictable navigation:

• RET navigates to thing-at-point in the current window, replacing the Org-roam buffer.
• C-u RET navigates to thing-at-point in the other window.

Emacs manual reference

Link: https://orgmode.org/org.html#Handling-Links

There's a minor reference to org-link-frame-setup, which appears to let you configure how files and directories are opened including other frames:

There's a minor reference to org-link-use-indirect-buffer-for-internals, which I opened up below:

Other references

• https://stackoverflow.com/questions/8881649/how-to-force-org-mode-to-open-a-link-in-another-frame

[kristijanhusak]

We could extend it to accept the command that opens the "point", and default it to edit. Would that work for you?

[chipsenkbeil]

We could extend it to accept the command that opens the "point", and default it to edit. Would that work for you?

I think that should work, yeah. As long as there is a way for me to specify the window to use versus the current window.

I think I'd do this by using wincmd:

---@param winnr integer # id of window where we will open buffer
---@return string # to supply to vim.cmd(...)
function make_cmd(winnr)
    local cmd = winnr .. "wincmd w" -- goes to window "winnr"
    return cmd .. " | edit"
end

Update

Leveraging loading a singular orgfile per preview in the org buffer. I was reading through how org-roam does this (via changelog and source), and it looks like it handles certain cases specially such as detecting if the link is in a heading and just showing the heading text or detecting if the link is in a list item and showing the entire list.

I'll be switching this back over to an org buffer soon, but here's a preview:

This is my logic for now, although I'm assuming I should create a new OrgFiles once with the path to the org roam directory, versus creating a new instance each time. So I'll be doing that change. Does loading org files check the modification time or something else to return a cached version? In my old approach, I would stat each file to check its mtime.sec against a cached version.

-- NOTE: Loading a file cannot be done within the results of a stat,
--       so we need to schedule followup work.
vim.schedule(function()
    require("orgmode.files")
        :new({ paths = opts.path })
        :load_file(opts.path)
        :next(function(file)
            ---@cast file OrgFile
            -- Figure out where we are located as there are several situations
            -- where we load content differently to preview:
            --
            -- 1. If we are in a list, we return the entire list (list)
            -- 2. If we are in a heading, we return the heading's text (item)
            -- 3. If we are in a paragraph, we return the entire paragraph (paragraph)
            -- 4. If we are in a drawer, we return the entire drawer (drawer)
            -- 5. If we are in a property drawer, we return the entire drawer (property_drawer)
            -- 5. If we are in a table, we return the entire table (table)
            -- 5. Otherwise, just return the line where we are
            local node = file:get_node_at_cursor({ opts.row, opts.col - 1 })
            local container_types = {
                "paragraph", "list", "item", "table", "drawer", "property_drawer",
            }
            while node and not vim.tbl_contains(container_types, node:type()) do
                node = node:parent()

                -- Check if we're actually in a list item and advance up out of paragraph
                if node:type() == "paragraph" and node:parent():type() == "listitem" then
                    node = node:parent()
                end
            end

            -- Load the text and split it by line
            local text = file:get_node_text(node)
            item.lines = vim.split(text, "\n", { plain = true })
            item.queued = false

            -- Schedule a re-render at this point
            opts.emitter:emit(EVENTS.CACHE_UPDATED)
            return file
        end)
end)

[kristijanhusak]

we could also make the argument a function, that would default to:

local edit_cmd = edit_cmd or function(file) return 'edit '..file end

vim.cmd(edit_cmd(filename))

That might give you more control over it.

I should create a new OrgFiles once with the path to the org roam directory, versus creating a new instance each time. So I'll be doing that change. Does loading org files check the modification time or something else to return a cached version?

Yes, you should go with the single instance. Here I have an Org instance that holds other instances. Regarding caching, I do same for files that are not loaded, and check the buffer changedtick if the file is loaded inside a buffer. You can see the logic here https://github.com/nvim-orgmode/orgmode/blob/b7c42e6fc5982afef5e322b33fe58eec9a09c76d/lua/orgmode/files/file.lua?plain=1#L108-L119

[chipsenkbeil]

we could also make the argument a function, that would default to:

local edit_cmd = edit_cmd or function(file) return 'edit '..file end

vim.cmd(edit_cmd(filename))

That might give you more control over it.

I think either solution should work.

Org roam buffer

I'm still hitting some quirks with creating an org buffer that is of buftype = nofile.

1. As you pointed out earlier, I have to supply .org at the end of the buffer name. It looks like as a result of this, orgmode updates the buffer name to be relative to the orgfiles directory for me. So now if I want to ensure that I'm not within my own buffer, I have to do vim.endswith(vim.api.nvim_buf_get_name(0), "org-roam-buffer.org") instead of equality.
2. It looks like my buffer's preferences to be both unlisted and a scratch buffer get overwritten. Maybe you're swapping out the buffer on me (don't know yet). This means that this temporary buffer shows up in the buffer list.
3. I still can't get folding to work. Running zx doesn't appear to update the folds.
4. Reloading the buffer clears the contents (this is on me to fix) and requires to jump elsewhere to refresh. But the callout is that the syntax highlighting appears to break partially when the buffers contents are reloaded again. The comment and heading colors.

https://github.com/nvim-orgmode/orgmode/assets/2481802/04aa86b3-ed93-42c6-b065-9e8c822d0596

[kristijanhusak]

I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better (vim.fn.tempname()..'org')? I'm not sure if that would work, but if you reuse it and write it each time it's changed, it might work.

[chipsenkbeil]

I would need to investigate how scratch buffer behaves. Could you try using a temp filename and see if it's better (vim.fn.tempname()..'org')? I'm not sure if that would work, but if you reuse it and write it each time it's changed, it might work.

Sure, I can experiment. I think the broader issue is it seems like orgmode really wants org buffers to represent files on disk, while I am making a buffer-only org document with content that I dynamically generate and update on cursor change.

[chipsenkbeil]

@kristijanhusak still no luck, so there must be something else I'm doing wrong. Even persisting the file to disk, the folds aren't working. Everything else is fine.

The logic is in the NodeView Window, where I'm maintaining an OrgFiles cache across all windows and another cache where I've calculated lines to show in a preview per backlink. All of that works fine, so I must have something wrong set up w/ creating the org buffer.

Maybe there's something else wrong in my setup. I have to hack orgmode a bit to get it to work, including a full reload (edit!) for the current buffer when the orgmode plugin first loads.

---

It does look like the ftplugin is triggering for org as I can see b:undo_ftplugin is set. For some reason, the fold levels aren't being detected. Everything has a foldlevel of 0.

Seems like if I change from being nomodifiable and nofile to a regular buffer and modifiable and then write the file, I can get folding. I also need to then reload the file via :e to get folds to fully work as some of them don't fold properly. When I took a look at your autocmds, it looks like you just do require("orgmode"):reload(file), which I tried to replicate with the name of the buffer, but so far I can't get it to work unless I make it modifiable, manually write via :w, and then reload via :e.

[kristijanhusak]

I investigated a bit what's happening, and I think just additionally doing filetype detect should solve an issue for you. Here's a fuction that I used for testing (simplest to be put in init.lua):

_G.test_org_buffer = function()
  local buf = vim.api.nvim_create_buf(false, true)
  vim.api.nvim_buf_set_option(buf, 'filetype', 'org')
  -- You can maybe use some naming convention like this, but not necessary
  vim.api.nvim_buf_set_name(buf, 'org:///org-roam-backlinks.org')
  vim.api.nvim_buf_set_lines(buf, 0, -1, false, { '* level 1', '** level 2', '** level 2.2' })
  vim.cmd('b'..buf)
  vim.cmd('filetype detect')
end

1. Open nvim
2. execute :lua test_org_buffer()

[chipsenkbeil]

@kristijanhusak seems like that works. I have it re-detecting the filetype every time I rewrite the buffer. Gotta figure out the best approach to do this though as the buffer can be visible or invisible, and you may not have it selected, either. I wanted to use vim.filetype.match({ buf = buf }), but it doesn't fix the issue. Only filetype detect seems to work.

Right now, trying to see if there is a way for me to set the fold level explicitly when the buffer is updated. I essentially want foldlevel of 1 such that the previews under the links are hidden by default, but can't figure out a way thus far. Balancing act with the orgmode settings getting applied.

[chipsenkbeil]

@kristijanhusak while filetype detect works, it's too difficult for me to retrigger it when the buffer can get updated elsewhere (e.g. cursor change in another window leads to an update). I'll put this on hold since I can't figure out what exactly about that is causing folding to work.

Will look at tackling the last MVP feature needed which is templating. From there, I can work on leveraging more of this plugin in place of custom code.

Another question for you: are citations supported in parsed files? Orgmode 9.5 added native support for them.

[kristijanhusak]

are citations supported in parsed files?

No, not yet. You need those for org-roam?

[chipsenkbeil]

@kristijanhusak for the minimum viable product? No. For academics that want to use the plugin? Yes.

The org-roam manual references support for citations including orgmode 9.5's built-in variant. For org-roam, they are used through ROAM_REFS, which to my understanding is a way to connect non-id links to nodes; however, orgmode citations only have a singular bracket on either side (youtube example).

[cite:@better_speech]

So alongside your get_links() method, we'd also need a get_citations() method given that they don't match the link syntax of double square brackets.

Technically, org-roam supports org-ref as well, which has the format of just cite:better_speech, but I'm fine just supporting the built-in format.

[chipsenkbeil]

@kristijanhusak took a look at the template class you referenced to extend.

I see that you have a list of expansions that can be applied and that the _compile_expansions method takes an optional list of expansions as a secondary argument (found_expansions), but it looks like that is never used. Ideally, I'd like to provide additional expansions (e.g. the node id, or references to the node under cursor when capturing), but it looks like there's no way to plug in new expansions on top of the existing ones for a template? If this was available, I think I could avoid having to extend an entirely new class in favor of just providing org-roam themed expansions.

https://github.com/nvim-orgmode/orgmode/blob/b7c42e6fc5982afef5e322b33fe58eec9a09c76d/lua/orgmode/capture/template/init.lua#L212-L220

-- Creates a template for use with org-roam nodes
local template = Template:new({
    template = {
        ":PROPERTIES:",
        ":ID: %i",
        ":ORIGIN: %n",
        ":END:"
    },

    -- Provide a mapping of expansions to use that includes default expansions along with new ones
    expansions = vim.tbl_extend("keep", Template.EXPANSIONS, {
        -- Generates an org id
        ["%i"] = function()
            return require('orgmode.org.id').new()
        end,

        -- Yields the id of the node under the cursor during capture
        ["%n"] = function()
            return get_id_of_node_under_cursor()
        end,
    }),
})

It could also be handy to support some pattern matching. Org roam's manual references expansion logic that supports looking up user-defined functions to invoke and use for substitution. For example, ${foo} would get translated into foo(node-under-cursor), org-roam-node-foo(node-under-cursor), or use completion to look up the input for the variable.

So having something like %{foo} would be handy, but it looks like your compilation of expansions limits to escaped magic patterns.

[kristijanhusak]

I guess we could add something like custom_expansions for this purpose, but you could also use "eval" expansion to call custom stuff (%(return require('orgmode.org.id').new())), here's an example test. Note that %i that you put in the example does something else in Orgmode

That pattern matching seems like an org-roam specific functionality. I'd prefer not to add plugin specific logic here. That's why I suggested to extend the class and override/extend methods with org-roam specific stuff.

[chipsenkbeil]

you could also use "eval" expansion to call custom stuff (%(return require('orgmode.org.id').new()))

Oh, I missed that detail. That does what I'd need.

Note that %i that you put in the example does something else in Orgmode

Good to know :) That was just an example, and I hadn't thought of what those would be.

That pattern matching seems like an org-roam specific functionality. I'd prefer not to add plugin specific logic here. That's why I suggested to extend the class and override/extend methods with org-roam specific stuff.

Fair enough. With the eval expansion you mentioned, I can get by. It just means that it'll be more cumbersome for the end user since they'll have to do something like %(return require("org-roam.node.title").get()) or %(return some_user_function(require("org-roam.node").at_cursor())).

[kristijanhusak]

It just means that it'll be more cumbersome for the end user since they'll have to do something like %(return require("org-roam.node.title").get()) or %(return some_user_function(require("org-roam.node").at_cursor())).

Yes, that is an option, but it is cumbersome. My initial plan was to extend the Template class for org-roam and compile these template variables in there. We could refactor a template class to accept custom hooks through something like template:on_compile(fn), and then you will be able to hook into it with your own stuff. Current functions could be also treated like builtin hooks. What do you think?

[chipsenkbeil]

It just means that it'll be more cumbersome for the end user since they'll have to do something like %(return require("org-roam.node.title").get()) or %(return some_user_function(require("org-roam.node").at_cursor())).

Yes, that is an option, but it is cumbersome. My initial plan was to extend the Template class for org-roam and compile these template variables in there. We could refactor a template class to accept custom hooks through something like template:on_compile(fn), and then you will be able to hook into it with your own stuff. Current functions could be also treated like builtin hooks. What do you think?

Yeah, that could work. To confirm, the idea is that passing a function to on_compile would let us modify the content? Would it take the content as input and return the modified version as output?

[kristijanhusak]

Yes, that's the idea. Example:

local template = Template:new({ template = 'This is {title}' })
template:on_compile(function(content)
return content:gsub('{title}', 'org-roam')
end)

local content = template:compile() -- Result: This is org-roam

[kristijanhusak]

Added in cc1c4c265e677b275b1bd14b380803bb1266357b

[chipsenkbeil]

Added in cc1c4c2

Great! I would say a minor nit is it would be nice to have on_compile return the template so you could chain like this:

local content = Template
    :new({ template = 'This is {title}' })
    :on_compile(function(content) return content:gsub('{title}', 'org-roam') end)
    :compile()

[kristijanhusak]

Sure, force pushed in bb89dfc44d889d8ababcb3c67c55c8be92cfc6e0

[chipsenkbeil]

If I haven't said it already, really appreciate how engaged and supportive you've been 😄

The plugin is shaping up, and I'll be tackling the templating over the next few days (maybe it'll be quick).

From there, I'll be working to flesh out testing a little more at an integration level and remove large portions of custom code (e.g. the custom parser) in favor of the core plugin. It's taking shape, which is exciting to see as I'm badly in need of org-roam capabilities in my day-to-day.

[kristijanhusak]

Thank you for the kind words :) I'm glad someone took the initiative to build this. It is not a small project to handle, but I know a lot of people want to use it. I don't need roam functionalities that often, but I had a few situations where it would be helpful.

As you go let me know if there's something else that might be helpful and we can discuss.

[chipsenkbeil]

@kristijanhusak I'm just about done with the initial version of the org-roam buffer. I ended up going back to not using an org syntax and instead having a custom buffer (video below) due to technical challenges I couldn't figure out. I like how it works now - I even have a nifty way to select opening backlinks in other windows.

The one thing I don't support is syntax highlighting when previewing the contents from a backlink. I'm putting that as a "nice to have" but was wondering if you had any thoughts on a way to perform syntax highlighting (and link concealing) for a range of text within a non-org buffer that is supposed to represent org syntax.

https://github.com/nvim-orgmode/orgmode/assets/2481802/d8c2ff4f-3a6b-4b3d-b924-ae547b21a98d

[chipsenkbeil]

@kristijanhusak also, one issue I'm facing that is pretty challenging is that OrgRange does not including the starting and ending offset, only the starting and ending line and column positions.

I use the offset in quite a few different locations throughout the org-roam plugin and it would be pretty difficult to update the logic to accept only line/column information. Is there any way I can re-calculate the offsets from OrgRange? Or better yet, have OrgRange include the offset positions?

Right now, I've written this to see if it would work:

---Converts from an nvim-orgmode OrgFile and OrgRange into an org-roam Range.
---@param file OrgFile #contains lines which we use to reconstruct offsets
---@param range OrgRange #one-indexed row and column data
---@return org-roam.core.parser.Range
function M:from_org_file_and_range(file, range)
    local start = {
        row = range.start_line - 1,
        column = range.start_col,
        offset = range.start_col - 1,
    }

    local end_ = {
        row = range.end_line - 1,
        column = range.end_col,
        offset = range.end_col - 1,
    }

    -- Reverse engineer the starting offset by adding
    -- the length of each line + a newline character
    -- up until the line we are on
    for i = 1, range.start_line - 1 do
        local line = file.lines[i]
        if line then
            start.offset = start.offset + string.len(line) + 1
        end
    end

    -- Reverse engineer the ending offset by adding
    -- the length of each line + a newline character
    -- up until the line we are on
    for i = 1, range.end_line - 1 do
        local line = file.lines[i]
        if line then
            end_.offset = end_.offset + string.len(line) + 1
        end
    end

    return M:new(start, end_)
end

[kristijanhusak]

I'm putting that as a "nice to have" but was wondering if you had any thoughts on a way to perform syntax highlighting (and link concealing) for a range of text within a non-org buffer that is supposed to represent org syntax.

AFAIK you could do that only if you used a tree-sitter highlighting for your org-roam buffer and inject org highlighting through treesitter injections. What technical challenges do you have with using an org filetype in the org-roam buffer? I know there were some issues with filetype detection but I thought we managed to solve those. Also, you mentioned that you have a bunch of buffers to manage and filetype detect needs to be run on the buffer. Did you consider using https://neovim.io/doc/user/api.html#nvim_buf_call() for those types of calls?

I use the offset in quite a few different locations throughout the org-roam plugin and it would be pretty difficult to update the logic to accept only line/column information

From the code, it seems that you need offset from the start of range. I'm curious why you need that? I understand it's probably hard to explain it concisely but please give it a try.

[chipsenkbeil]

AFAIK you could do that only if you used a tree-sitter highlighting for your org-roam buffer and inject org highlighting through treesitter injections. What technical challenges do you have with using an org filetype in the org-roam buffer? I know there were some issues with filetype detection but I thought we managed to solve those.

It seemed like using filetype detect worked, but I needed to continually invoke it whenever the buffer was updated, which happens when you move your cursor to a different node.

I had a lot of hack logic to try to get filetype detect to work in different scenarios, and I finally ran into an issue of switching windows to the buffer in order to use filetype detect triggering autocmds that in turn re-ran the buffer render and causing loops. There was also an issue with some options getting overridden, but I can't remember which ones in my pursuit to get the org buffer to work.

Also, you mentioned that you have a bunch of buffers to manage and filetype detect needs to be run on the buffer. Did you consider using https://neovim.io/doc/user/api.html#nvim_buf_call() for those types of calls?

I honestly forgot about nvim_buf_call and wonder if it would have helped avoid triggering autocmds when it switches to a window containing the buffer. I think I'm going to leave this as-is for now, but in the future could have a configuration option to let people choose what type of buffer they want.

A nice bonus of the new buffer is that I can lazily look up previews from files instead of pulling them all at once to fill in the org buffer. And I can control the navigation a bit more as the current buffer lets you press <Enter> on a link to open it in a different window only (versus replacing the org-roam buffer). I also have control to jump to not only the line but also the column position of the link versus only being able to have links to lines in org.

If/when I use org for an org-roam buffer, I'd need the feature we talked about in terms of having opening link under cursor support a parameter that lets me replace edit with something like 1234wincmd w | edit.

I use the offset in quite a few different locations throughout the org-roam plugin and it would be pretty difficult to update the logic to accept only line/column information

From the code, it seems that you need offset from the start of range. I'm curious why you need that? I understand it's probably hard to explain it concisely but please give it a try.

There are a few reasons why I need an offset, some of which can be switched over to line and column:

1. I use it for interval trees. I wrote a tree.lua to support fast detection of data structures under cursor. a. I use this to figure out the node under cursor by mapping a section's starting and ending offset into the tree with the node's data as the value. b. I use this to quickly figure out which node each link within a file belongs to by putting nodes into the tree and seeing which node is lowest on the tree for a link's offset. c. I use this to determine if the cursor is over a link.
2. I use it for ordering of sections, links, and more, which could be switched to line and column position.
3. I use it to calculate the range for a section that includes a heading and other elements, which could probably be switched once I use OrgFile in place and load up headings, but I don't know. I just need section ranges.
4. I use it for a lot of my parser logic, which will go away if I can fully switch to OrgFile.
5. I use it to get the string content of something (link, section, tags, etc) so I can hash them, but I don't use this feature yet so not important.

I think interval tree usage is the biggest blocker.

[kristijanhusak]

There are a few reasons why I need an offset, some of which can be switched over to line and column

This seems a lot more complicated than I assumed at first.

I wouldn't calculate offset anything better than you do, considering that it's usually created from TSNode, which only contains the range information. Since I don't always have access to a file and its lines when creating OrgRange, I think you will have to stick with your version.