Trigger org database update at runtime

seflue commented 1 month ago

Does this feature exist in Emacs orgmode core?

N/A

Orgmode link

No response

Feature value

Current Limitation

I frequently create new org files during a Neovim session (typically using org-roam). However, since org files are parsed at plugin startup, these new files are not reflected in operations like refiling or searching for headlines. This creates a disconnect between the actual file system and the plugin's internal database.

Desired Functionality

I'm looking for a way to update the orgmode database at runtime. If this functionality doesn't exist, I believe it would be a valuable addition to the plugin. Here are my thoughts on potential implementation:

Is there an existing method to load new files into the database during a session? If not, could we implement one?
It would be useful to have a way to add the current file to the database (if not already present), possibly through a keybinding.
This functionality could potentially be utilized through an API method by plugins like org-roam to automatically add files to the database upon creation.

Benefits

Improved workflow for users who frequently create new org files during a session.
More consistent and up-to-date results for searches and other org-mode operations.
Better integration with plugins that dynamically create org files.

Additional context

Questions and Implementation

Is there any existing functionality that partially addresses this need?
How should we handle files that are already in the database?
Would a full database refresh option be sufficient, despite the fact that it is not very performant, or would it be worth to allow adding single files?
What would be the best entry points in the codebase to implement this feature?

I'm willing to implement this feature myself if it's deemed valuable. @kristijanhusak Could you point me towards the relevant parts of the codebase and also sketch out, how you would like to see such a feature implemented? Where I should focus my efforts?

@chipsenkbeil As far as I understood it, you are updating the org-roam node database on creation of new nodes. Is there something, we can learn from your implementation to consider within this feature? And would it be interesting for you to also update the orgmode database on node creation in org-roam?

I'm open to discussing this feature further and can provide additional context if needed. Thanks for considering this enhancement.

kristijanhusak commented 1 month ago

@seflue you can reload all files from the org_agenda_files with this:

require('orgmode').files:load(true)

Without the true it will load only if things are not currently loaded. true is basically a force load.

I'm not sure if this solves your issue with org-roam, but this reloads the agenda files. You can achieve the same thing if you open up agenda and do R to reload.

seflue commented 1 month ago

Thanks, this really helps. I actually had the false belief, that the load-function only affects already loaded files, not new files. I think, we can close this issue.

seflue commented 1 month ago

I have just found notes and an old branch of an earlier attempt from a few months ago to solve the problem I mentioned - and now I also know where I failed:

There is a function files:load(filename), which I thought would do exactly what I proposed in this ticket: specifically load an agenda file, even if the file is not in the database yet.

However, this did not work for me, new files were not loaded. @kristijanhusak Did I misunderstand something in the code?

kristijanhusak commented 1 month ago

@seflue load method loads all files, and only argument a boolean force as explained in previous comment.

You have 2 ways to load a single file:

files:load_file(filename) - This will just load the file, but it will not appear in the agenda
files:add_to_paths(filename) - This loads the file and adds it to the paths, which means it should appear in the agenda, at least for the current session.

seflue commented 1 month ago

So if I understand you correctly, files:add_to_paths(filename) is actually doing something similar like Emacs org modes org_agenda_file_to_front?

Emacs Orgmode actually uses a separate buffer where the user can add files, which are used in the agenda. Are you open to add a similar functionality to nvim-orgmode? I would actually find something like this useful, because it would allow to customize the set of agenda files, depending of the current working context. I have a lot of org files and I often really miss a functionality to reduce the noise in the agenda.

kristijanhusak commented 1 month ago

I don't know how org-agenda-file-to-front works, but from what I googled, yes. I'm just not sure if Emacs really updates the configuration file to update the paths, or it's only for the session.

Emacs Orgmode actually uses a separate buffer where the user can add files, which are used in the agenda

How does this exactly work? Is it like a list of files in the file manager, a picker like Telescope, or something else?

seflue commented 1 month ago

It's like a list of files, I think, it supports globbing and it actually works more like the harpoon file from ThePrimagen/harpoon, but addressing org files, not buffers. As I understood it, it is initialized from the configuration and updates work only for the session, but I actually need to do more research on that. If we would have this kind of a mechanism, it would be easy to support a plugin, which allows to handle context based org sessions.

chipsenkbeil commented 1 month ago

@seflue for org roam, there are two functions within loader.lua that read in org files and insert/modify/remove nodes.

At work, so don't have time to read everything right now. :) These functions are supposed to support forcing reload of org files, so if something's not working like expected, we may need to look into it.

kristijanhusak commented 1 month ago

I'm ok with adding some options to add/remove files from the session. Please do some research how it exactly works and we can go from there. I'm most curious about one thing:

If the file is removed from the session, is there a way to get it back with some force refresh (like refreshing the agenda), or it needs to be put back manually? Of course, this is just for the session. Restarting the session would load the file again.

seflue commented 1 month ago

@kristijanhusak: I will have a look into it and open a new issue as soon as I have a clearer picture.

@chipsenkbeil: When creating a new node, it would be great, if we could also refresh the orgmode database. Currently I have the issue, when I create a new node with org-roam, the new file is not listed, when I search for it using Telescope-orgmode. If I refresh the orgmode database like @kristijanhusak proposed, it actually gets listed.

chipsenkbeil commented 1 month ago

@kristijanhusak: I will have a look into it and open a new issue as soon as I have a clearer picture.

@chipsenkbeil: When creating a new node, it would be great, if we could also refresh the orgmode database. Currently I have the issue, when I create a new node with org-roam, the new file is not listed, when I search for it using Telescope-orgmode. If I refresh the orgmode database like @kristijanhusak proposed, it actually gets listed.

Happy to take a PR for this. :smile: The org file loading logic is some of the more complex in the roam plugin.

seflue commented 1 month ago

@kristijanhusak: I will have a look into it and open a new issue as soon as I have a clearer picture. @chipsenkbeil: When creating a new node, it would be great, if we could also refresh the orgmode database. Currently I have the issue, when I create a new node with org-roam, the new file is not listed, when I search for it using Telescope-orgmode. If I refresh the orgmode database like @kristijanhusak proposed, it actually gets listed.

Happy to take a PR for this. 😄 The org file loading logic is some of the more complex in the roam plugin.

Done. See PR #64.

nvim-orgmode / orgmode