Workspace-wide queries for tags, todos, etc.

[sofubi]

Issues

• [X] I have checked existing issues and there are no existing ones with the same request.

Feature description

As I grow my notes workspace over time I'm finding that I'd like to be able to have some queries on my main index.norg for easy access to things like tags and todos.

I've looked through the open issues as well as the roadmap and haven't found much in the way of this so I'd like to propose a feature. Maybe something like the following example:

@query
select name from tags
for each tag from tags
- {tag}
end
@end

The above code would render a list of folded headings that each contain a list of links to files tagged:

* Tags
** sql
  - {$:link_to_file.norg:}[file]
** lua
  - {$:link_to_other_file.norg:}[other_file]

I realize that as a flat file structure this really necessitates something like SQLite or otherwise. I saw that there was something on the roadmap concerning db interfaces and Neorg so maybe this is part of that? It's always possible too to create a SQLite db that keeps track of only metadata objects and is portable, ideally maintaining flat files while gaining search/query functionality.

Inspiration for this mainly comes from Obsidian queries which I've used in the past and enjoyed. Porting this functionality to Neorg would be ideal as using Obsidian feels cumbersome and it's not the easiest to manage (imo, ymmv).

I'm happy to lend a hand on this if possible but I wanted to raise the issue here in case something is being discussed or considered.

Help

Yes

Implementation help

A discussion concerning the merits of the feature and any prior discussion surrounding it should be had. Otherwise I'm pretty comfortable implementing feature work and asking questions as needed.

[benlubas]

This feature would be awesome, but it is a ways off still.

The neorg way to do this would be with macros afaik. Those are only planed, not implemented. Blog post here: https://vhyrro.github.io/posts/neorg-macros/

If you want to implement something yourself I'd do it as an external module. module tutorial

Databases and search/indexing are interesting topics, definitely want some solution there. I've sorta explored this in neorg-se, but this is more just my way to get category completions than anything else.

There where plans for a shared neorg database. But those haven't really developed. Norgberg is the name given to that. Neorgopolis is another future project that was in the works but has been put on the back burner. It would enable the database as well as a few other features.

Join the discord if you would like to get some of these moving, or honestly, if you implement something as you've described in this post as an external module, I would totally start using it.

[benlubas]

Oh, and there's also dynomark which might be adaptable to norg syntax. I've never looked into it though, but I kinda want to.

edit: unfortunately the code for this project is not very well organized. Also it's parsing and searching every file each time a query is run, so that will not scale. Best to start from scratch :/

[sofubi]

Thanks so much for the helpful info!

I've joined the Discord and will check in on the Norgberg/Neorgopolis progress when I can but I don't have the time currently to jump in at the beginning of those.

As far as macros- has development begun on implementing those? I read through the blog post and definitely agree that those would likely be the best way to implement the replacements for the queries.

I'm curious about implementing a module that maintains a small, single-file db with FTS (maybe of only titles) and some other searchability around meta that would allow for fast querying. Considering that, I may be interested in pitching in on the macro work depending on the current state.

Otherwise there's always a possibility to generate pages with some sort of CLI that utilizes the database and runs on some sort of command hook (maybe like a template expansion like macros but more manual) but I feel like it's nicer to be able to write queries dynamically.

I see that a lot of the feature work surrounding Neorg is in Rust, is that something that I should stick to? I haven't written much Rust but happy to do so.

[benlubas]

As far as macros- has development begun on implementing those?

Nope

I'm curious about implementing a module that maintains a small, single-file db with FTS (maybe of only titles) and some other searchability around meta that would allow for fast querying

I think making it a little larger and storing todo items would be really really worth it

Otherwise there's always a possibility to generate pages with some sort of CLI that utilizes the database and runs on some sort of command hook (maybe like a template expansion like macros but more manual) but I feel like it's nicer to be able to write queries dynamically.

I agree, I think this should be a neorg module that works dynamically, not a CLI tool that generates entire files.

I'm imagining a plugin that parses a structure like this:

@query SELECT TODO FROM ~/notes WHERE UNDONE
@end

And then dynamically populates the inside of that block with the results. Users can run a user command like :Neorg queries run or something. And then you could provide an autocommand config to run it on file open as well.

Rust, is that something that I should stick to?

Using rust gives you access to the rust parser, but there's nothing saying you have to use it. You can use the TS parser, but that's... kinda broken. So grepping for todo items is probably the best bet right now (lmao).

The future GTD and macro expansion system will be in rust afaik. Certainly if you write this in rust it would be appreciated if we plan to reuse your code. But I would not recommend using rust if you don't know it just b/c we're planning to use rust.

The strongest argument for using rust might just be the mlua crate, which provides pretty seamless interop between rust and lua. and easy install if you setup with luarocks.

[benlubas]

btw if you do all of the SQL stuff, and query parsing. I would happily do the lua/neovim plugin side if you want help

[sofubi]

I think making it a little larger and storing todo items would be really really worth it

Great thought! I'll be sure to include that.

I agree, I think this should be a neorg module that works dynamically, not a CLI tool that generates entire files.

This makes so much sense. I was trying to think of a way to work around not having macros so having a command exposed by the module to populate works great. I really like keeping the query in the opening anchor as well.

Certainly if you write this in rust it would be appreciated if we plan to reuse your code. But I would not recommend using rust if you don't know it just b/c we're planning to use rust.

I've spent time learning Rust syntax and concepts so I'm not walking in blind. I'm going to spend some time to see if it makes sense as I've been looking for the right application. I spend most of my time writing Python and I don't think that makes sense here either way. That all being said, can you point me in the direction of a good example of a Neorg module in Rust? I can dig around for some but I'd imagine there are one or two that stand out as "should be like this".

btw if you do all of the SQL stuff, and query parsing. I would happily do the lua/neovim plugin side if you want help

That's great to know, thank you! I'm gonna get a start on this and then I'll probably reach out on discord and we can collaborate.

Thanks again for the convo, this has been helpful! I'll report back here when I'm a bit further along.

[benlubas]

Neorg se (linked above) is the only module that uses rust. it does so with mlua, I was and still am not super familiar with it, so there could be easier ways to do things, but I'm pretty satisfied with the way neorg se is setup. Definitely reach out for help in the discord if you have questions about it. Vhyrro has used it in the past too.

[max397574]

just gonna mention this here if we'd again have an api to get tasks for a workspace it wouldn't be too complicated to reactivate the pickers for that from neorg-telescope