search

ndwarshuis / org-sql

SQL backend for Emacs Org-Mode
GNU General Public License v3.0
97 stars 9 forks source link

Plans? #3

Closed alphapapa closed 4 years ago

alphapapa commented 4 years ago

Hi,

I found this package today through https://github.com/l3kn/org-zettelkasten, which I also stumbled upon today. A couple of years ago I was working on a branch of helm-org-rifle that indexes Org files in SQLite, also inspired by John Kitchin's work, but I haven't touched it in a while. Since then I've worked on org-ql, and one of the long-term ideas I have for it is to have an indexed backend, especially for Org files that aren't open in Emacs. So it would be interesting if this package could be a part of that.

So, if I may, what are your long-term plans for this package?

Thanks.

ndwarshuis commented 4 years ago

Thank you for the interest :) I haven't worked on this in a while mostly because I've been working on om.el. Indeed, making this was one of the inspirations for om.el, since alot of the functions I needed to use I had to make from scratch but would be useful elsewhere, hence library.

As far as plans, I don't see a huge need to add additional features unless there are other suggestions. The data this collects I think is fairly complete as far as representing org files in tabular format. Some current ideas which I see as incremental:

I still haven't gotten around to making a front end for this (the original vision using R/python per the README) but that's also going to be a priority moving forward. Indirectly that may call for additional changes here.

Did you have any features in mind that are missing?

alphapapa commented 4 years ago

Cool. My idea for an org-rifle indexer was to do the indexing in another process which would receive commands to index files from the main Emacs process, or run in a batch mode from a script, and then have an Emacs UI for searching the database interactively.

Quick question: have you experimented with SQLite full-text search indexes? In another project I did some tests with different kinds of FTS indexes, with and without different kinds of database compression, which you might find interesting: https://github.com/alphapapa/sword-converter/blob/master/notes.org#conclusion FTS indexing would be a crucial feature, of course, and SQLite provides a few different options. Compression would probably not be useful here, of course.

ndwarshuis commented 4 years ago

Cool. My idea for an org-rifle indexer was to do the indexing in another process which would receive commands to index files from the main Emacs process, or run in a batch mode from a script, and then have an Emacs UI for searching the database interactively.

Like another process within emacs? As of now the sql update doesn't run asynchronously, but that shouldn't be hard to add if needed.

Quick question: have you experimented with SQLite full-text search indexes? In another project I did some tests with different kinds of FTS indexes, with and without different kinds of database compression, which you might find interesting: https://github.com/alphapapa/sword-converter/blob/master/notes.org#conclusion FTS indexing would be a crucial feature, of course, and SQLite provides a few different options. Compression would probably not be useful here, of course.

Nope, I haven't really tried anything beyond SQLite 'defaults.' Off the top of my head FTS shouldn't be too hard to add given the tables and operations I use here. Please open a separate issue for this with additional details as needed.

alphapapa commented 4 years ago

I'm just throwing some ideas out there. :)

BTW, when you're ready to do testing/CI, you may find this helpful: https://github.com/alphapapa/makem.sh