Ideas

[gnusupport]

In my opinion, package `ekg' is not opinionated, rather, more human friendly, and personally I can't see it as substitute for other applications, I see it as unique program.

So I propose changing your introduction: "The ekg module is a simple but opinionated note taking application. It is a substitute for such other emacs applications such as org-roam or denote. ekg stands for emacs knowledge graph."

There are a few core ideas driving the design of ekg. The first is that a title and a tag are the same thing.

If I may say, I deal with many tags, tag I consider a class of attributes or properties to an object.

I have elementary objects, they have their types, subtypes, types can be actionable or not, there may be additional properties, there are relations, etc. All those are properties in some sense.

Tags are properties across everything and are in itself important. I use notion of tag types as well:

 1          Default
 2          Skill
 3          Topic
 4          Language
 5          Action
 6          Place
 7          Sales
 8          Computer

And notion that user should be able to enter tag types.

What we like in databases is use of intersection. It implies for me that the more various sets of properties are there, the better we may pin point to various destinations.

Practically it means we can better find the note we are looking for.

We can easier find relationships between objects.

Tags as separate properties from object name are useful to create more intersections. Eliminating tags eliminate usefulness.

Back to searching, I have mentioned intersections. That is basic principle by which various database based searches work. There are different tables, different columns, types, classes defined in the databases.

When searching we can then design functions such as:

1. searching by tags only
2. searching by tags and including name words as tags
3. etc.
4. etc. various combinations

How I have understood your idea is that you wish to say that searching through tags and names of objects is helpful. Sure it is.

But it is not the only way of making intersections, there is number

3. and number 4. and X different ways for intersections.

If we think of merging various properties into "one" to provide better search results, than I can recommend the full text search, one reference is here:

Hyperscope full text search with PostgreSQL: https://hyperscope.link/3/6/7/6/8/Hyperscope-full-text-search-with-PostgreSQL-36768.html

As then, I can update tokens to include names of the tags, to include the name of object, description, text, language, country, related currency, names of related people, etc.

I can't say that excluding tags alone is beneficial, but I can say that including tags is beneficial in various search functions.

I have given you example in PostgreSQL, and I think full text search in SQLite does not exist. Even building on it is less productive for future, as it is single user database.

Build products that are multi-user and collaboration based.

This isn’t unique to ekg, other tools such as Logseq also consider tags to be equivalent to pages of the same name, although this functionality is limited since tags can only be just one word.

In my work I have table tags

                                              Table "public.tags"
┌───────────────────┬──────────────────────────┬───────────┬──────────┬───────────────────────────────────────┐
│      Column       │           Type           │ Collation │ Nullable │                Default                │
├───────────────────┼──────────────────────────┼───────────┼──────────┼───────────────────────────────────────┤
│ tags_id           │ integer                  │           │ not null │ nextval('tags_tags_id_seq'::regclass) │
│ tags_datecreated  │ timestamp with time zone │           │ not null │ CURRENT_TIMESTAMP                     │
│ tags_datemodified │ timestamp with time zone │           │          │                                       │
│ tags_usercreated  │ text                     │           │ not null │ CURRENT_USER                          │
│ tags_usermodified │ text                     │           │ not null │ CURRENT_USER                          │
│ tags_name         │ text                     │           │ not null │                                       │
│ tags_description  │ text                     │           │          │                                       │
│ tags_languages    │ integer                  │           │ not null │ 1                                     │
│ tags_tag1         │ integer                  │           │          │                                       │
│ tags_tag2         │ integer                  │           │          │                                       │
│ tags_tag3         │ integer                  │           │          │                                       │
│ tags_tagtypes     │ integer                  │           │ not null │ 1                                     │
│ tags_hidden       │ boolean                  │           │ not null │ false                                 │
│ tags_people       │ integer                  │           │          │                                       │
│ tags_rank         │ integer                  │           │ not null │ 0                                     │
└───────────────────┴──────────────────────────┴───────────┴──────────┴───────────────────────────────────────┘

You may notice that tag has its own properties. It has its name, and because tag is addressed by its unique ID, the name can have spaces. It may be long. It can contain any chars. It can have its description. It may be in different language.

It may be tagged by three other tags. Why is that useful? Tag once any object, and computer will update it with any additonal tags. If I tag something with "Video" computer will tag it with "Media" and "Video", as maybe that is what user wants. But I do not keep to add relevant synonyms or relevant tags, I can add just one of them. If that feature becomes very useful, I would simply provide tagging of tags.

Because tags are addressed by ID as integer and not by name, I can rename tags on the object without losing the tag from the object. All objects then appear with the renamed tag!

Then there are tag types. Very useful.

Maybe there is elementary object, like note, speaking about "Technical School", but that they provide skill of "Optician" is not derived from the name alone. That is one example among many why tags are useful.

It is however useful concatenating tags with name and searching among them, that is one of intersections that are helpful to human.

In org-roam, a tag is just a tag, so you can have a note called “emacs” and a tag called “emacs”, but these are not related.

Okay

ekg takes the idea a step further: there are (mostly) no titles, only tags. So, instead of writing text in a note called “emacs”, just write a note and tag it with “emacs”. There is no “title”, only tags.

In my opinion that design is backwards, it does not help.

If it is searching by name, or splitting name into words to search by those words, then it is not "tag".

Tag is a property separate from object. It may have different name than anything in the object. I can have "USD" for US dollar as tag.

Tags are properties of the object that have no special group or class.

It is useful to have tags in existence which are not related to any object. This is for human to have easier work. What if you have tags for currencies such as "USD", "EUR", "GBP", and you wish to tag object with it, but in that moment you did not have "GBP" ready, so instead of writing it every single time, you can prepare the tag list for user to select it easily.

Elementary Objects: https://www.dougengelbart.org/content/view/110/460/#2a1a

My elementary objects have "Currency" property directly attached, and whatever is more important it becomes directly attached to the object. Tags are there as dettached properties that may belong to any object.

Separate functions can be made to automatically relate objects to tags. Such as making tags out of the name. As that seems to be that what you designed.

Example is with the object named "NonGNU Emacs Lisp Package Archive" that may be automatically tagged by tags such as "Emacs" and "Archive".

If you write another note about emacs, also tag it “emacs”, and maybe something else too. Or tag it something more involved, like an idea: “emacs’s power derives from putting all data in buffers, and making all commands deal with buffers.” That’s a perfectly fine tag, and if you notice a connecting idea, you can tag it with this as well.

That is right and good. My tags may also be of arbitrary length.

Though tags are useful because they are more simplistic ideas, not complex.

Their usefulness is derived from their combination, not from their quality.

Their meaning shall be elementary meaning, not complex meaning.

Their purpose shall be generation of intersections.

A tag like "Emacs" is less useful, as it would find anything about Emacs. Then "Emacs" combined with "ekg" is more useful. Isn't it?

Using name of object to generate tags is useful. But excluding tags as such is not.

The advantage of this method is that it solves something that has bothered me for a while about the recent suite of tools like org-roam: backlinks are non-symmetrical. If you enter a note in your org-roam daily about emacs, and link it to the emacs note, then when you go to the emacs note, you have to explicitly enable the backlinks buffer to see the daily entry where you first entered it. Systems such as Logseq and the original Roam have backlinks alongside normal content, but this doesn’t seem possible in emacs, where a buffer of a file is expected show the file, and tricks with overlays can’t solve the issue. Even if it could, I want a system in which it doesn’t matter where you enter the data, it shows up in the original place the same as everywhere else it is linked to, not as a backlink, but just as part of the content. Having notes with no title, only tags, makes this possible, because there is no longer a difference between linking and writing in the context in, both are denoted by tags.

In my opinion your solution is only one of many solutions. It is not necessary to be so, it can be implemented in various ways.

As a consequence of this design, notes can be small, because to add another note to a subject, you don’t need to append to an existing note, you can create another note.

That is right, multiple notes shall be available for any possible relation.

Additionally, ekg has another key difference: it uses sqlite instead of the filesystem. When notes are small and do not have titles, files don’t make a lot of sense anymore.

File system is one way of "tagging" and sorting of files. It has directories, filenames, hierarchical structure and access by variety of means.

It is up to user to "sort" his stuff, relate to each other. It is not as flexible as database.

Additionally, the filesystem is limited. Even in org-roam, which uses it, it needs to be augmented with sqlite anyway to enable fast querying of tags and other operations. The sqlite-only approach also means it is much easier to make certain kinds of changes, since they only involve changing the database and not the text as well. In general, text and data are separated as much as possible here, so there’s no need or desire for the text to have to store data as well, we leave that completely to the database.

In my opinion your introduction is difficult to comprehend. It is good if you make video or screenshots.

Prefixed tags

Another concept, loosely applied in ekg is that of tags with standard prefixes. By default, date tags are prefixed with “date/”. This is a way to distinguish date tags from other kinds of tags. Most tags shouldn’t need it, but it often is useful to have prefixes to group tags in some way. For instance, perhaps all idea tags should be prefixed with “idea/”. In my ekg repository I use in my company, I have “person/” as a tag prefix for my coworker’s username.

We speak here of tag types. Think of implementing it. If you implement "Default" tag type and let user add any other tag type, that way you will give useful system. Users can then decide that tag belongs to tag type "Idea", or "Skill" or maybe "Time".

The benefit of this is that it’s now possible to narrow in on just tags of a certain type if necessary.

We speak here basically of tags for tags. Or properties of the tags. And I find it very good notion for future.

There are a few other types of prefixes commonly used for tags. One is that titled resources have default tags that are prefixed with “doc/”, followed by the name of the document. Removed tags are prefixed with “trash/”, but these are normally invisible to the user. There’s a section on these trash tags below which goes into more detail.

I would not do it that way to allow human mistake, for human to add string with slash, rather using tag types.

I have table tagging, if tag is removed from object, it is removed from table tagging, the table references tag with elementary object or with people. But there is use for the tag to remain in the database, as for future selection of tags.

In this case tag can be related to people object or document object. There is hundred of other tables, but for those two tags are most useful.

                                                    Table "public.tagging"
┌──────────────────────┬──────────────────────────┬───────────┬──────────┬───────────────────────────────────────────────────┐
│        Column        │           Type           │ Collation │ Nullable │                      Default                      │
├──────────────────────┼──────────────────────────┼───────────┼──────────┼───────────────────────────────────────────────────┤
│ tagging_id           │ integer                  │           │ not null │ nextval('peopletags_peopletags_id_seq'::regclass) │
│ tagging_datecreated  │ timestamp with time zone │           │ not null │ CURRENT_TIMESTAMP                                 │
│ tagging_datemodified │ timestamp with time zone │           │          │                                                   │
│ tagging_usercreated  │ text                     │           │ not null │ CURRENT_USER                                      │
│ tagging_usermodified │ text                     │           │ not null │ CURRENT_USER                                      │
│ tagging_tags         │ integer                  │           │ not null │                                                   │
│ tagging_people       │ integer                  │           │          │                                                   │
│ tagging_hyobjects    │ integer                  │           │          │                                                   │
│ tagging_description  │ text                     │           │          │                                                   │
└──────────────────────┴──────────────────────────┴───────────┴──────────┴───────────────────────────────────────────────────┘

My tagging is such that I can tag with existing tags or newly created tags. Tag searching uses function completing-read-multiple to allow me finding multiple tags.

[ahyatt]

Thank you very much for all these excellent notes.

Let me respond to some of this:

First, I agree that we really should have a full-text search. That would be the responsibility of the triples package, which I also maintain. I need to try this out, I hope that the emacs 29 sqlite has the necessary capabilities to do this. Anyway, I completely agree that it would contain tags and text.

About tags with tags, yes, right now ekg is very close to being able to do that. It just needs the ability to have tags have notes, which is very natural IMHO and fits in with how at least I want to organize things. Since the UI is note-driven, once we have notes, the user can tag the tag with whatever tags they want. Whether this has any interesting effects is something I'll have to think about, your experience on this is interesting though.

I also do have a video I've recorded, but I don't want to do any sort of advertisement of this package until it is in a repository. I'll see if I can make the text a bit more understandable, though.

[gnusupport]

In PostgreSQL it is easy to provide full text search. I strongly suggest switching to PostgreSQL and going away from single user model, single computer model. Collaboration with multiple users is so much more useful.

I have handled searches by using function like following:

(defun rcd-sql-search-snippet-for-and-column (column query &optional operator logic)
  (let* ((words (split-string query nil t (rx (any whitespace))))
     (operator (or operator "~*"))
     (logic (or logic "AND"))
     (my-and (mapconcat 
          (lambda (e)
            (concat " " column " " operator " " 
                (sql-escape-string e) " "))
          words logic)))
     my-and))

What it does is creating SQL queary with AND or OR logic for all words:

(rcd-sql-search-snippet-for-and-column "my_db_column" "query words I am searching for") ➜ " my_db_column ~* E'query' AND my_db_column ~* E'words' AND my_db_column ~* E'I' AND my_db_column ~* E'am' AND my_db_column ~* E'searching' AND my_db_column ~* E'for' "

and

(rcd-sql-search-snippet-for-and-column "my_db_column" "query words I am searching for" "LIKE" "OR") ➜ " my_db_column LIKE E'query' OR my_db_column LIKE E'words' OR my_db_column LIKE E'I' OR my_db_column LIKE E'am' OR my_db_column LIKE E'searching' OR my_db_column LIKE E'for' "

As that helps in constructing SQL queries to search this and that. On my side this is mostly used function.

[gnusupport]

For PostgreSQL:

Mastering PostgreSQL Tools: Full-Text Search and Phrase Search - Compose Articles: https://compose.com/articles/mastering-postgresql-tools-full-text-search-and-phrase-search/

PostgreSQL: Documentation: 15: Chapter 12. Full Text Search: https://www.postgresql.org/docs/15/textsearch.html

FOR SQLite:

SQLite FTS5 Extension: https://www.sqlite.org/fts5.html

[gnusupport]

I suggest looking into following design:

TECHNOLOGY TEMPLATE PROJECT OHS Framework : https://www.dougengelbart.org/content/view/110/460/

Objects are basic content packets of an arbitrary, user and developer extensible nature. Types of elementary objects could contain:

• text
• graphics
• equations
• tables
• spreadsheets
• canned-images
• video
• sound
• code elements, etc.

I strongly suggest being able to record any kind of object. It does not mean saving video into database, but if video is on the file system, it would mean indexing the video from file system or putting it in the file system structure devised by program, and note has type of video.

I use that approach. See more on that link about Open-Document Hypertext Systems, as following that design makes programs most useful.