Parse Tags from the Note itself

[cspann]

Expected

Wouldn't it be better to indicate Tags by e.g "@tag" in the notes itself to make notes really self contained. Otherwise all tags will be lost when migrating to another Tool.

Actual behaviour

Tags ar not contained in the documents but in a separate sqlite db

Steps to reproduce

none required

Output from the debug settings

Relevant log output (you have to enable enable the log widget)

[pbek]

Thank you for your suggestion, but this is about the 4th or 5th time such a request was made and every time an other "format" to store tags was suggested.

Problems are:

• there is no standard to include tags in a markdown file
  • how to handle nested tags?
• it would require that QON parses and modifies notes by itself when a tag is added to a note (which isn't a very good idea in my opinion and might lead to a lot of troubles)
• it needs to work for several 1000 notes in multiple note subfolders
• the note needs to be reparsed every time the user types something

[cspann]

there is no standard to include tags in a markdown file

Good point, but like gitlab/github where they have enhanced markdown I see no special problem here.

    how to handle nested tags?

What are nested tags?

it would require to that QON parses and modifies notes by itself when a tag is added to a note (which isn't a very good idea in my opinion and might lead to a lot of troubles)

I wouldn't handle it that way. To add a Tag the user has to write "@tag" somewhere in the document and you then index the file for this tag in your database

it needs to work for several 1000 notes in multiple note subfolders

Parsing is done only on save or when you import a new set of notes. The problem is not that you store the notes in a separate database, but that you store them only there.

Does this make my request more clear?

[pbek]

Good point, but like gitlab/github where they have enhanced markdown I see no special problem here.

there still is no standard and every feature request wanted something else

What are nested tags?

tags in tags

I wouldn't handle it that way. To add a Tag the user has to write "@tag" somewhere in the document and you then index the file for this tag in your database

In the current interface user can add tags with a button, or tag multiple notes, or move tags to other tags, or remove tags from multiple notes and you can rename or remove tags (remove a tag from a note and removing a tag altogether)... The "new way" to handle tags would be needed to be a 2nd, alternate way with a different user interface.

Parsing is done only on save or when you import a new set of notes. The problem is not that you store the notes in a separate database, but that you store them only there.

Parsing would be done every time the current note gets saves (there are people who do that every second) while typing and when all notes need to get reloaded (for example when external changes in a note folder happen).

[cspann]

there still is no standard and every feature request wanted something else

I see no problem in the lack of a standard as long as the tag escape char does not interfere with markdown. When I would have to migrate I could easily replace your tagging symbol with the new one with a simple sed call on all files. The marker should just not be part of markdown. Laverna has this problem because it does exactly what I suggest but it uses the # as tag marker and instantly creates a tag. This is not good as # also prefixes h3 and higher indexed headers.

What are nested tags?

tags in tags

I would say nesting is against the very purpos of tags and would not support it. I would even think about not allowing spaces in Tags

[Maboroshy]

I've used the app that does something very close to request - Wikidpad. It's made around parsing notes for meta-data. I really love the concept, but I ditched it in favor of QOwnNotes without any regret. It's simply not worth the hustle.

Want to save tags outside of SQL? There's an in-app script to put it to a file name.

Want to make it portable? Well, it's not really portable, since there's no standard, and no other apps parse in-text tags. To get tags for Epsilon notes, which I use on my phone, I've ended up getting them from SQL and making an index note with links to all notes sorted by tags.

Getting tags and putting them as the first line of every note in "@" format is not a big thing. But what to do with that next?

[Losiara]

Markdown is an html. Tags my be placed in \ tag as attrs or values. This will be ok in exporting to html/pdf and easy for parsing.

[Maboroshy]

Could you clarify? The app renders <tag>Test</tag> as text. I personally prefer [tag]: (Test) for hidden text, but it has a problem with spaces inside brackets.

[Maboroshy]

I've put some script prototype:

import QtQml 2.0
import com.qownnotes.noteapi 1.0

QtObject {
        function handleNoteTextFileNameHook(note) { 
        if (note.noteText.substring(0,6) == "[tag]:") {
            note.noteText = "[tag]: (" + note.tagNames() + ")\n" + substring(note.noteText.indexOf("\n") + 1)
            script.log( "[tag]: (" + note.tagNames() + ")\n" + note.noteText.split("\n").slice(1).join("\n") )
        } 
        else {
            note.noteText = "[tag]: (" + note.tagNames() + ")\n" + note.noteText
            script.log( "[tag]: (" + note.tagNames() + ")\n" + note.noteText )
        }
    }
}

But it won't work because note.noteText is a one-way thing, I can't write to it from script. You can see how it is supposed to work in the log, try use smaller notes. Tags syntax can be altered.

As far as I understand, it can be implemented only:

• if @pbek will make some function to freely edit note text from a script or a hook like handleNoteTextFileNameHook but for text.
• via running external shell commands which I can write only for Linux. But it's easier to rip all data directly from SQL for all notes, and I already have a shell script for that.

Also It would be good to be able get nested tags like nested/tag.

[Losiara]

For example <t tag="a, b, c"><tag> or <tag>a, b, c</tag> and so on. I thought that it should been rendered as invisible in preview without any additional efforts. Because of unknown tags.

IIANM there is a function convertHtmlToText. It should omit this html in rendering.

[Maboroshy]

I use markdown because it's portable and "future-proof", so I'm trying not to put anything tied up to specific app. So I don't think using convertHtmlToText is a convenient solution.

<t tag="a, b, c"><tag> is hidden in QOwnNote, <tag>a, b, c</tag> isn't. But both are visible in Epsilon Notes I use on my phone.

Anyway, if there'd be a script function to edit text, it will be easy to change format.

[Losiara]

But both are visible in Epsilon Notes

Thats because Epsilon notes is out of standart. Usually Markdown - is "wrapped and rendered as html". So "unknown tags" should be ok. But I realised that webengine is not an option for this project. So your option seems right.

[fairplay]

Any inline tagging will be easy to implement using scripting, but API lacks "removeTag" method, is it possible to add it?

[pbek]

@fairplay can you please explain in more detail what you are trying to do?

[fairplay]

I have a little script which parse third line for tags in org-mode format (":tag1:tag2:tag3:")

import QtQml 2.0

QtObject {
    function onNoteStored(note) {
        var lines = note.noteText.split(/\n/);
        var tags = lines[2].split(':');
        for (var i = 0; i < tags.length; i++) {
            var tag = tags[i];
            if (tag.length == 0) {
                continue;
            }
            script.tagCurrentNote(tag);
        }
    }
}

But I need to clean up existing tags before setting new ones. Did I miss smth?

[pbek]

Does that script already work? It's a bit dangerous, because script.tagCurrentNote tags the current note, and not the note that is provided with onNoteStored. script.tagCurrentNote also takes care of updating the user interface of the current note.

And your script only would take care of the current note (when you modify it in QOwnNotes), it doesn't take care of all other notes.

[fairplay]

Ye, thanks, it was mistake, but usually it works because it seems that note stored is current note usually. Nevertheless, it causes the error "UNIQUE constraint failed" on the second call, obviously. Can I manipulate note's tags directly somehow, without "tagCurrentNote" usage?

[pbek]

The links between tags and notes are stored in notes.sqlite in your note folder, if that is what you mean.

[fairplay]

I understood that, I mean "directly" using scripting API.

[pbek]

No there isn't. If you would want to play around with the code, https://github.com/pbek/QOwnNotes/blob/develop/src/api/noteapi.cpp is a good point to start. You would need Tag::removeLinkToNote to remove the tag from a note. See: https://github.com/pbek/QOwnNotes/blob/develop/src/entities/tag.cpp#L539

[pbek]

@fairplay, @Maboroshy something to play around with

17.09.5

• added two new scripting-methods addTag and removeTag to NoteApi to add and remove tags from notes (for Issue #530)
  • custom tagging could now be implemented by parsing the note text for custom tag handlers and adding/removing tags from the note
  • for more information please take a look at the scripting documentation

[pbek]

And I'm working on a QQmlListProperty<NoteApi> NoteApi::fetchAll(int limit, int offset) to fetch all notes, but I haven't tested it yet.

[pbek]

There now is a new release, could you please test it and report if it works for you?

[Maboroshy]

Good stuff. I'll start looking into tags "parser". But it would still require note text modification, that can be done only externally now. When user drops tag by UI the tag should also disappear from the note text. So more Python job again. Not that I'm having anything against it though.

fetchAll thing is interesting. I'm still hoping to resolve the issues I had with externally creating and tagging note in a single script run.

[pbek]

Yes, the new methods were only meant for "one way". Parsing the note text and adding/removing tags. You really also want to modify the note text?

[Maboroshy]

I think it's a must. User would still be able to modify tags by UI, and he will expect the script reacts to it accordingly.

Also, I think that important part of in-text tagging is that such tags have a position. So selecting a tag should bring cursor to it. But I leave it to some second or third version if one ever will be. It would be quite painful.

[pbek]

There are so many operations with tags... Tags can also be removed altogether (I guess than all notes have to be checked and modified) and notes can be tagged in bulk. There must be a hook for everything.

The script would have to decide what to do if a tag should be added to or removed from a note.

[pbek]

And all notes would need to be scanned for tags constantly (every time when a note tree reload would be requested).

[pbek]

Reference - external tagging feature requests

• (Question) Is there a specific reason the tags are stored outside the notes themselves ? · Issue #323 · pbek/QOwnNotes · GitHub
• Suggestion for enhancement: Write the tags to the fils · Issue #338 · pbek/QOwnNotes · GitHub
• Enhancement: Add ability to enter tags directly in each file/document · Issue #379 · pbek/QOwnNotes · GitHub
• Sync tags with ownCloud / Nextcloud tags · Issue #472 · pbek/QOwnNotes · GitHub
• Feature request: Different tag system · Issue #603 · pbek/QOwnNotes · GitHub
• Parse Tags from the Note itself · Issue #530 · pbek/QOwnNotes · GitHub

[pbek]

@Maboroshy I started the implementation of such a tagging system. One could then implement a script like this to handle tagging: https://github.com/pbek/QOwnNotes/blob/ccec0bb61b764405c7a07a098410063c817e8900/doc/scripting/note-tagging.qml

[Maboroshy]

Don't you want to talk it through?

With what I know I would do the following script:

Main tag container is note text. Tag db is a secondary container.

Init: 1) Poll tag db for tags of all notes; 2) Parse all note's text for tag words; 3) If there are differences alter tag db based on note text.

Changes in tag db: 1) Detect modification by watching db file file system events; 2) On file change - repoll tag db; 3) Modify note texts accordingly.

Changes in some notes: 1) Detect modification by watching note folder file system events; 2) Reparse note files that were changed; 3) Modify tag db accordingly.

To make such script I don't need many hooks and stuff. I need cross-platform tag db interface that works with all note files, rather than only visible ones, and has no issues with sub-folders or nested tags. I'd rather not use direct SQL queries because they will break on db structure changes. Python can do the rest.

[Maboroshy]

From example script I don't understand how would it work with nested tags. Like two "subtags" with the same name. Also there should be "rename" action.

[pbek]

From example script I don't understand how would it work with nested tags. Like two "subtags" with the same name.

there will be no subtags, the first tag with the name will match

Also there should be "rename" action.

don't know yet, maybe remove and add, maybe an other action in the script...

[pbek]

My current plan is that as soon as such a script is activated the tag db will be "ignored" and tagging information will come from the notes only (synced from the notes into the db).

[pbek]

The current tags will stay in the note db, but the linking to notes will be synced from the notes itself, new tags will be created in this process. They can then be moved around into a parent tag the usual way.

[Maboroshy]

Rename is needed to save position of tag in text. If you remove then add, tag will appear in default position rather than the one user put the tag to.

I think there better be an initial dialog to put all db tags to text, since the db data will be lost.

[pbek]

Good idea!

[pbek]

I think there better be an initial dialog to put all db tags to text, since the db data will be lost.

Hm, but I'm not sure yet how to prevent that the tag-links are removed from the database as soon as the tagging-script is activated...

[pbek]

17.09.6

• added the first alpha version of external note tagging support via the scripting engine (for Issue #323, Issue #338, Issue #379, Issue #472, Issue #530 and Issue #603)
  • as soon as a script is activated that implements the new function noteTaggingHook note tagging will be handled by that function
  • please install the script note-tagging.qml to test the basic functionality and report in Issue #530.
    • your internal tag database and your notes might get modified
    • don't use this script on your production note folder, please make a copy of it and test it there!
  • following features should already work via the QOwnNotes user interface
    • initially importing tags like @tag from your notes and overwriting your current tag assignment
      • you will not loose your tags tree, just the former assignment to notes
      • you can still move tags into other tags
      • if more than one tag has the same name in your tag tree the first hit will be assigned
    • adding a tag to a note will add the tag to the note text
    • removing a tag from a note will remove the tag from the note text
    • removing of tags in the tag list will remove those tags from your notes
    • renaming of tags in the tag list will rename those tags in your notes
    • bulk tagging of notes in the note list will add those tags to your notes
    • bulk removing of tags from notes in the note list will remove those tags from your notes
  • for more information please take a look at the scripting documentation

[pbek]

There now is a new release, could you please test it and report if it works for you with the script note-tagging.qml?

[Maboroshy]

Testing. It works, but sometimes makes surprising stuff. I'll start posting what I find.

If there're two notes named 1, and only one of them is tagged, filtering by the tag will show both 1 notes. Maybe that is also true for normal tagging.

[Maboroshy]

Unable to see tagged items from other sub-folders hurts. I know it's not related to the issue, but I did not touched tagging for a few months, and it hurts me again like the first time.

[Maboroshy]

Looks like tags are parsed on note list reload. So, when I put tag to note, I don't see update until I wonder around note structure a little. That's confusing. With no sub-folder structure it can be even more confusing, since you get updates only manually or creating-deleting notes.

I think the best way would be reparse current note on each save. Don't know if it's appropriate for the app's architecture.

[Maboroshy]

Sometimes app crashes silently on manual folder reload after some in-text tag modification. I'll investigate further. Maybe something is racing.

[Maboroshy]

When I tag a note by UI, I won't see update in text until update. It's worse with removing by UI - if I continue editing unupdated note text, the tag will remain.

[Maboroshy]

Non-latin tags are broken. When I put @幻 to text or tag that by UI, it just won't sync.

[pbek]

I think the best way would be reparse current note on each save. Don't know if it's appropriate for the app's architecture.

the problem is that the note is stored to disk in the background every X seconds, I'll see if I can parse the note before that...

[Maboroshy]

I was thinking about parsing right after that.

[pbek]

When I tag a note by UI, I won't see update in text until update. It's worse with removing by UI - if I continue editing unupdated note text, the tag will remain.

yes, I'm still trying to get around that somehow...

[pbek]

Non-latin tags are broken. When I put @幻 or tag that by UI, it just won't sync.

interesting, so the tag doesn't get into the note text? does the log panel say anything?

[pbek]

I was thinking about parsing right after that.

the background mechanism doesn't know anything about tags... I'll try to do it when the note gets saved to RAM