pbek / QOwnNotes

QOwnNotes is a plain-text file notepad and todo-list manager with Markdown support and Nextcloud / ownCloud integration.
https://www.qownnotes.org/
GNU General Public License v2.0
4.69k stars 414 forks source link

Parse Tags from the Note itself #530

Closed cspann closed 6 years ago

cspann commented 7 years ago

Expected

Wouldn't it be better to indicate Tags by e.g "@tag" in the notes itself to make notes really self contained. Otherwise all tags will be lost when migrating to another Tool.

Actual behaviour

Tags ar not contained in the documents but in a separate sqlite db

Steps to reproduce

none required

Output from the debug settings

Relevant log output (you have to enable enable the log widget)

Maboroshy commented 7 years ago

The log says nothing about this tag on reload. I suppose that means script doesn't "see" it.

pbek commented 7 years ago

The log says nothing about this tag on reload. I suppose that means script don't "see" it.

the regular expression of the script matches \w+, that excludes other characters, so you can use spaces and even , to separate the tags... everyone can implement their own mechanism...

pbek commented 7 years ago

Sometimes app crashes silently on manual folder reload after some in-text tag modification. I'll investigate further. Maybe something is racing.

interesting, I never had that... it would help to know where it crashes (maybe you want to fire up QtCreator) ;)

Maboroshy commented 7 years ago

Check: @Österreich or

@Hi
@Dobrý den
@Привет
@ハロー
@שלום
pbek commented 7 years ago

with \B@([^\s,]+)\b there are some more hits, but not all, you can test it yourself here: https://regex101.com/r/ZfuAZp/1/

pbek commented 7 years ago

I found it, we need the u (unicode) flag to make it work, but that isn't supported by JavaScript. Phew...

pbek commented 7 years ago

17.09.7

Maboroshy commented 7 years ago

I found weird bug. 1) Empty folder with no sub-folders. Create 2 new notes, default names. 2) Put @1 to one, @2 to other; 3) Refresh manually to see the tags in UI; 4) Delete 1 tag by UI; 5) Two notes have the same text now. It means one notes' content is is lost.

It works every time for me.

pbek commented 7 years ago

Thank you for testing that... Oh the fun of modifying notes in the background. Makes me remember why I opposed it in the first place... :)

Maboroshy commented 7 years ago

Maybe it's because you've started hacking RAM. I was hacking saved files for a long time, mostly without any issues, because that scenario is the use case for the app.

Maboroshy commented 7 years ago

I found it, we need the u (unicode) flag to make it work, but that isn't supported by JavaScript. Phew...

QML has JS syntax, but can have different capabilities. Maybe RegExpValidator supports unicode. On the other hand, Qt docs says it's regexp fully supports unicode, so you could pass C++ regex function to QML to bypass QML limitations.

Maboroshy commented 7 years ago

There maybe changes about feedback when I use UI for tagging. But I haven't yet seen any improvements, when I put tags to text. I still have to manually refresh folder every time. And it still crashes sometimes, 3 times during my testing today.

Maboroshy commented 7 years ago

Also I want to propose supporting nested tags in scripting. So that script would add some regexp defined tags to specific nested tag rather than to root.

That can be a step to expanding in-text tagging to a universal note attributes rather than just tags. For example adding @>21.09.2017 could create tag with date in dates nested tag. Or @!10:00 can add tag to reminders nested tag and be parsed by external reminder app, that will show a pop-up with note content at 10 o'clock.

Something like this feature. I considered it as killer one, when I used that app.

pbek commented 7 years ago

Maybe it's because you've started hacking RAM. I was hacking saved files for a long time, mostly without any issues, because that scenario is the use case for the app.

the troubles start when note files get written to disk and/or the DB file gets written (when tags are added)... then the note folder might get reloaded which causes the notes in the memory-database get reloaded... while not all tags and notes are stored to disk...

pbek commented 7 years ago

RegExpValidator

seems to need QtQuick

pbek commented 7 years ago

There maybe changes about feedback when I use UI for tagging.

adding tags via the UI should work better

pbek commented 7 years ago

Also I want to propose supporting nested tags in scripting. So that script would add some regexp defined tags to specific nested tag rather than to root.

right now, not even "normal" tags are working flawlessly because all the race conditions that stuff is causing

pbek commented 7 years ago

when I put tags to text. I still have to manually refresh folder every time.

the alternative is that the notes has to get parsed completely (and constantly) every time someone enters text, I'm not sure if that is a good idea

Maboroshy commented 7 years ago

the troubles start when note files get written to disk and/or the DB file gets written (when tags are added)... then the note folder might get reloaded which causes the notes in the memory-database get reloaded... while not all tags and notes are stored to disk...

Maybe that is the point to put effort to. Making the app unbeatable by any modifications done to db or files while it's running.

the alternative is that the notes has to get parsed completely (and constantly) every time someone enters text, I'm not sure if that is a good idea

I'm still under illusion that it can work on note files save. Save interval can be tuned by user and can't be less than 1 second.

Never done or read about something like that, but maybe the parsing scope can be narrowed down by tracking cursor position/actions. Some regexp benchmarks can show, if it worth the hassle. Maybe raw parsing will be fast enough for everything.

ludenticus commented 7 years ago

If the idea behind the script is to automatically parse the note tags, then, the UI shouldn't be able to delete tags, right? I mean, every add/modify/delete action should be done within the text. In this case,

  1. A subtag could be separated by a colon: @tag:subtag, @long tag:long subtag
  2. The «Note tags» widget will only read the tags/subtags: no option to add, no option to delete
  3. The «Tags» navigation widget maybe will retain only the color related functions. I don't know if it is technically possible to retain the rename function with this approach.
Maboroshy commented 7 years ago

Let's assume you have 100 notes tagged with one tag. Then you suddenly realise that this tag is not correct or relevant anymore. Should you manually replace/delete it in 100 notes or make a few clicks for a best note app do it for you?

Maboroshy commented 7 years ago

Add and delete brings nothing wrong. Keeping them will minimize UI changes and user confusion.

ludenticus commented 7 years ago

Of course, I'd prefer that QON makes the job. That goes without saying. Bulk tagging is quite convenient. But after reflecting on your comment, I see the problem is the creation of a new tag: Tag rename and Tag delete shouldn't be major problems, aside from the serious technical challenges they pose. However, create tag is in itself open to question, as you should determine where to put the tag within the note. Some people like it in the end, some others right after the title, or even after some (sub)header. This is why I was reluctant to let the UI manage the tags if they are to be parsed from the notes.

Maboroshy commented 7 years ago

That's why it is the script. It can be modified to suit anything user wants. Having UI to add tags to predefined position, that suits the user, is better than just disabling that function.

pbek commented 7 years ago

17.09.8

pbek commented 7 years ago

There now is a new release, could you please test it and report if it works better for you?

Maboroshy commented 7 years ago

Looks like note text overwriting is gone. But I get the stable app crash on mass tag deletion by UI. After crash I see only one if tags is deleted.

pbek commented 7 years ago

yeah, there still can happen some race conditions, I had to rewrite quite some things to get around some of them :sweat:

pbek commented 7 years ago

17.09.9

pbek commented 7 years ago

There now is a new release, could you please test it and report if it works for you?

Maboroshy commented 7 years ago

Had one unrepeating crash on refresh, but no crashes on multiple tags operations.

pbek commented 7 years ago

What is refreshing and what do you have to do that it refreshes?

Maboroshy commented 7 years ago

I mean "Reload note folder" button. I always use it to force the script parsing tags.

pbek commented 7 years ago

17.10.0

Maboroshy commented 7 years ago

You beat me to it. Looks like I'll have to make pull requests to this script.

pbek commented 7 years ago

If it's an improvement just go ahead, if it's a new tagging system you can create a new script. Btw. did you find anything else that doesn't behave well in the external note tagging system?

Maboroshy commented 7 years ago

I don't see much space for some new tagging system for now. I've implemented a setting to configure tag marker, so everybody can choose what suits them. Personally I got quite many misfires after parsing my "production" notes. So I'll be moving from @, which is a sane default though.

Core functionality works quite reliably now for me. I think that this issue is resolved.

ludenticus commented 7 years ago

Wouldn't it be easier if the user could directly add a RegExp to the settings of the script within the QON Settings window, as with other scripts?

Maboroshy commented 7 years ago

That was my first thought. But that requires quite a skilful user. There are in-fact two regexps that must be carefully adjusted to work together.

I think that there must be basic configuration for a casual user, like what characters you want to put before tag word. Regexp gurus can hack the script file to tune everything to their liking.

pbek commented 7 years ago

Awesome, @Maboroshy I can't wait to see the final script! :)

ludenticus commented 7 years ago

The script is working with my notes (+300), no subfolders.

I have @tag and @tag one, so I went the comma route. As I have as well BibTeX code, my RegExp is the following:

\B@([^_,{\n]+)

It parses all my tags, but I found some room for improvement:

  1. There is no tag count
  2. Tags with no notes aren't deleted
  3. I need to restart QON to update tags
Maboroshy commented 7 years ago

I see the number of notes tagged in the tag pane. Or you mean some other tag count?

As far as I understood, you don't need to restart the app to reparse tags. You need to reload note list by changing sub-folder, and all notes in the list will be loaded to RAM and parsed. Manually reloading note folder also works, I do that all the while testing.

I have an untested idea about using onNoteStored hook, regexp job and script.tagCurrentNote for instant tag parsing.

Maboroshy commented 7 years ago

The more I play with these new tagging the more I like it. I'm going to start using tags again, after long time since I ceased that due to their lack of portability. I think @pbek yet again made a leap for the app's usability, the one that can be compared with flexible UI or sub-folders. I hope support for nested in-text tags will follow.

ludenticus commented 7 years ago

Oh, my fault! I hadn't noticed a tag that was too long, and hence the tag count was «hidden». So, forget and forgive my first observation. Regarding the third, it is enough, as you state, to reload the note folder.

Maboroshy commented 7 years ago

Is there any update on support for nested in-text tags?

I'm thinking about starting to write a reminder app I was thinking about for a long time. It will be parse text for special reminder tags, which can be QON in-text tags at the same time. Making all such tags automatically reside inside single nesting tag would be very useful.

pbek commented 7 years ago

If you were asking me... No, there isn't. It would be a hell to implement that.

Maboroshy commented 7 years ago

Basically it's about the scripting function that would place a tag into other tag. UI has the same action. Could you expose it as the scripting function?

I could then parse two tags and place one into another.

pbek commented 7 years ago

Basically it's about the scripting function that would place a tag into other tag.

What would the use of that? All the other tag scripting functions of this feature request are not support it.

Maboroshy commented 7 years ago

Maybe I don't understand something. From my point of view it looks as follows.

noteTaggingHook parses @tag and puts it to the "tag root" - the first level of tags. After tags are parsed user can arrange tags nesting by UI: create new top tag and put tag inside it. noteTaggingHook will handle that correctly - new notes with @tag will be shown under top/tag.

So I thought that, having the ability to parse both top and tag by regexps, so to support nested tags we lack a function that puts tag into the top. Having that the hook can parse tag and top, and then put tag into top each time. Built-in tag names collision handling will make it safe.

Do I miss something?

pbek commented 7 years ago

noteTaggingHook parses @tag and puts it to the "tag root"

Actually the first matching tag with the same name will be used first...

How should the nested tag support work in your opinion? The current scripting function noteTaggingHook only supports a tagName.