Parse text when creating a task: e.g., "do this :: #tag, hard Monday"

[Alys]

As described in the ADD NEW: PARSE TEXT: "do this :: #tag, hard Monday" on Habitica | Trello Trello card, we'd like Habitica to interpret certain kinds of text entry so tasks and rewards can be added/edited without a lot of clicks.

The suggestion in the card is that descriptions could be separated from parsed text with "::" two colons; text without colons would simply be added with current behavior, however we don't have to go with that idea.

Michal Stasa has indicated willingness to create this feature so I'm making this issue for us to discuss the details.

Some things that are essential:

• Parsing should be done server-side (i.e., by the API) rather than client-side so that the website, official mobile apps, and third-party tools can all benefit from it.
• Must work regardless of the language used to input tasks.
• RTL languages must be considered (I am not certain of what considerations would be needed so I'm hoping others who have experience with RTL languages will comment here).
• Parsing should not happen accidentally (e.g., if someone happens to use :: for their own reasons, then they wouldn't want the parsing to take effect). Perhaps this should be handled by an option that must be switched on before parsing occurs and/or parsing only happens if you start the task with certain punctuation or code words and/or the user should perhaps be able to choose what words/symbols are used for that.

FYI @GitHubSphinx because of the language issues

[crookedneighbor]

Thoughts and questions in no particular order:

• If a mutually exclusive update is sent via the text field and the task body, which should take precedence? IE, if I send:

  {
    "text": "My task text :: Monday",
     "repeat": {
         "m": false
     }
  }

  Should the value in the repeat object take precedence? Or should the value parsed from the text field take precedence?

• Tags can have spaces in them. This will make them hard to parse. If I have the tags "Tag" and "Tag Monday" does :: #Tag Monday get parsed to tag Tag Monday or does it tag Tag and apply Monday as a repeat option.
• Each kind of task will have type specific text. This will be tricky to implement.
• Should the same parsing logic apply when updating a task?
• Parsing should not happen accidentally <-- I'd prefer not to have something extra to be configured on the user. I'd be fine with passing a flag in the task creation/update to enable parsing.
• Languages are going to be hard regardless. Anything that involves text will be difficult to handle with regard to translations. We could have particular translation strings for this and use that for the parsing check, but there are still a variety of edge cases that will make it tricky.
• RTL languages must be considered I suppose we'll have to keep a list of RTL languages and apply slightly different logic to make it work. Shouldn't be too big of a deal.

[rbreu]

Re your first bullet point: Shouldn't the parsed text take precedence? Because you want to create a new task without having to look at the defaults of the other options in the interface, and if the client doesn't implement the parsing, it will probably send defaults along because it can't know that the other values aren't needed.

[Alys]

Parsing should not happen accidentally <-- I'd prefer not to have something extra to be configured on the user. I'd be fine with passing a flag in the task creation/update to enable parsing.

Fine with me as long as the flag is unique enough that it's unlikely to occur in normal text. I think the presence of :: alone won't be enough. Maybe we could make it something like if the task starts with :: and also has :: within it (e.g., to separate the text from the other parameters) then it's considered for parsing.

[ghost]

@Alys I'm afraid I won't be of much help =(

Must work regardless of the language used to input tasks

If I understand you right, you want users to input keywords in the language they prefer and they should still be recognized and parsed. I know that i18n can be a can of worms for setting due dates for todos:

• slyle of dd/mm or mm/dd
• in word combinations different forms/words can be used in different context: e.g. next in "next week" and next in "next month" can be different; there might be several ways to say one and the same thing like "every Monday" / "each Monday" / "on Mondays" (it may be worse with "every 3 day(s)")
• etc.

So it just may need a more careful investigation.

Preliminary suggestions:

• some apps have an option of what to do when the keywords have been recognized: keep them in the title/comment (to allow editing and copying in case of false positive parsing) or delete them from title/comment - if you stick to :: pattern, it shouldn't be an issue
• I'd suggest identifying meaningful parts of keywords to trigger parsing, e.g. I'd like to call a task "hardish" and keep it in its title - but still it should be parsed as "hard" - that may be needed for some languages that can have different prefixes and postfixes for different types of tasks (habits/dailies/todos) and users will tend to write words in a way that is more natural

[Alys]

@GitHubSphinx That was useful! It's given us extra ideas to consider.

The date format can probably be best handled by requiring the player to use the same format that they've chosen in the website's settings.

We'd probably want to limit the words that could be used as keywords (for example, mandate that in English you must use "every Monday" rather than "on Mondays"). The entry form could have an optional cheat-sheet as a reminder of which words and symbols can be used (also the chosen date format). It would be nice to allow more flexibility but I think that just can't be done with a reasonable amount of effort when you consider the number of languages to be supported.

[san7hos]

Hi to all wizards and trolls,

Michal Stasa aka Santhos speaking:

I have a small insight first. You are all trying to think all cases through. While that is important, I would like to suggest a bit more agile approach. Would it be a problem to start with a limited feature, let's say target language would be english and the feature would work only when creating a new task? I am also quite busy programmer, so the smaller the chucks of work ready to ship are, the better.

@crookedneighbor

Thoughts and questions in no particular order:

If a mutually exclusive update is sent via the text field and the task body, which should take precedence? Should the value in the repeat object take precedence? Or should the value parsed from the text field take precedence?

I suggest the text takes precedence. It is more work to create such a text than to accidentally click on a checkbox, so I would trust the text more thus give it more weight.

Tags can have spaces in them. This will make them hard to parse. If I have the tags "Tag" and "Tag Monday" does :: #Tag Monday get parsed to tag Tag Monday or does it tag Tag and apply Monday as a repeat option.

I suggest that tags with spaces will have to be enclosed with quotes or a similar like symbol. I also think that all tags should be created prior to using the text, so the text will not automatically create new tags. I think this will be essential otherwise there will be a rise of tags-with-typos monsters.

Each kind of task will have type specific text. This will be tricky to implement.

I would start small, trying to implement it for one case and then refactor and spread out.

Should the same parsing logic apply when updating a task?

No and the text would not be stored, the text parser API would parse the text and then send it to the standard "create task" API.

Parsing should not happen accidentally <-- I'd prefer not to have something extra to be configured on the user. I'd be fine with passing a flag in the task creation/update to enable parsing.

I think it would not be a bad idea to have that in the general settings, default would be off, and each user would have to enable it. It is a feature for advanced users anyway and could only be confusing for newcomers. We could also create a field where a user would be able to specify his/her own symbol to activate the feature (obviously depending on the final format).

Languages are going to be hard regardless. Anything that involves text will be difficult to handle with regard to translations. We could have particular translation strings for this and use that for the parsing check, but there are still a variety of edge cases that will make it tricky.

Agree, this is always a dirty business.

RTL languages must be considered I suppose we'll have to keep a list of RTL languages and apply slightly different logic to make it work. Shouldn't be too big of a deal.

I will definitely need a right-to-left lang consultant. I have no clue what the structure of those languages is thus I am incapable of designing the functionality for them. That goes for asian languages as well.

[Alys]

In general, I like the idea of starting small and adding features later, but I'm not in favour of starting with this feature only working for English. That would send a bad message to all our players for whom English is not their native language. Habitica tries to be as inclusive as possible, and that message would say the opposite. We currently don't have perfect support for other languages but are trying to improve it. New features are translatable from the start, and in my personal opinion, this one shouldn't be any different.

Also, from a coding point of view, I suspect that if something was written to work for just one language, it would be a lot more trouble later to make it flexible enough to deal with different languages and language directions.

[paglias]

Can we start with something language agnostic like title #tag #"multi word tag" getting parsed into a task with title "title" and tagged with tag and multi word tag? Basically it would transform into tag the first word after # (or inside #"tag here") and remove #tag (note the whitespace before #) from the title

[san7hos]

@alys while I understand completely, I would say it depends on how you tell the story to the habiticans. Nonetheless,it is your call ultimately.

@paglias I support that approach

[librarianmage]

Regarding the keywords and such, Wunderlist allows keywords in task creation (after switching on an option), and then highlights each "chunk" of text it recognizes during task entry. There is also an option to keep the keywords in the task name.

[jcasimir]

I would love a shorthand like:

• ^m for Medium difficulty
• ^h for Hard difficulty

That's the feature that most frequently requires me to go into the task details. It could also use ^0 ^1 ^2 ^3 to avoid the i18n issue :)

[JohnKacz]

@jcasimir That's much clearer than what I had in mind (!, !!, !!!, etc.) which implies priority/urgency.

So I would now advocate for tags, difficulty, and due dates support be the first features to be worked on because they are (IMO) the main use cases for the most commonly added task type (To-Dos)

• Tags: #tag and #"tag with spaces"
• Difficulty: ^0, ^1, ^2, ^3
• Due Date: ???

Date gets a little hard because of the i18n issue, but maybe use [ and ]

[librarianmage]

What about tags that have quotation marks in them?

On Mon, Nov 14, 2016 at 1:19 PM John Kaczmarek notifications@github.com wrote:

@jcasimir https://github.com/jcasimir That's much clearer than what I had in mind (!, !!, !!!, etc.) which implies priority/urgency.

So I would now advocate for tags, difficulty, and due dates support be the first features to be worked on because they are (IMO) the main use cases for the most commonly added task type (To-Dos)

• Tags: #tag and #"tag with spaces"
• Difficulty: ^0, ^1, ^2, ^3
• Due Date: ???

Date gets a little hard because of the i18n issue, but maybe use [ and ]

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/HabitRPG/habitica/issues/8045#issuecomment-260433329, or mute the thread https://github.com/notifications/unsubscribe-auth/ALz_88VONPIpuxwEHeClnvTqBcAkgtAFks5q-LROgaJpZM4KCR0E .

[JohnKacz]

@MathWhiz, I guess you could say the same thing about tags that have # in them. I guess you could change it to be #tag# and #tag with spaces# but I'd be curious to know how many people use tags with quotation marks.

[jcasimir]

If I get enough work done today my reward to myself is trying to get up to speed with hacking on the project. If I make progress there I'll take a first swing at the difficulty parsing as a proof-of-concept.

[snarsnartheworkalien]

Just wanted to chime in that this can be seen as an accessibility issue. Do accessibility issues get more resources from Habitica?

Fine motor movements of the mouse can be challenging for certain hand and wrist limitations, while keyboard entry can be easier. I'd also consider this when choosing your delimiter text.

[Alys]

In the Trello card for this issue, Nathaniel Nelson has expressed interest in working on this.

In some of the comments above, # has been suggested as an indicator for the special codes for tags, attributes, etc. I suggest not using # or other markdown formatting characters because it could cause confusion.

[JohnKacz]

@Alys, could you explain how markdown formatting might cause confusion? Does Habitica use markdown in other places? If so where and for what? Bold, italic, etc.?

Seems like not conflicting with markdown would be pretty limiting if it's a real requirement.

[librarianmage]

Markdown is used pretty much anywhere user input happens. Bold, italics, code, headings — all of it is used. On Tue, Sep 5, 2017 at 9:42 PM John Kaczmarek notifications@github.com wrote:

@Alys https://github.com/alys, could you explain how markdown formatting might cause confusion? Does Habitica use markdown in other places? If so where and for what? Bold, italic, etc.?

Seems like not conflicting with markdown would be pretty limiting if it's a real requirement.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HabitRPG/habitica/issues/8045#issuecomment-327358396, or mute the thread https://github.com/notifications/unsubscribe-auth/ALz_8_N3lU16TvwulGZZpBXutzAxOWmNks5sfgacgaJpZM4KCR0E .

[JohnKacz]

So it is. I just tested it out. So # Heading 1 would be different from #tag because of the space. So it still might not be an issue but could make parsing a little more tricky I guess.

Perhaps @ could work as an alternative?

[san7hos]

You could parse it before handing it to the markdown engine, you just need to make sure not to break any markdown.

[librarianmage]

You will also want to make sure to check out how each client processes Markdown to make sure it doesn't break.

On Wed, Sep 6, 2017 at 1:16 AM Michal Stasa (Santhos) notifications@github.com wrote:

You could parse it before handing it to the markdown engine, you just need to make sure not to break any markdown.

Michal Staša

Santhos.net www.santhos.net

+420 773 454 793 michal.stasa@santhos.net

On 6 September 2017 at 04:55, John Kaczmarek notifications@github.com wrote:

So it is. I just tested it out. So # Heading 1 would be different from

tag because of the space. So it still might not be an issue but could

make parsing a little more tricky I guess.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/HabitRPG/habitica/issues/8045#issuecomment-327360145 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AIdlqdQK0M2FADhUzJfGZsh7LLezhet3ks5sfgm1gaJpZM4KCR0E

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/HabitRPG/habitica/issues/8045#issuecomment-327385795, or mute the thread https://github.com/notifications/unsubscribe-auth/ALz_8wJzDmb_i1i6mkKHBkUEK4-9IOLSks5sfjjKgaJpZM4KCR0E .

[deltaidea]

Todoist has a great implementation of such feature being optional:

• You write "foo tomorrow".
• "tomorrow" is highlighted with a colored box around it. This word now has special meaning.
• If you press Backspace, the highlighting will be removed. Now the word is just a part of the text.

[Alys]

@JohnKacz @San7hos I'm not so concerned with confusion in how it's coded but instead with confusion in the minds of the users. I don't want the Socialites and bug hunters to have to keep explaining that # something and #something have different meanings. :) It's the kind of thing that would cause repeated questions.

In addition, while # Heading 1 currently has a different effect than #Heading 1, that might not always be the case. If we change our markdown engine, #Heading 1 might become the way to do top-level headings (it has been in the past when we used a different markdown engine than now).

Even if top-level headings were prohibited in task titles, anyone who was familiar with markdown would be confused by the different meaning of #.

If characters used for markdown are avoided, the feature will be simpler to explain to users, easier for them to use, and will have code that is probably simpler, more robust, and more future-proof. There are several other characters or character combinations that could be used instead of # (@ as JohnKacz suggests, although @@ would be better to avoid conflicts with email addresses or profile names). For all those reasons, I consider not conflicting with markdown to be a real requirement.

@deltaidea I like that idea (as long as "tomorrow" and similar words worked in all languages), although we want the parsing to be done on the server side so client-side conveniences like that would need to be an additional feature, not the core feature (and hence they could be added at a later stage if they weren't part of the first release of this feature).

[deltaidea]

I'm not familiar with the API but here's an edge case. Please correct me if I'm being an idiot.

• Suppose I send { title: "foo p1", priority: null } when creating a new daily.
• Suppose the server parses p1 as priority = 1
• What if I send { title: "foo p1", priority: 2 }? Which priority setting takes precedence?

This applies to anything else parsed from the text, like tags or days.

[Alys]

@deltaidea You're not being an idiot at all! That's an important thing to consider.

You could make a case for either ... p1 or priority: 2 taking precedence but my feeling at the moment is that it would be most useful to have title: "foo p1" take precedence. For example, a user could be creating tasks with a third-party tool that hard-coded the priority: 2 attribute but the user wanted to set their own priority using Habitica's text parsing.

I think that if we can come up with a robust way for the user to indicate that they want text parsing to take effect in the task they are creating, then the text parsing rules should have the highest priority.

In fact if we allow the text parsing rules to take precedence, it probably makes this feature easier to implement. Habitica itself imposes some default task attributes (e.g., all tasks have priority 1 by default), so if text parsing always overrides other information, then Habitica's defaults will be ignored appropriately.

Task attributes that can have multiple values give a trickier edge case. If the user specifies tag "abc" in the task title and specifies tag "def" in the tag attribute, we could give the task both tags, but should we? Or should we give it only tag "abc"? Probably the latter for consistency with how we would handle attributes that can have only a single value.

[NQNStudios]

Cross-posting from the Trello thread.

@Alys I would be ready to start, but it seems I'm stopped because of the big overhaul
going on. Is this realm of the code likely to be changed in the overhaul, or would I be
safe to revert to a working version of the repo, and get started?

In response to the RTL and language-agnostic requirements that I'm just now reading, I have no experience with coding for RTL support but I'm more than willing to do some learning and give it my best shot, because the skillset will come in handy. Here's an article that I haven't read fully but looks relevant. https://docs.microsoft.com/en-us/globalization/localizability/mirroring-awareness

If there are contributors who do have experience coding for RTL support, of course it would be great to have some advice.

[Alys]

@Natman64 The website's pages will all be changing in the overhaul, so I don't recommend working on the front-end part of this yet, but the website redesign will make very few changes to the API's code, which is where the parsing needs to be done, so you'd be safe to start there!

[fncll]

I was just wondering if natural language parsing is still being worked on? It would be such a timesaver to not have to open every task to adjust settings like difficulty, due date, etc!

[NQNStudios]

Personally I gave up on it because I have no idea how to implement the feature in question while supporting every language the user could be using.

[veeeeeee]

This is a large feature that would preferably get more attention and time than we currently have resources for. We're closing it for the time being.