App - Parsing text using Markdown

[mhdatie]

https://github.com/Redgram/redgram-for-reddit/tree/dev/Redgram/app/src/main/java/com/matie/redgram/ui/common/utils/text

[mhdatie]

@TeeKoo

[mhdatie]

Reasources

https://www.reddit.com/r/reddit.com/comments/6ewgt/reddit_markdown_primer_or_how_do_you_do_all_that/c03nik6/

https://www.reddit.com/r/regex/comments/5kz24n/need_some_assistance_in_mimicking_markdown_with/dbrqjl5/

[mhdatie]

StringDecorator class is used to build a Spannable string by appending pieces of text and specifying the Span of each append operation. It doesn't parse a full text as it acts like a builder.

We can add a method to the builder that parses markdown only. Then call this method in a custom Textview. The builder has reference to the current class and it sets the Spannable text.

The changes would include using this custom view instead of TextView in the XMLs and call the custom method since setText(Charsequence t) is declared final, so it cannot be overriden.

A second way is to use StringDecorator builder and use the existing TextView.

A third way is to keep StringDecorator untouched and use the custom TextView to parse any markdown text.

Note: We will need the parser for the custom EditText on text change for link and comment submission..so the same parser need to be available in a common place.

[mhdatie]

I have decided to use the Builder Pattern. This is because I wanted to make sure that we apply the spans that we need and not have a default impl that could be overriden. If in the future we want to add our custom spans or we give the user the ability to choose which spans they want or the type of action they want to perform, we need to have some kind of control over what to show and over the kind of span that we need to apply on the whole text based on a Regex pattern.

So there are default methods that all use the some final parse(...) method. We pass to the builder the text and view, and after applying the spans one by one, we call the build() method to set the spannable text to the view.

All parsing methods accept a common parameter, which is Object... spans. The reason for this choice of parameter is that when finding matches using regular expression, we want to span the markdown AND replace the markdown with the actual text, while keeping the spans applied. Multiple spans is to be able to apply a set of spans first and then replace the text with the matcher grouping in the specified regex.

This is ongoing, because we will need to apply this parsing also on EditText after each key stroke for example to detect changes in the text, so it needs to be efficient.

Note: not tested but commented out a way to use it on a thread. (See PostFragment.java)

[mhdatie]

https://www.reddit.com/r/HFY/wiki/ref/faq/formatting_guide

[mhdatie]

Three issues that I'm encountering:

• I cannot use the same Span object multiple times - it keeps moving to the next match, so only the last match is captured
• This example:
  • [this /r/redditdev post **hello** *hello* ~~*hello*~~](https://www.reddit.com/r/redditdev/comments/37nmt7/upcoming_changes_to_subreddit_and_user_links_in/)
  • Result: this /r/redditdev post hello hello hello
  • In this case, if I parse /r/redditdev the span will overlay the URL text and this is NOT the expected behavior. The set of hello should have their Spans applied though
• Last issue is replacing the matching text with the captured groups. The problem here is that the Matcher will hold a reference to the start and end of all matches, and when the string is modified, the next match becomes invalid, and in some case I was receiving out of bound exceptions.

What I can do for the third point is so modify one match only, then run the Matcher again for the next one, etc but that is inefficient.

For the first point I have to somehow create new spans for each match or apply the span for each Mardown operation separately instead of passing Span objects as parameters.

I will tackle the second issue in the end

[mhdatie]

The latest commit solves the first and second scenarios

• I cannot use the same Span object multiple times - it keeps moving to the next match, so only the last match is captured
• This example:
  • [this /r/redditdev post **hello** *hello* ~~*hello*~~](https://www.reddit.com/r/redditdev/comments/37nmt7/upcoming_changes_to_subreddit_and_user_links_in/)
  • Result: this /r/redditdev post hello hello hello
  • In this case, if I parse /r/redditdev the span will overlay the URL text and this is NOT the expected behavior. The set of hello should have their Spans applied though

I still need to know how to only display the captured groups on the fly. Substrings and replacing by the groups captured is not doable as it alters the string, so maybe the best way is to build a final Spannable from left to right?

[mhdatie]

All texts in links display https://www.reddit.com/r/regex/comments/5kz24n/need_some_assistance_in_mimicking_markdown_with/dbrqjl5/

I also commented out the Cache interceptor in order to avoid Http errors #18

[mhdatie]

I believe we will need to capture the data in groups and store them somehow and build a new spannable string with only the data. Right now the span is applied on the whole Markdown captured by regular expressions.

[mhdatie]

Latest Resources on MD

https://www.reddit.com/r/reddit.com/comments/6ewgt/reddit_markdown_primer_or_how_do_you_do_all_that/

https://www.reddit.com/r/changelog/comments/mg1j6/reddit_change_new_markdown_interpreter/

[mhdatie]

I'm compiling a list of regex patterns (with examples) for the following:

• superscript --> https://regex101.com/r/QBoHeG/3
• full link --> https://regex101.com/r/kBhqyr/3 (same results as this but shorter regex)
• ordered list --> https://regex101.com/r/kBhqyr/9
• unordered list --> https://regex101.com/r/kBhqyr/8 (temp - not correct)
• separator --> https://regex101.com/r/kBhqyr/4
• code block --> https://regex101.com/r/kBhqyr/6
• block quote --> https://regex101.com/r/kBhqyr/5
• user/subreddit --> https://regex101.com/r/2iKB7k/1

IMPORTANT:

• replace multiple inline spaces in between words with a single space
• replace single \n to none (or single space)
• replace 2 or more \n into one \n

Needs optimizing:

• MD link (current --> https://regex101.com/r/syx8hO/1) (optimized --> https://regex101.com/r/syx8hO/2)

Some regex engines cannot capture multiple groups for nested markdown. It ends up capturing the last group only. So I believe we will need to apply the markdown on the first level, apply the MD, replace the text and check for any nested level, and then repeat.

Implementation will continue as soon as the full list is finalized.

[mhdatie]

I needed a way to determine the line number based on the most updated text.

I had to introduce a custom SpannableStringBuilder MDSpannableStringBuilder that triggers a listener on each span addition. In the case of SuperScriptSpans I had to trigger the listener only when the last span is added. SO I added a trigger flag so the listener isn't invoked on the fly all the time.

After setting the new text to the TextView, view.getLayout() is usually null as it's not instant. To solve this, I added:

ViewTreeObserver vto = view.getViewTreeObserver(); vto.addOnGlobalLayoutListener(() -> {...

This global listener operates on the TextView in context only. However, since it's possible to have multiple superscripts that need update and the fact that the global tree listener is NOT synchronous, I had to add the spans that I receive in a Queue and later I operate on each separate span based on a FIFO approach.

The global listener is only triggered when a view change is made (ex. setText()), so on every span affected, I had to update the text, which will trigger the listener again, and operate on any span residing in the Queue