Fix RangeError in files with multiple forward slashes

[MichalBryxi]

• Fixes #255.
• Issues stems from templates that have forward slash in them. Which are pretty much all of them: </div> or {{/if}} will be an issue.
• I extracted codeToGlimmerAst functionality to DRY the code.
• I experimented with many variants of the code as I believed that we don't need to put the whole content into the replaced template like here, but I could not find a combination that would work. So I believe it's a combination of template.contents, template.range.end, template.range.start operating in different "units" and then something breaks when there is different number of multibyte characters. So I decided to keep the template.contents as is and use the PLACEHOLDER instead.
• For now I just decided to set PLACEHOLDER = '-';. Is that a bad idea? All tests are green ...
• I added tons of units tests for preprocess.ts to make sure that function in isolation works as expected.

[MichalBryxi]

Ok, this PR is ready for review. It fixed the issues. I failed with every other avenue. It adds good chunk of test coverage. And cleans up few bits.

cc: @gitKrystan

[miguelcobain]

Maybe it would be beneficial to use a placeholder character that is less common than -? Unless I'm completely missing the way this change works.

[MichalBryxi]

That's possible, with few notes:

• The ${PLACEHOLDER} character should not (famous last words) make it anywhere back to the user. It's used just to "take the same amount of byte space" as / does.
• We can choose anything from the "basic" ASCII table (0 - 127), because those characters take the same amount of space. Tried few characters 128+ and they all make an issue.
• I am not compltely sure about this, but I think it needs to be a character that has no "special" meaning in JS comments. This is because we replace the <template> ... </template> with JS comment: {/* ... */}and we must ensure that things inside that comment don't cause an issue in the layers that code.
• We're replacing the forward slash, because that one automatically breaks assumption above by ending the JS comment.
• We could also replace the star symbol as it has the same issue, because it's part of the "closing a comment" sequence (*/).
• Thinking about lower-ascii placeholder character that does not have any special meaning in JS, so that it gives people a hint if it ever leaks:
  • # - Private fields
  • $ - jQuery
  • % - modulo
  • @ - decorators
  • ^ - XOR
  • ~ - bitwise NOT
• I could not come up with anything smarter, but if anything I'd use the tilde character ~ as I did not really see too many people doing "bitwise NOT" in their code. Every other character can be seen here and there and can be confusing if this ever causes a problem.

Pushed.

[gitKrystan]

Thanks for looking into this. Just got back from Spring Break and it was nice to see something magically removed from my TODO list while I was gone. 🎉

I've been working to remove this preprocess hack for a while because it's been the cause of many bugs, and I don't entirely understand why it's necessary.

In the meantime, this fix seems to work.

Can you take a look at the lint failures? Then I will merge.

[lint:eslint] /home/runner/work/prettier-plugin-ember-template-tag/prettier-plugin-ember-template-tag/tests/unit-tests/preprocess.test.ts
[lint:eslint]    1:1   error  Run autofix to sort these imports!                                             simple-import-sort/imports
[lint:eslint]   54:63  error  Async arrow function has no 'await' expression                                 @typescript-eslint/require-await
[lint:eslint]   56:17  error  Use `for...of` instead of `.forEach(...)`                                          unicorn/no-array-for-each
[lint:eslint]   56:36  error  The variable `i` should be named `index`. A more descriptive name will do too  unicorn/prevent-abbreviations

Thanks also for setting up the unit tests!

[MichalBryxi]

Oh, sorry for not checking the linting. I guess I rely too much on "push & let CI tell me" 🙃

[MichalBryxi]

Pushed. The linter gods should be satisfied now @gitKrystan ...

[MichalBryxi]

My hunch to as of why this is necessary is that we're working with two (or maybe three) "systems":

1. The code that is being passed around as simple string.
2. The parsed Template that has items like r.range.start and range that need to "align well" with it's own contents and also the code from (1).
3. The parsed "Glimmer template" that has similar properties to (2) and needs to satisfy the same set of requirements as in (2).

AND

The amount of characters (a, b, c) and multi byte characters has to match in all cases. For example the code breaks in one way if I try to simply count how many bytes are occupied and replace the contents of the template with the same amount of single byte characters. And it then fails in other way when I count the number of characters and replace the template with the same amount of single byte characters.

I suspect (speculation) that for "proper" fix we'd need to:

• Replace the template contents with correct amount of single byte characters
• Adjust the start, end, range and probably some other properties on the template
• And possibly also adjust the start, end and range in subsequent templates (they had to move, right?)

But IDK, that's just a feeling from my experiments.