Additional non-breaking space added at the end of lines

janko commented 4 years ago

Prerequisites

[x] Put an X between the brackets on this line if you have done all of the following:
- Reproduced the problem in Safe Mode: https://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
- Followed all applicable steps in the debugging guide: https://flight-manual.atom.io/hacking-atom/sections/debugging/
- Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
- Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
- Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

I'm using highlights as a library outside of Atom. It works great, except that I noticed that in certain cases it adds trailing <span> </span> at the end of lines.

This is not a problem visually, but when developers are copy-pasting code into their editors, the trailing whitespace will get pasted as well.

Steps to Reproduce

Run the following script:

const Highlights = require('highlights')

const highlighter = new Highlights()

const rubyCode = `Sequel.migration do
  change do
    add_column :photos, :image_data, :text # or :jsonb
  end
end`

let html = highlighter.highlightSync({
  fileContents: rubyCode,
  scopeName: `source.ruby`,
})

console.log(html)

Expected behavior:

I expect only whitespace to be translated into  .

Actual behavior:

Additional trailing   character is added after the # or :jsonb comment:

<pre class="editor editor-colors">
  <div class="line"><span class="source ruby"><span class="support class ruby"><span>Sequel</span></span><span class="punctuation separator method ruby"><span>.</span></span><span>migration&nbsp;</span><span class="keyword control start-block ruby"><span>do</span></span></span></div>
  <div class="line"><span class="source ruby"><span>&nbsp;&nbsp;change&nbsp;</span><span class="keyword control start-block ruby"><span>do</span></span></span></div>
  <div class="line"><span class="source ruby"><span>&nbsp;&nbsp;&nbsp;&nbsp;add_column&nbsp;</span><span class="constant other symbol ruby"><span class="punctuation definition constant ruby"><span>:</span></span><span>photos</span></span><span class="punctuation separator object ruby"><span>,</span></span><span>&nbsp;</span><span class="constant other symbol ruby"><span class="punctuation definition constant ruby"><span>:</span></span><span>image_data</span></span><span class="punctuation separator object ruby"><span>,</span></span><span>&nbsp;</span><span class="constant other symbol ruby"><span class="punctuation definition constant ruby"><span>:</span></span><span>text</span></span><span>&nbsp;</span><span class="comment line number-sign ruby"><span class="punctuation definition comment ruby"><span>#</span></span><span>&nbsp;or&nbsp;:jsonb</span><span>&nbsp;</span></span></span></div>
  <div class="line"><span class="source ruby"><span>&nbsp;&nbsp;</span><span class="keyword control ruby"><span>end</span></span></span></div>
  <div class="line"><span class="source ruby"><span class="keyword control ruby"><span>end</span></span></span></div>
</pre>

I believe this is because highlights converts empty strings into  , we can see there is one extra empty string in the list of tokens:

[ [ { value: 'Sequel', scopes: [Array] },
    { value: '.', scopes: [Array] },
    { value: 'migration ', scopes: [Array] },
    { value: 'do', scopes: [Array] } ],
  [ { value: '  change ', scopes: [Array] },
    { value: 'do', scopes: [Array] } ],
  [ { value: '    add_column ', scopes: [Array] },
    { value: ':', scopes: [Array] },
    { value: 'photos', scopes: [Array] },
    { value: ',', scopes: [Array] },
    { value: ' ', scopes: [Array] },
    { value: ':', scopes: [Array] },
    { value: 'image_data', scopes: [Array] },
    { value: ',', scopes: [Array] },
    { value: ' ', scopes: [Array] },
    { value: ':', scopes: [Array] },
    { value: 'text', scopes: [Array] },
    { value: ' ', scopes: [Array] },
    { value: '#', scopes: [Array] },
    { value: ' or :jsonb', scopes: [Array] },
    { value: '', scopes: [Array] } ],
  [ { value: '  ', scopes: [Array] },
    { value: 'end', scopes: [Array] } ],
  [ { value: 'end', scopes: [Array] } ] ]

Reproduces how often:

Every time.

Versions

Highlights 3.1.4

janko commented 4 years ago

Indeed, this is where an empty string value gets converted into a space (because an empty string is falsy), which then gets converted into  :

https://github.com/atom/highlights/blob/0a4fadae979b4d83d7a858d593c1977dc000a417/src/highlights.coffee#L169

rsese commented 4 years ago

Thanks for the report!

I'm using highlights as a library outside of Atom. It works great, except that I noticed that in certain cases it adds trailing <span> </span> at the end of lines.

Just to mention up front, with our current resources we're specifically not prioritizing issues that aren't causing a problem in Atom. To clarify, is this behavior causing a particular issue in Atom?

janko commented 4 years ago

Not that I know of, I don't use Atom, I've only experienced it when using the library standalone.

atom / highlights