Codemirror oddness

[danielnitsche]

I think I have this worked out. Given the following text: * ~em~

Codemirror expects this chain of events from token

1. stream.next()
2. return 'li'
3. stream.next()
4. stream.next()
5. return 'em li'
6. stream.next()
7. stream.next()
8. stream.next()
9. return 'em li'

The resulting HTML is:

<pre>
<span class="cm-li">*</span>
<span class="cm-em cm-li"> ~em~</span>
</pre>

Therefore, my understanding is you need to assign (ie. return) opening tokens at each character position, and, if you want that token to span multiple characters, return the same token and the end character of that token.

Clear as chocolate mud cake?

[cgiffard]

Whoah, how long did you sit stuffing with this for? :)

With operations where we're just advancing to the next token with stream.next(), do we have to return null instead of a token value?

Or if we know the token boundaries, do we just stream.next() until we hit them, and then return the value?

[danielnitsche]

It took a little while, not my whole weekend though :)

stream.next() advances to the next character, not token (although I'm not familiar with the specific definition of a token), and you can call it as many times as you like per codemirror token function. I think this means as you say, we stream.next() until the next duckdown token boundary is found.

The other modes seem to return null sometimes, but it doesn't seem strictly necessary. Perhaps returning null means something to codemirror I'm not familiar with.

[cgiffard]

So to clarify my previous message (which I don't think came across properly,) you call stream.next() until you hit a token boundary, and then return?

[danielnitsche]

Sounds right to me yep. Additionally, I think codemirror assumes that if a token name matches a previous name, it will wrap the characters in token in a separate span. If you return a unique token name just once, it will assume the current, single character is a token on its own, and wrap that. Like you see in the example above.