search

r-lib / downlit

Syntax Highlighting and Automatic Linking
https://downlit.r-lib.org
Other
90 stars 22 forks source link

factor out tokenization and export it #187

Open thomasp85 opened 7 months ago

thomasp85 commented 7 months ago

As discussed outside of GitHub, this PR factors out the tokenisation that downline uses in highlight() and exports it for use by other packages. It is also being used by highlight() now.

I have not added tests since the behaviour is covered by the extensive testing of highlight(), but let me know if you want to have this